search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.39k stars 1.26k forks source link

ingestion job OOM on a complex json #10229

Open fritzb opened 1 year ago

fritzb commented 1 year ago

The following JSON will cause OOM for the ingestion job:

{"lane_completeness":{"false_negative_count":6,"false_negative_measurement":116.265,"false_negative_feature_ids":["27666694","27666695","27666698","27666699","27666696","27666697"],"false_positive_count":3,"false_positive_measurement":116.263,"false_positive_feature_ids":["26946590","26946589","26946588"],"total_issues_measurement":232.52800000000002,"total_issues_count":9,"total_measurement":73532.706,"total_count":2030},"lane_connectivity":{"false_negative_count":0,"false_negative_measurement":0,"false_negative_feature_ids":[],"false_positive_count":0,"false_positive_measurement":0,"false_positive_feature_ids":[],"total_issues_measurement":0,"total_issues_count":0,"total_measurement":73532.706,"total_count":2030},"lane_direction":{"false_count":0,"false_measurement":0,"false_feature_ids":[],"total_measurement":73532.706,"total_count":2030},"lane_restriction":{"false_count":123,"false_measurement":4110.732999999999,"false_feature_ids":["26954139","26954141","26954140","26954131","26954130","26954129","26954135","27425199","26954134","26954133","27425200","26954144","26956033","26956032","26954043","26954042","26954041","26954044","27383071","27383070","26954075","26954078","26954077","26954076","26954066","26954065","26954064","26954063","26954105","26954104","26954083","26954082","26954081","26954080","26956030","26956029","26955871","26955898","26955897","26955896","26955891","26955890","26955889","26955888","26955895","26955892","26955883","26955882","26955887","26955886","26955872","27382763","27382765","27382764","27382766","27382723","27382725","27382724","27382726","26954270","26954269","26954299","26954298","26954297","26954303","26954302","26954301","26954300","26954289","26954288","26954295","26954294","26954293","26954292","26954281","26954280","26954285","26954284","26954273","26954272","26954277","26954276","26954315","26954314","26954304","26955387","26955384","26955390","26955389","26955388","26955383","26955382","26955381","26962819","27383269","27383259","26960665","26958617","26960671","26958622","26960659","26958611","26960653","26958605","26958598","26960700","26960691","26960683","26960677","26960705","26960710","7311668","26960581","26960634","26960639","26960624","26960629","26960619","26958573","26960614","27383367","26947133","26947141"],"total_measurement":73532.706,"total_count":2030},"lane_speedlimit":{"false_count":82,"false_measurement":3312.8169999999996,"false_feature_ids":["26954143","26954142","26954141","26954140","26954131","26954130","26954129","26954135","26954134","26954133","26954132","26956035","26956034","26956033","26956032","26956036","27383025","27383026","26956031","26956030","26956029","27382763","27382765","27382764","27382766","27382723","27382725","27382724","27382726","26955451","26954271","26954270","26954269","26954268","26954299","26954298","26954297","26954303","26954302","26954301","26954300","26954290","26954289","26954288","26954295","26954294","26954293","26954292","26954283","26954282","26954281","26954280","26954287","26954286","26954285","26954284","26954275","26954274","26954273","26954272","26954279","26954278","26954277","26954276","26954315","26954314","26954316","26954304","26946763","26946762","26946761","26946766","26946765","26946764","26946523","26946522","26946521","26946520","26946527","26946526","26946525","26946524"],"total_measurement":73532.706,"total_count":2030},"lane_divider":{"false_negative_count":18,"false_negative_measurement":510.3875979291562,"false_negative_feature_ids":["26613748-27369322-64182856-0-27633519","26825956-26825875-1122194151-0-27634932","26825764-26825623-1895572963-0-27634932","26831954-26831913-1915559709-0-27636585","27634953-26837680-1016872863-0-27634932","26837680-26825956-1096618590-0-27634932","26832635-27634953-878825235-0-26729749","26831913-26825561-911944602-0-27634932","27634953-26837680-2002177669-0-26729749","26832635-27634953-1617122605-0-26832418","26831954-26831913-1789961779-0-27634932","26825875-26825764-1341470975-0-27634932","26825623-26831954-1313653835-0-27634932","26825623-26831954-1439251765-0-27636585","27634953-26837680-1721528859-0-26832418","26831913-26825561-786346672-0-27636585","26832635-27634953-1261990121-0-26730628","26825764-26825623-2021170893-0-27636585"],"false_positive_count":15,"false_positive_measurement":510.3875264134282,"false_positive_feature_ids":["26613748-27369322-981063286-0-26529237","26832635-26837680-498818694-0-26730628","26831954-26831913-1549041869-0-26832418","26825875-26825764-482474373-0-26730628","26837680-26825956-1955615192-0-26730628","26831913-26825561-52948000-0-26730628","26832635-26837680-143686210-0-26832418","26825956-26825875-1981190753-0-26730628","26825764-26825623-1443430685-0-26832418","26825623-26831954-2025349813-0-26832418","26831954-26831913-1646008915-0-26730628","26831913-26825561-44019046-0-26832418","26832635-26837680-881983580-0-26729749","26825623-26831954-2122316859-0-26730628","26825764-26825623-1540397731-0-26730628"],"total_issues_measurement":1020.7751243425844,"total_issues_count":33,"total_measurement":80675.2281863184,"total_count":2485},"lane_divider_style":{"false_count":42,"false_measurement":1298.1180273025948,"false_feature_ids":["26837947-26912755-430075243-0-26730460","26833082-26832635-463734690-0-27367976","26912755-26832880-54375730-0-26730460","26893021-26838770-2055919089-0-26842063","26833082-26832635-814699793-0-26735762","26912755-26832880-1096931333-0-26735617","26824959-26825037-1381073191-0-26904039","26831954-26831913-1771689827-0-26731380","26838078-26837971-1929125790-0-26842063","26832880-26833082-401385435-0-26735617","26837971-26851353-1122313533-0-26735617","26831913-26825561-178628912-0-26731380","26831967-26831921-380113934-0-26730774","26825037-26851904-374400135-0-26725921","26831921-26825591-734458752-0-26730774","26833082-26832635-437801709-0-26733373","26825875-26825764-385507327-0-26832418","26837373-26837111-1115878050-0-26735762","26851353-26837947-422022092-0-26735617","26851353-26837947-729284971-0-26730460","26825623-26831954-2046969525-0-26731380","26851904-26841085-2078004263-0-26725921","26825956-26825875-2078157799-0-26832418","26912022-26842031-1913343144-0-26735354","26841085-26841202-1296947823-0-26725921","26840535-26838922-1998479139-0-26842063","26837947-26912755-1581683764-0-26735617","26842115-26912022-581835822-0-26735354","26844365-26842115-1980048878-0-26735354","26825623-26831967-2056119565-0-26730774","26838770-26840535-465661131-0-26842063","26892376-26893021-164302081-0-26842063","26838922-26838078-586424693-0-26842063","26832880-26833082-401385435-1-26735762","26832880-26833082-1270545795-0-26730460","26824926-26824944-1665802689-0-26904039","26824904-26824926-1210316989-0-26904039","26837680-26825956-1543109148-0-26832418","26824944-26824959-1324528993-0-26904039","26842031-26892376-1288857653-0-26842063","26832635-26837373-1754437167-0-26735762","26838770-26840535-938013490-0-26729532"],"total_measurement":80675.2281863184,"total_count":2485},"lane_divider_color":{"false_count":0,"false_measurement":0,"false_feature_ids":[],"total_measurement":80675.2281863184,"total_count":2485},"lane_divider_material":{"false_count":0,"false_measurement":0,"false_feature_ids":[],"total_measurement":80675.2281863184,"total_count":2485},"road_boundary":{"false_negative_count":2,"false_negative_measurement":38.83976538873871,"false_negative_feature_ids":["26832635-27634953-517262708-0-26696451","27634953-26837680-2144879064-0-26696451"],"false_positive_count":1,"false_positive_measurement":38.83966278347907,"false_positive_feature_ids":["26832635-26837680-1243546107-0-26696451"],"total_issues_measurement":77.67942817221778,"total_issues_count":3,"total_measurement":15555.671521942104,"total_count":519},"road_boundary_type":{"false_count":1,"false_measurement":3.9238851740144844,"false_feature_ids":["26824629-26824714-1252792951-0-26824616"],"total_measurement":15555.671521942104,"total_count":519},"total_issues_count":293,"total_issues_measurement":10056.574464991409}

Here is the stack trace:

Start building IndexCreator!
Got exception to kick off standalone data ingestion job -
java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
    at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:152) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:125) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:165) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:196) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
Caused by: java.lang.RuntimeException: Failed to generate Pinot segment for file - s3://deepmap-anga-production/metrics/etl_staging/pinot_ingest/map_making_metrics/date=2023-02-03/part-0-0
    at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(SegmentGenerationJobRunner.java:309) ~[pinot-batch-ingestion-standalone-0.11.0-shaded.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
    at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
    at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.ArrayList.<init>(Unknown Source) ~[?:?]
    at org.apache.pinot.spi.utils.JsonUtils.unnestResults(JsonUtils.java:461) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.spi.utils.JsonUtils.unnestResults(JsonUtils.java:469) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.spi.utils.JsonUtils.unnestResults(JsonUtils.java:469) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.spi.utils.JsonUtils.unnestResults(JsonUtils.java:469) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.spi.utils.JsonUtils.flatten(JsonUtils.java:444) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.segment.local.segment.creator.impl.inv.json.BaseJsonIndexCreator.add(BaseJsonIndexCreator.java:92) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(SegmentColumnarIndexCreator.java:491) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:235) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.plugin.ingestion.batch.common.SegmentGenerationTaskRunner.run(SegmentGenerationTaskRunner.java:119) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(SegmentGenerationJobRunner.java:287) ~[pinot-batch-ingestion-standalone-0.11.0-shaded.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
    at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner$$Lambda$794/0x00007fa5ec9cf960.run(Unknown Source) ~[?:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
    at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
    at java.lang.Thread.run(Unknown Source) ~[?:?]
java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
    at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:152)
    at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:125)
    at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130)
    at org.apache.pinot.tools.Command.call(Command.java:33)
    at org.apache.pinot.tools.Command.call(Command.java:29)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
    at picocli.CommandLine.access$1300(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
    at picocli.CommandLine.execute(CommandLine.java:2078)
    at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:165)
    at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:196)
Caused by: java.lang.RuntimeException: Failed to generate Pinot segment for file - s3://deepmap-anga-production/metrics/etl_staging/pinot_ingest/map_making_metrics/date=2023-02-03/part-0-0
    at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(SegmentGenerationJobRunner.java:309)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.ArrayList.<init>(Unknown Source)
    at org.apache.pinot.spi.utils.JsonUtils.unnestResults(JsonUtils.java:461)
    at org.apache.pinot.spi.utils.JsonUtils.unnestResults(JsonUtils.java:469)
    at org.apache.pinot.spi.utils.JsonUtils.unnestResults(JsonUtils.java:469)
    at org.apache.pinot.spi.utils.JsonUtils.unnestResults(JsonUtils.java:469)
    at org.apache.pinot.spi.utils.JsonUtils.flatten(JsonUtils.java:444)
    at org.apache.pinot.segment.local.segment.creator.impl.inv.json.BaseJsonIndexCreator.add(BaseJsonIndexCreator.java:92)
    at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(SegmentColumnarIndexCreator.java:491)
    at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:235)
    at org.apache.pinot.plugin.ingestion.batch.common.SegmentGenerationTaskRunner.run(SegmentGenerationTaskRunner.java:119)
    at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(SegmentGenerationJobRunner.java:287)
    at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner$$Lambda$794/0x00007fa5ec9cf960.run(Unknown Source)
    ... 5 more

Nice to have if the bad json can be caught with a hint of the offending row.

Jackie-Jiang commented 1 year ago

Created #10236 to include more information when it runs into exception/error

Please refer to https://docs.pinot.apache.org/basics/indexing/json-index about how to reduce the memory usage

fritzb commented 1 year ago

Thanks. As of this issue, one row with 8kb json will cause OOM on 100Gb node. Looks like an infinite loop/recursion.

Jackie-Jiang commented 1 year ago

It is caused by cross array flattening, where by default Pinot calculates all combinations of each element from multiple arrays. This can be disabled by setting disableCrossArrayUnnest in the json index configs