harishwable
commented
6 years ago Immediate response appreciated...!!!!
Open harishwable opened 6 years ago
harishwable
commented
6 years ago Immediate response appreciated...!!!!
Please help on below issue! I am trying to ingest Parquet file into druid. but ingestion job fails. I have added extension - parquet-extensions and using druid version:-druid-0.11.0
Please refer below logs and suggest! ----------------------------------------------Logs------------------------------------ 2018-02-16T07:44:48,538 INFO [main] io.druid.indexer.path.StaticPathSpec - Adding paths[hdfs://10.1.1.21:8020/druid/sample_data/wikipedia_list.parquet] 2018-02-16T07:44:49,117 INFO [main] io.druid.indexer.JobHelper - Uploading jar to path[/tmp/druid-indexing/wikipedia/2018-02-16T074448.117Z_175a264e94ae43048c80676e366df66c/classpath/hadoop-annotations-2.7.3.jar] 2018-02-16T07:44:50,012 INFO [main] io.druid.indexer.JobHelper - Renaming jar to path[/tmp/druid-indexing/classpath/hadoop-annotations-2.7.3.jar] 2018-02-16T07:44:50,089 INFO [main] io.druid.indexer.JobHelper - Uploading jar to path[/tmp/druid-indexing/wikipedia/2018-02-16T074448.117Z_175a264e94ae43048c80676e366df66c/classpath/netty-all-4.0.23.Final.jar] 2018-02-16T07:49:48,396 WARN [ResponseProcessor for block BP-746179933-10.1.1.21-1482215734354:blk_1074402781_662163] org.apache.hadoop.hdfs.DFSClient - Slow ReadProcessor read fields took 62553ms (threshold=30000ms); ack: seqno: 19 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 168370116536, targets: [DatanodeInfoWithStorage[10.1.1.32:50015,DS-d6e35861-d700-4fb8-a058-83d180991ad7,DISK], DatanodeInfoWithStorage[10.1.1.31:50015,DS-6475098f-861b-4be9-9ac8-c6e6d707dbc6,DISK], DatanodeInfoWithStorage[10.1.1.33:50015,DS-a061895a-2870-4d47-9f10-af315fa9e680,DISK]] 2018-02-16T07:50:31,723 WARN [ResponseProcessor for block BP-746179933-10.1.1.21-1482215734354:blk_1074402781_662163] org.apache.hadoop.hdfs.DFSClient - Slow ReadProcessor read fields took 30872ms (threshold=30000ms); ack: seqno: 21 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 180878915841, targets: [DatanodeInfoWithStorage[10.1.1.32:50015,DS-d6e35861-d700-4fb8-a058-83d180991ad7,DISK], DatanodeInfoWithStorage[10.1.1.31:50015,DS-6475098f-861b-4be9-9ac8-c6e6d707dbc6,DISK], DatanodeInfoWithStorage[10.1.1.33:50015,DS-a061895a-2870-4d47-9f10-af315fa9e680,DISK]] 2018-02-16T07:50:47,679 WARN [ResponseProcessor for block BP-746179933-10.1.1.21-1482215734354:blk_1074402781_662163] org.apache.hadoop.hdfs.DFSClient - DFSOutputStream ResponseProcessor exception for block BP-746179933-10.1.1.21-1482215734354:blk_1074402781_662163 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2282) ~[hadoop-hdfs-2.7.3.jar:?] at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244) ~[hadoop-hdfs-2.7.3.jar:?] at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:733) [hadoop-hdfs-2.7.3.jar:?] 2018-02-16T07:50:47,685 WARN [DataStreamer for file /tmp/druid-indexing/wikipedia/2018-02-16T074448.117Z_175a264e94ae43048c80676e366df66c/classpath/netty-all-4.0.23.Final.jar block BP-746179933-10.1.1.21-1482215734354:blk_1074402781_662163] org.apache.hadoop.hdfs.DFSClient - Error Recovery for block BP-746179933-10.1.1.21-1482215734354:blk_1074402781_662163 in pipeline DatanodeInfoWithStorage[10.1.1.32:50015,DS-d6e35861-d700-4fb8-a058-83d180991ad7,DISK], DatanodeInfoWithStorage[10.1.1.31:50015,DS-6475098f-861b-4be9-9ac8-c6e6d707dbc6,DISK], DatanodeInfoWithStorage[10.1.1.33:50015,DS-a061895a-2870-4d47-9f10-af315fa9e680,DISK]: bad datanode DatanodeInfoWithStorage[10.1.1.32:50015,DS-d6e35861-d700-4fb8-a058-83d180991ad7,DISK] 2018-02-16T07:51:05,955 WARN [main] org.apache.hadoop.hdfs.DFSClient - Slow waitForAckedSeqno took 375851ms (threshold=30000ms) 2018-02-16T07:51:05,959 INFO [main] io.druid.indexer.JobHelper - Renaming jar to path[/tmp/druid-indexing/classpath/netty-all-4.0.23.Final.jar] 2018-02-16T07:51:05,994 INFO [main] io.druid.indexer.JobHelper - Uploading jar to path[/tmp/druid-indexing/wikipedia/2018-02-16T074448.117Z_175a264e94ae43048c80676e366df66c/classpath/slf4j-api-1.7.10.jar] 2018-02-16T07:51:06,346 INFO [main] io.druid.indexer.JobHelper - Renaming jar to path[/tmp/druid-indexing/classpath/slf4j-api-1.7.10.jar] 2018-02-16T07:51:06,368 INFO [main] io.druid.indexer.JobHelper - Uploading jar to path[/tmp/druid-indexing/wikipedia/2018-02-16T074448.117Z_175a264e94ae43048c80676e366df66c/classpath/slf4j-log4j12-1.7.10.jar] 2018-02-16T07:51:08,066 INFO [main] io.druid.indexer.JobHelper - Renaming jar to path[/tmp/druid-indexing/classpath/slf4j-log4j12-1.7.10.jar] 2018-02-16T07:51:08,077 INFO [main] io.druid.indexer.JobHelper - Uploading jar to path[/tmp/druid-indexing/wikipedia/2018-02-16T074448.117Z_175a264e94ae43048c80676e366df66c/classpath/zookeeper-3.4.6.jar] 2018-02-16T07:53:11,480 WARN [main] org.apache.hadoop.hdfs.DFSClient - Slow waitForAckedSeqno took 123393ms (threshold=30000ms) 2018-02-16T07:53:11,485 INFO [main] io.druid.indexer.JobHelper - Renaming jar to path[/tmp/druid-indexing/classpath/zookeeper-3.4.6.jar] 2018-02-16T07:53:11,751 INFO [main] io.druid.indexer.path.StaticPathSpec - Adding paths[hdfs://10.1.1.21:8020/druid/sample_data/wikipedia_list.parquet] 2018-02-16T07:53:11,860 INFO [main] org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at vmcloudera.htintra.net/10.1.1.21:8032 2018-02-16T07:53:12,360 WARN [main] org.apache.hadoop.mapreduce.JobResourceUploader - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2018-02-16T07:53:12,367 WARN [main] org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2018-02-16T07:53:12,826 INFO [main] org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1 2018-02-16T07:53:12,827 INFO [main] org.apache.parquet.hadoop.ParquetInputFormat - Total input paths to process : 1 2018-02-16T07:53:13,512 INFO [main] org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1 2018-02-16T07:54:01,296 WARN [ResponseProcessor for block BP-746179933-10.1.1.21-1482215734354:blk_1074402787_662170] org.apache.hadoop.hdfs.DFSClient - Slow ReadProcessor read fields took 47277ms (threshold=30000ms); ack: seqno: 0 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 64593253, targets: [DatanodeInfoWithStorage[10.1.1.33:50015,DS-a061895a-2870-4d47-9f10-af315fa9e680,DISK], DatanodeInfoWithStorage[10.1.1.31:50015,DS-6475098f-861b-4be9-9ac8-c6e6d707dbc6,DISK], DatanodeInfoWithStorage[10.1.1.32:50015,DS-d6e35861-d700-4fb8-a058-83d180991ad7,DISK]] 2018-02-16T07:54:41,292 WARN [ResponseProcessor for block BP-746179933-10.1.1.21-1482215734354:blk_1074402787_662170] org.apache.hadoop.hdfs.DFSClient - Slow ReadProcessor read fields took 39996ms (threshold=30000ms); ack: seqno: 1 reply: SUCCESS reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 63327344, targets: [DatanodeInfoWithStorage[10.1.1.33:50015,DS-a061895a-2870-4d47-9f10-af315fa9e680,DISK], DatanodeInfoWithStorage[10.1.1.31:50015,DS-6475098f-861b-4be9-9ac8-c6e6d707dbc6,DISK], DatanodeInfoWithStorage[10.1.1.32:50015,DS-d6e35861-d700-4fb8-a058-83d180991ad7,DISK]] 2018-02-16T07:54:58,716 WARN [main] org.apache.hadoop.hdfs.DFSClient - Slow waitForAckedSeqno took 105120ms (threshold=30000ms) 2018-02-16T07:54:58,719 INFO [main] org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1518762719502_0003 2018-02-16T07:54:58,785 INFO [main] org.apache.hadoop.mapred.YARNRunner - Job jar is not present. Not adding any jar to the list of resources. 2018-02-16T07:55:00,175 INFO [main] org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1518762719502_0003 2018-02-16T07:55:00,202 INFO [main] org.apache.hadoop.mapreduce.Job - The url to track the job: http://vmcloudera.htintra.net:8088/proxy/application_1518762719502_0003/ 2018-02-16T07:55:00,202 INFO [main] io.druid.indexer.DetermineHashedPartitionsJob - Job wikipedia-determine_partitions_hashed-Optional.of([2013-08-30T00:00:00.000Z/2013-09-02T00:00:00.000Z]) submitted, status available at: http://vmcloudera.htintra.net:8088/proxy/application_1518762719502_0003/ 2018-02-16T07:55:00,203 INFO [main] org.apache.hadoop.mapreduce.Job - Running job: job_1518762719502_0003 2018-02-16T07:55:32,324 INFO [main] org.apache.hadoop.mapreduce.Job - Job job_1518762719502_0003 running in uber mode : false 2018-02-16T07:55:32,325 INFO [main] org.apache.hadoop.mapreduce.Job - map 0% reduce 0% 2018-02-16T07:55:32,345 INFO [main] org.apache.hadoop.mapreduce.Job - Job job_1518762719502_0003 failed with state FAILED due to: Application application_1518762719502_0003 failed 2 times due to AM Container for appattempt_1518762719502_0003_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://vmcloudera.htintra.net:8088/proxy/application_1518762719502_0003/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1518762719502_0003_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:601) at org.apache.hadoop.util.Shell.run(Shell.java:504) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. 2018-02-16T07:55:32,360 INFO [main] org.apache.hadoop.mapreduce.Job - Counters: 0 2018-02-16T07:55:32,362 ERROR [main] io.druid.indexer.DetermineHashedPartitionsJob - Job failed: job_1518762719502_0003 2018-02-16T07:55:32,363 ERROR [main] io.druid.cli.CliHadoopIndexer - failure!!!! java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141] at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:117) [druid-services-0.11.0.jar:0.11.0] at io.druid.cli.Main.main(Main.java:108) [druid-services-0.11.0.jar:0.11.0] Caused by: io.druid.java.util.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed! at io.druid.indexer.JobHelper.runJobs(JobHelper.java:390) ~[druid-indexing-hadoop-0.11.0.jar:0.11.0] at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.11.0.jar:0.11.0] at io.druid.indexer.JobHelper.runJobs(JobHelper.java:368) ~[druid-indexing-hadoop-0.11.0.jar:0.11.0] at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:132) ~[druid-services-0.11.0.jar:0.11.0] at io.druid.cli.Main.main(Main.java:108) ~[druid-services-0.11.0.jar:0.11.0]
What could the solution for the same.....?