search

linkedin / dynamometer

A tool for scale and performance testing of HDFS with a specific focus on the NameNode.
BSD 2-Clause "Simplified" License
131 stars 34 forks source link

Error when running start-workload.sh #60

Closed sunchao closed 5 years ago

sunchao commented 6 years ago

In the README it says:

./bin/start-workload.sh
    -Dauditreplay.input-path hdfs:///dyno/audit_logs/
    -Dauditreplay.num-threads 50
    -nn_uri hdfs://namenode_address:port/
    -start_time_offset 5m
    -mapper_class_name AuditReplayMapper

However, it seems both Dauditreplay.input-path and -Dauditreplay.num-threads are not valid options. Only valid options are: nn_uri, start_time_offset, start_timestamp_ms, and mapper_class_name?

xkrogen commented 6 years ago

Ah.. The -D flag is used to specify configurations, not input arguments -- the Client is run using ToolRunner so the standard Hadoop argument parsing is done for -D. However, it needs to have an = instead of a space separating the config key and value. Can you try the following instead?

./bin/start-workload.sh
    -Dauditreplay.input-path=hdfs:///dyno/audit_logs/
    -Dauditreplay.num-threads=50
    -nn_uri hdfs://namenode_address:port/
    -start_time_offset 5m
    -mapper_class_name AuditReplayMapper
sunchao commented 6 years ago

Thanks @xkrogen ! and sorry for the late reply. I've opened a PR to fix a few small issues I encountered while running start-workload.sh.

hexiangheng commented 5 years ago

when i running start-workload.sh but failed The running command: ./bin/start-workload.sh -Dauditreplay.input-path=hdfs:///dyno/hdfs-audit.log -Dauditreplay.num-threads=50 -nn_uri hdfs://EC130:9000/ -start_time_offset 5m -mapper_class_name AuditReplayMapper

SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hxh/hadoop/hadoop-3.3.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hxh/hadoop/dynamometer-fat-0.1.0-SNAPSHOT/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Exception in thread "main" java.lang.ClassNotFoundException: Class com.linkedin.dynamometer.workloadgenerator.AuditReplayMapper not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2529) at com.linkedin.dynamometer.workloadgenerator.WorkloadDriver.getMapperClass(WorkloadDriver.java:146) at com.linkedin.dynamometer.workloadgenerator.WorkloadDriver.run(WorkloadDriver.java:99) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at com.linkedin.dynamometer.workloadgenerator.WorkloadDriver.main(WorkloadDriver.java:139) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

xkrogen commented 5 years ago

Hm, yes, this issue is what PR #62 is trying to fix. There's a small issue there.. Let me put together another patch for that

xkrogen commented 5 years ago

Hey @sunchao and @hexiangheng , I submitted my own PR for this #84, can you take a look there and see if it looks good?

xkrogen commented 5 years ago

Closed by PR #84