search

aaron7 / lima

Group Project Lima
1 stars 0 forks source link

DosJob Run Test #1

Closed aaron7 closed 11 years ago

aaron7 commented 11 years ago

After 5-10 mins of running the job on the HDFS local system - around 100% CPU on all 4 cores 6GB memory used: Gets to the final reduce and gives this error

13/02/25 00:23:18 INFO mapreduce.Job:  map 90% reduce 28%
13/02/25 00:23:21 INFO mapreduce.Job:  map 94% reduce 28%
13/02/25 00:23:22 INFO mapreduce.Job:  map 100% reduce 28%
13/02/25 00:23:22 INFO mapreduce.Job: Task Id : attempt_1361746790989_0001_r_000000_0, Status : FAILED
Error: java.lang.ClassCastException: class uk.ac.cam.cl.groupproject12.lima.hadoop.DosJob$DoSAttack
    at java.lang.Class.asSubclass(Class.java:3049)
    at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getSequenceFileOutputKeyClass(SequenceFileAsBinaryOutputFormat.java:86)
    at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getRecordWriter(SequenceFileAsBinaryOutputFormat.java:123)
    at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:494)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:419)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)

13/02/25 00:23:23 INFO mapreduce.Job:  map 100% reduce 0%
13/02/25 00:23:26 INFO mapreduce.Job: Task Id : attempt_1361746790989_0001_r_000000_1, Status : FAILED
Error: java.lang.ClassCastException: class uk.ac.cam.cl.groupproject12.lima.hadoop.DosJob$DoSAttack
    at java.lang.Class.asSubclass(Class.java:3049)
    at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getSequenceFileOutputKeyClass(SequenceFileAsBinaryOutputFormat.java:86)
    at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getRecordWriter(SequenceFileAsBinaryOutputFormat.java:123)
    at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:494)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:419)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)

13/02/25 00:23:32 INFO mapreduce.Job: Task Id : attempt_1361746790989_0001_r_000000_2, Status : FAILED
Error: java.lang.ClassCastException: class uk.ac.cam.cl.groupproject12.lima.hadoop.DosJob$DoSAttack
    at java.lang.Class.asSubclass(Class.java:3049)
    at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getSequenceFileOutputKeyClass(SequenceFileAsBinaryOutputFormat.java:86)
    at org.apache.hadoop.mapred.SequenceFileAsBinaryOutputFormat.getRecordWriter(SequenceFileAsBinaryOutputFormat.java:123)
    at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:494)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:419)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)

13/02/25 00:23:36 INFO mapreduce.Job:  map 100% reduce 100%
13/02/25 00:23:36 INFO mapreduce.Job: Job job_1361746790989_0001 failed with state FAILED due to: 
13/02/25 00:23:36 INFO mapreduce.Job: Counters: 31
    File System Counters
        FILE: Number of bytes read=2400
        FILE: Number of bytes written=1986816
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=1295056907
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=60
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=0
    Job Counters 
        Failed reduce tasks=4
        Launched map tasks=20
        Launched reduce tasks=4
        Data-local map tasks=20
        Total time spent by all maps in occupied slots (ms)=9574328
        Total time spent by all reduces in occupied slots (ms)=1199720
    Map-Reduce Framework
        Map input records=8749823
        Map output records=4083
        Map output bytes=661630
        Map output materialized bytes=669916
        Input split bytes=2240
        Combine input records=0
        Spilled Records=4083
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=20825
        CPU time spent (ms)=260980
        Physical memory (bytes) snapshot=5161984000
        Virtual memory (bytes) snapshot=21500129280
        Total committed heap usage (bytes)=4141350912
    File Input Format Counters 
        Bytes Read=1295054667
Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:891)
    at uk.ac.cam.cl.groupproject12.lima.hadoop.RunDosJob.main(RunDosJob.java:60)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
aaron7 commented 11 years ago

Need someone to run on the Cloudera VM to check. Build a jar file with the RunDOSJob.java as the main class and call: hadoop jar runJob.jar You will need to put the csv file into the HDFS under input (as described in code). Use: nfdump -r data.nfcapd -o "fmt:%ra,%ts,%te,%pr,%sa,%da,%sp,%dp,%pkt,%byt,%flg,%tos" -q -N >> netflow_anonymous.csv to generate the csv file for the job and use HDFS's put operation to put it into the file system.