search

citiususc / BigBWA

BigBWA is a new tool that uses the Big Data technology Hadoop to boost the performance of the Burrows–Wheeler aligner (BWA).
GNU General Public License v3.0
31 stars 6 forks source link

invalid ELF header error #12

Open Maryom opened 6 years ago

Maryom commented 6 years ago

Hi,

Thanks for this repo.

When I ran:

yarn jar BigBWA-2.1.jar com.github.bigbwa.BigBWA -D mapreduce.input.fileinputformat.split.minsize=123641127 -D mapreduce.input.fileinputformat.split.maxsize=123641127 -D mapreduce.map.memory.mb=7500 -m -p --index /Data/HumanBase/lambda_virus -r ERR000589.fqBD ExitERR000589

I got this error:

18/01/28 16:45:31 INFO mapreduce.Job: Running job: job_1517157509312_0002
18/01/28 16:45:37 INFO mapreduce.Job: Job job_1517157509312_0002 running in uber mode : false
18/01/28 16:45:37 INFO mapreduce.Job:  map 0% reduce 0%
18/01/28 16:45:42 INFO mapreduce.Job: Task Id : attempt_1517157509312_0002_m_000000_0, Status : FAILED
Error: /mnt/yarn/usercache/hadoop/appcache/application_1517157509312_0002/container_1517157509312_0002_01_000002/tmp/libbwa8142273599705513639.so: /mnt/yarn/usercache/hadoop/appcache/application_1517157509312_0002/container_1517157509312_0002_01_000002/tmp/libbwa8142273599705513639.so: invalid ELF header (Possible cause: endianness mismatch)
18/01/28 16:45:47 INFO mapreduce.Job: Task Id : attempt_1517157509312_0002_m_000000_1, Status : FAILED
Error: /mnt/yarn/usercache/hadoop/appcache/application_1517157509312_0002/container_1517157509312_0002_01_000003/tmp/libbwa7110550933609652176.so: /mnt/yarn/usercache/hadoop/appcache/application_1517157509312_0002/container_1517157509312_0002_01_000003/tmp/libbwa7110550933609652176.so: invalid ELF header (Possible cause: endianness mismatch)
18/01/28 16:45:52 INFO mapreduce.Job: Task Id : attempt_1517157509312_0002_m_000000_2, Status : FAILED
Error: /mnt/yarn/usercache/hadoop/appcache/application_1517157509312_0002/container_1517157509312_0002_01_000004/tmp/libbwa6662802721160382805.so: /mnt/yarn/usercache/hadoop/appcache/application_1517157509312_0002/container_1517157509312_0002_01_000004/tmp/libbwa6662802721160382805.so: invalid ELF header (Possible cause: endianness mismatch)
18/01/28 16:45:58 INFO mapreduce.Job:  map 100% reduce 100%
18/01/28 16:45:58 INFO mapreduce.Job: Job job_1517157509312_0002 failed with state FAILED due to: Task failed task_1517157509312_0002_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

18/01/28 16:45:58 INFO mapreduce.Job: Counters: 16
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=3
        Rack-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=3085785
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=13131
        Total time spent by all reduce tasks (ms)=0
        Total vcore-milliseconds taken by all map tasks=13131
        Total vcore-milliseconds taken by all reduce tasks=0
        Total megabyte-milliseconds taken by all map tasks=98482500
        Total megabyte-milliseconds taken by all reduce tasks=0
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0

I used hadoop cluster on Amazon EMR.

Maryom commented 6 years ago

Note that I used EFS to share the index between nodes in the cluster as you can see all files are there:

[hadoop@ip-172-31-2-103 efs]$ aws s3 cp s3://mariamup/nice/lambda_virus . --recursive
download: s3://mariamup/nice/lambda_virus/lambda_virus.fa to ./lambda_virus.fa
download: s3://mariamup/nice/lambda_virus/lambda_virus.dict to ./lambda_virus.dict
download: s3://mariamup/nice/lambda_virus/lambda_virus.fa.ann to ./lambda_virus.fa.ann
download: s3://mariamup/nice/lambda_virus/lambda_virus.fa.amb to ./lambda_virus.fa.amb
download: s3://mariamup/nice/lambda_virus/lambda_virus.fa.bwt to ./lambda_virus.fa.bwt
download: s3://mariamup/nice/lambda_virus/lambda_virus.fa.sa to ./lambda_virus.fa.sa
download: s3://mariamup/nice/lambda_virus/lambda_virus.fa.fai to ./lambda_virus.fa.fai
download: s3://mariamup/nice/lambda_virus/lambda_virus.fa.pac to ./lambda_virus.fa.pac
[hadoop@ip-172-31-2-103 efs]$ ls
lambda_virus.dict  lambda_virus.fa.amb  lambda_virus.fa.bwt  lambda_virus.fa.pac
lambda_virus.fa    lambda_virus.fa.ann  lambda_virus.fa.fai  lambda_virus.fa.sa

Then, I ran:

yarn jar BigBWA-2.1.jar com.github.bigbwa.BigBWA -D mapreduce.input.fileinputformat.split.minsize=123641127 -D mapreduce.input.fileinputformat.split.maxsize=123641127 -D mapreduce.map.memory.mb=7500 -m -p --index /home/hadoop/efs/lambda_virus -r ERR000589.fqBD ExitERR00058

Then, I got error:

Error: /mnt/yarn/usercache/hadoop/appcache/application_1517229999487_0003/container_1517229999487_0003_01_000002/tmp/libbwa9035204717035358356.so: /mnt/yarn/usercache/hadoop/appcache/application_1517229999487_0003/container_1517229999487_0003_01_000002/tmp/libbwa9035204717035358356.so: invalid ELF header (Possible cause: endianness mismatch)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

Is BigBWA requires 32-bit ? because my machines are 64-bit

jmabuin commented 5 years ago

Please, check your application logs to find the error:

yarn logs -applicationId application_1517157509312_0002

Where application_1517157509312_0002 must be replaced with the failed application ID

Maryom commented 5 years ago

I checked here are some of the error:

[fclose] No such file or directory

[Java_com_github_sparkbwa_BwaJni_bwa_1jni] Error saving stdout.

19/03/10 16:54:47 INFO DAGScheduler: failed: Set()

Beooz commented 5 years ago

hi,I have some problems. The first one is : 19/03/28 13:53:47 INFO mapreduce.Job: map 100% reduce 0% 19/03/28 13:53:51 INFO mapreduce.Job: map 100% reduce 100% 19/03/28 13:53:51 INFO mapreduce.Job: Job job_1552831617942_0008 completed successfully it said taht it had completed successfull but there
Map input records=131250 Map output records=1 Map output bytes=30 Map output materialized bytes=38 Input split bytes=108 Combine input records=0 Combine output records=0 Reduce input groups=1 Reduce shuffle bytes=38 Reduce input records=1 Reduce output records=0 it didn't have any output. I wad used two smaller fastq filess. then I through the files on HDFS: hdfs dfs -ls /user/root/ExitERR000589/* -rw-r--r-- 1 root supergroup 4919873 2019-03-28 13:53 /user/root/ExitERR000589/Input0_1.fq -rw-r--r-- 1 root supergroup 4921106 2019-03-28 13:53 /user/root/ExitERR000589/Input0_2.fq -rw-r--r-- 1 root supergroup 0 2019-03-28 13:53 /user/root/ExitERR000589/_SUCCESS -rw-r--r-- 1 root supergroup 0 2019-03-28 13:53 /user/root/ExitERR000589/part-r-00000 so, It didn't have any output. But I don't know that where were the Input_1.fq and Input_1.fq files come from? I didn't do it.

The seconde one is about "--index". I don't know what function about this arguement have. I change this arguement's value then I get a same result. And it export so much exceptions about java.lang.ArrayIndexOutOfBoundsExceptio. But I run it at another computer it didn't appear,