background:
I am a totally new to hadoop and spark
I run the program csbwamem in CDH 5.5 quick start virtual machine, equipped with 8 cpu cores and 50G memory. ( for the physical machine, 16 physical cores (32 logical cores) and 64G memory)
For data, hg19 human genome, with original parameter-setting bwa program for indexing, a PE raw read data of 1.6G per file.
For the executing command:
spark-submit --driver-memory 16g --executor-memory 2g --master yarn-client cloude-scale-bwamem-0.2.2-assembly.jar cs-bwamem -oPath ~/data/out 1 ~/data/hg19bwaidx ~/datahg19
when the program was running, I got a warning, which is:
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
It seems that the warning was repeating for the whole night and the program did not process any further.
I just wondering how much resource should I assigned to the program?
background: I am a totally new to hadoop and spark
I run the program csbwamem in CDH 5.5 quick start virtual machine, equipped with 8 cpu cores and 50G memory. ( for the physical machine, 16 physical cores (32 logical cores) and 64G memory) For data, hg19 human genome, with original parameter-setting bwa program for indexing, a PE raw read data of 1.6G per file. For the executing command: spark-submit --driver-memory 16g --executor-memory 2g --master yarn-client cloude-scale-bwamem-0.2.2-assembly.jar cs-bwamem -oPath ~/data/out 1 ~/data/hg19bwaidx ~/datahg19
when the program was running, I got a warning, which is: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources It seems that the warning was repeating for the whole night and the program did not process any further.
I just wondering how much resource should I assigned to the program?