search

geotrellis / geotrellis-chatta-demo

Demo of GeoTrellis - weighted overlay and zonal summary for University of Tennessee at Chattanooga.
50 stars 34 forks source link

ETL does not parallelize tasks #26

Open marfago opened 7 years ago

marfago commented 7 years ago

Hi,

I'm not sure this is a problem with geotrellis or spark or my configuration. Starting from geodocker-cluster and with some upgrade of the actual chattademo project (basically a porting to scalal 2.11) I was able to run the chatta demo. What I noticed is that, even using multiple spark workers, all the tasks are always executed sequentially and not in parallel. Is there any parameters to tune up to increase parallelism?

pomadchin commented 7 years ago

Hi @marfago, interesting, how can it be executed sequentially across executors? o: Can you attach your spark-submit command and screenshot of spark web ui with jobs, executors and tasks? Btw, we can discuss it in our gitter channel :)

marfago commented 7 years ago

hi @pomadchin , find attached the PNGs.

Let me elaborate a little bit. I have the phisical servers, SN and FN: SN hosts spark master, spark worker, accumulo and hdfs name and data while FN hosts just a spark worker. Both node are in a docker network. I have also slightly changed the demo in order to ingest 10 raster images and the mask.

I would expect the ETL to fully allocate all CPUs, but both servers are quite idle and, in spark UI, I can just see one task at time running on one of the servers.

Any suggestions?

ingest.txt fn-load sn-load sparkexecutors sparkjobs sparkui