jdries
commented
1 month ago Made a few improvements:
- masking was doing reproject twice, causing same tasks to recompute, even running udf 2 times
- source name partitioning now computed in driver, seems to work
- apply_dimension target='bands' was refactored to use the generic transformTimeDimension, which does better detection of the partitioner on incoming cube, avoiding needless repartitioning
- pixel wise mask performed a join that entirely removed the partitioner
Detected other things:
- scala number of threads is set globally to 1, to avoid reading errors with gdal, but now other operators for feature computation no longer have parallelism, added another way to disable use of threads for reading
Need to check:
- does reading angles still work as expected? (Done, seems fine!)
- on production, the stage before actual reading data took a lot of time, check if this is still the case after redeploy
- masking in load_collection: what if the mask has keys outside the valid bounds of partitioner?
To partition the rdd by source name, the keys are retrieved from the RDD, but this triggers full computation, which is even partially repeated later on, because not all stages are reusable. In a particular job, computing those distinct keys seems to take a full cpu hour (across all tasks).
Extra benefit of avoiding this is fewer spark jobs and stages, so simplifying the UI a bit.