card4l does "data standardization" as one giant task but I think this breakdown is easier to understand
it looks like a pipeline now
let's us profile individual steps more effectively
let's us track dependencies better
better still would be to track which HDF5 groups are prerequisites to the tasks
down the track maybe they can be separate luigi tasks, bringing back most of the advantages of multifile_workflow but with a level of granularity I am more comfortable with
Changes:
not much, mostly moving code around
I think the dependence on the NBAR/SBT bands are clearer now
the esun_values were calculated as a side-effect to interpolating modtran parameters, which is probably not ideal, so refactored it to a separate function
have separated the interpolation and the reflectance phases
Tests:
tested on a couple of sample scenes/granules, the refactoring did not introduce any changes
have learned that for rough areas terrain_shadow_masks can be pretty expensive (luckily they are rare)
for Sentinel-2, modtran and interpolation takes up a much bigger fraction of time compared to Landsat (most probably because more bands)
Why?
card4ldoes "data standardization" as one giant task but I think this breakdown is easier to understandluigitasks, bringing back most of the advantages ofmultifile_workflowbut with a level of granularity I am more comfortable withChanges:
esun_valueswere calculated as a side-effect to interpolating modtran parameters, which is probably not ideal, so refactored it to a separate functionTests:
terrain_shadow_maskscan be pretty expensive (luckily they are rare)