search

pytroll / trollflow2

Next generation Trollflow. Trollflow is for batch-processing satellite data using Satpy
https://trollflow2.readthedocs.org/
GNU General Public License v3.0
10 stars 15 forks source link

When using use_tmp_file or staging_dir, move/rename files as soon as they are finished #129

Open gerritholl opened 2 years ago

gerritholl commented 2 years ago

Feature Request

Is your feature request related to a problem? Please describe.

I'm using use_tmp_file or staging_dir to prevent files being shipped out by our file distribution system prematurely. Currently, trollflow2 stores all files to temporary files, then renames/moves them when they are all finished. It would be desirable to move/rename each file as soon as it's finished, so that products are delivered to users sooner.

Describe the solution you'd like

I would like that (optionally?) files are renamed to their final name as soon as they're finished.

Describe any changes to existing user workflow

None if the behaviour is optional. But I think I'm the only user anyway.

Additional context

Make the production so fast that it doesn't matter ;-)

gerritholl commented 2 years ago

This may be easier said than done. The save_datasets plugins first collects all the dask graphs, then computes them all simultaneously. Maybe moving the file could be added as a task to the dask graph in some way?

gerritholl commented 2 years ago

May this also be relevant for the FilePublisher? I think that when using FilePublisher, it also only published when all products have been produced, losing the opportunity to ship out files as soon as they are finished?

gerritholl commented 2 years ago

Maybe https://github.com/pytroll/trollflow2/pull/138 can be of help here?