Is your feature request related to a problem? Please describe.
The watcher service can watch multiple source locations and it's used as a dependency in the download activity to get the package from the watcher location that triggered the processing workflow. There are cases where the watcher is a filesystem watcher and downloading may not be needed, and I'm looking at options to run preprocessing child workflows remotely where the initial download would happen there and the preprocessing result (zipped bag) would be shared using a different bucket.
Describe the solution you'd like
I'd like a download activity that only relies on bucket(s), ideally in the temporal-activities repository, where it could be shared with those preprocessing child workflows.
Describe alternatives you've considered
Keep the current download activity while adding a new one as described in the solution above. Run them (or not) based on the preprocessing configuration.
jraddaoui
commented
7 months ago
Is your feature request related to a problem? Please describe.
The watcher service can watch multiple source locations and it's used as a dependency in the download activity to get the package from the watcher location that triggered the processing workflow. There are cases where the watcher is a filesystem watcher and downloading may not be needed, and I'm looking at options to run preprocessing child workflows remotely where the initial download would happen there and the preprocessing result (zipped bag) would be shared using a different bucket.
Describe the solution you'd like
I'd like a download activity that only relies on bucket(s), ideally in the temporal-activities repository, where it could be shared with those preprocessing child workflows.
Describe alternatives you've considered
Keep the current download activity while adding a new one as described in the solution above. Run them (or not) based on the preprocessing configuration.
Additional context
None.