As a developer, I'd like to have access to the results as soon as ServiceX has put them in minio.
Assumptions
The query/request to ServiceX is the same as if waiting for a a list of files, or an awkward array rendering of those files.
The information will be delivered asynchronously, as it is returned from ServiceX.
Caching of the results from ServiceX will not occur - no need to download locally in this case.
Caching of the list of files won't occur. This is incase ServiceX is restarted and the results are no longer available.
Approach
Add a new async iterator return. It will return:
The minio bucket net
The filename in that bucket
The URL to give one HTTP access (valid for some sort amount of time).
To surface this, split the _get_files_from_servicex method in two, before and after the download is triggered. Only the first half is needed (and should work on its own).
Notes
The immediate goal is to be able to use this in funcX - as triggered by @BenGalewsky 's work.
Work Plan
[x] Write Code & Tests
[x] Make sure the test script works
[x] Write Docs
[x] Suppresss download marching ants when this is called
There is an issue with some versions of python 3.8 - it causes an exception on shutdown - but only on windows, apparently. Not clear how big a problem this is...
As a developer, I'd like to have access to the results as soon as ServiceX has put them in
minio
.Assumptions
Approach
Add a new
async
iterator return. It will return:minio
bucket netTo surface this, split the
_get_files_from_servicex
method in two, before and after the download is triggered. Only the first half is needed (and should work on its own).Notes
The immediate goal is to be able to use this in
funcX
- as triggered by @BenGalewsky 's work.Work Plan