ssl-hep / ServiceX_frontend

Client access library for ServiceX
https://servicex-frontend.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
5 stars 11 forks source link

Return the minio list of files async #132

Closed gordonwatts closed 3 years ago

gordonwatts commented 4 years ago

As a developer, I'd like to have access to the results as soon as ServiceX has put them in minio.

Assumptions

  1. The query/request to ServiceX is the same as if waiting for a a list of files, or an awkward array rendering of those files.
  2. The information will be delivered asynchronously, as it is returned from ServiceX.
  3. Caching of the results from ServiceX will not occur - no need to download locally in this case.
  4. Caching of the list of files won't occur. This is incase ServiceX is restarted and the results are no longer available.

Approach

Add a new async iterator return. It will return:

To surface this, split the _get_files_from_servicex method in two, before and after the download is triggered. Only the first half is needed (and should work on its own).

Notes

The immediate goal is to be able to use this in funcX - as triggered by @BenGalewsky 's work.

Work Plan

gordonwatts commented 3 years ago

There is an issue with some versions of python 3.8 - it causes an exception on shutdown - but only on windows, apparently. Not clear how big a problem this is...