search

aai-institute / lakefs-spec

An fsspec implementation for the lakeFS project.
http://lakefs-spec.org/
Apache License 2.0
37 stars 4 forks source link

Async support? #279

Open ion-elgreco opened 2 months ago

ion-elgreco commented 2 months ago

What is the motivation and/or use case?

Is async supported in lakefs spec?

How can we implement this feature?

No response

AdrianoKF commented 2 months ago

Thanks for raising this issue, @ion-elgreco!

Currently, there is no async support in lakeFS-spec, since the lakeFS client does not support it either (only for import operations, which are outside of the scope of lakeFS-spec). We might consider this feature in the future depending on the overall demand.

Could you elaborate a bit on your use case?

ion-elgreco commented 2 months ago

@AdrianoKF I need to read a couple of hundred thousands text files and parse them with regex. Doing this asynchronous is much faster.

I have currently written it with s3fs and lakefs S3 gateway. Seeing 15-20x performance increase versus sequential reads with lakefs_spec