search

aai-institute / lakefs-spec

An fsspec implementation for the lakeFS project
http://lakefs-spec.org/
Apache License 2.0
39 stars 4 forks source link

Integrate new `lakefs` SDK wrapper into the project #195

Closed nicholasjng closed 9 months ago

nicholasjng commented 10 months ago

The new lakefs SDK wrapper proposal is available in beta: https://pypi.org/project/lakefs/

Should therefore be installable via pip install --pre lakefs.

Even though it is not completely ready yet, we can see if there is functionality that can already supersede some of our APIs (I'm thinking the client helpers especially), and save our learnings in this thread to ease migration to the lakefs package later.

nicholasjng commented 10 months ago

One particularly interesting thing will be how to handle client autodiscovery: The wrapper APIs support omission of the client everywhere, which prompts an autodiscovery mechanism similarly to what we do now, but as a stateful session opened via init.

Will we use the "one file system, one session" approach or should we opt to reuse a session?

AdrianoKF commented 9 months ago

Update: the first stable releases, v0.1.0 and v0.1.1, have just landed on PyPI: https://pypi.org/project/lakefs/#history

nicholasjng commented 9 months ago

Closed by #231.