treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data
https://docs.lakefs.io
Apache License 2.0
4.46k stars 359 forks source link

[discovery] Split lakeFS packages with controllable dependencies #8362

Open arielshaqed opened 1 week ago

arielshaqed commented 1 week ago

Modify lakeFS to allow using individual components in a controlled disciplined manner. The goal is to allow other these components to be used in external projects. On a small scale, think of logging, cache, maybe some config helpers. On a larger scale, think of extracting KV.

arielshaqed commented 1 week ago

One way to do this might be to make it a multi-module workspace. Here's a brief tutorial on multi-module workspaces. In brief, this would create smaller modules which would have smaller dependencies - and no circular dependencies! As a first step, we might create a github.com/treeverse/lakefs/util that has no lakeFS dependencies, and includes logging, maybe some config helpers, cache, maybe testutil. #7616 was an earlier attempt to do this.

itaiad200 commented 1 week ago

Image