search

treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data
https://docs.lakefs.io
Apache License 2.0
4.48k stars 359 forks source link

Allow incremental creation of symlinks #2762

Closed johnnyaug closed 1 year ago

johnnyaug commented 3 years ago

The lakectl metastore create-symlink takes table data stored in lakeFS, and creates a Symlink type table pointing at the data in order for it to be queryable in Athena.

Right now, every time you run the command - it will write the table all over again.

After creating the table for the first time, new partitions may be created in my table. Allow me to run the command again only on specific partitions, to add those partitions to the existing table.

johnnyaug commented 1 year ago

Closing since metastore operations are not really maintained anymore. We can reopen in the future if the need arises.