treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data
https://docs.lakefs.io
Apache License 2.0
4.47k stars 359 forks source link

Import truncates keys, resulting in empty keys that yields error #3588

Closed Jonathan-Rosenberg closed 1 year ago

Jonathan-Rosenberg commented 2 years ago

If the given URL to import from is itself a file, e.g. s3://bucket/some/path/to/key/ -> file, lakeFS will truncate the given URL and will iterate over the keys past it (including it), so it will also try to store the empty key but will fail since pebble does not allow empty keys. The returned error looks like this: writing range from entry iterator: writing record: setting key and value: pebble: keys must be added in order: #0,DEL, #0,SET

itaiad200 commented 2 years ago

Yap - lakeFS like S3 doesn't accept empty keys. We should prettify this error - but there's not much we can do given that replacing that key is an awkward ux.

nopcoder commented 1 year ago

Note that this issue occurs when you import a directory marker (empty name) to the root branch - an empty on the branch level fails.