search

treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data
https://docs.lakefs.io
Apache License 2.0
4.46k stars 359 forks source link

BUG: Discovery: data partitions: xid bug creates xid objects with wrong time #8330

Open N-o-Z opened 3 weeks ago

N-o-Z commented 3 weeks ago

We've discovered that the way xid creates ids with time is causing uint wrap around.

See more information here Although we opened the issue, I don't think it's going to be fixed in the near future (repository is not really being maintained)

We need to do some research and understand the implications of this bug on our data structure and how to mitigate them:

  1. Are partitions really being created in descending order - if so can we identify when that will stop being the case
  2. What are the implications of upgrading to a version with a fix / a different package that does not have this issue
  3. Currently Unified GC is working without optimizations so this issue does not affect it ATM. We need to think about how to approach this once we will need to implement the cutoff optimizations.