search

treeverse / lakeFS

lakeFS - Data version control for your data lake | Git for data
https://docs.lakefs.io
Apache License 2.0
4.46k stars 359 forks source link

Add a lakeFS / XID time slug decoder #8323

Closed arielshaqed closed 3 days ago

arielshaqed commented 3 weeks ago

Add a lakeFS / XID time slug decoder

Decodes object names nicely, e.g.

❯ lakectl -c fs --pre-sign=false stat lakefs://repo/main/50m
Path: 50m
Modified Time: 2024-10-28 17:52:50 +0200 IST
Size: 52428800 bytes
Human Size: 52.4 MB
Physical Address: s3://treeverse-ariels-test-us/repos/s3-dev-repo/data/gcnl78vqmpl9eb7rqlr0/csfr6kfqmpl9eb7rqlt0,jMHnLFOwK2RtCVV_aE0iClDtfP7ehfORTH_siX9V5e8
Checksum: 4d98d5ac224a2a14a2b7efac5d100757-10
Content-Type: application/octet-stream

❯ go run ./contrib/decoder/decode_partition/ csfr6kfqmpl9eb7rqlt0
2024-10-28 15:52:49 +0000 UTC (asc)
3945-03-05 08:07:11 +0000 UTC (desc)

But does not decode the partition name ("desc"); not sure why not.

❯ go run ./contrib/decoder/decode_partition/ gcnl78vqmpl9eb7rqlr0
2039-09-29 13:48:19 +0000 UTC (asc)
3930-04-04 10:11:41 +0000 UTC (desc)

(Happy to modify if reviewer can explain this!)

github-actions[bot] commented 3 weeks ago

E2E Test Results - Quickstart

11 passed
github-actions[bot] commented 3 weeks ago

E2E Test Results - DynamoDB Local - Local Block Adapter

13 passed
N-o-Z commented 3 weeks ago

Add a lakeFS / XID time slug decoder

Decodes object names nicely, e.g.

❯ lakectl -c fs --pre-sign=false stat lakefs://repo/main/50m
Path: 50m
Modified Time: 2024-10-28 17:52:50 +0200 IST
Size: 52428800 bytes
Human Size: 52.4 MB
Physical Address: s3://treeverse-ariels-test-us/repos/s3-dev-repo/data/gcnl78vqmpl9eb7rqlr0/csfr6kfqmpl9eb7rqlt0,jMHnLFOwK2RtCVV_aE0iClDtfP7ehfORTH_siX9V5e8
Checksum: 4d98d5ac224a2a14a2b7efac5d100757-10
Content-Type: application/octet-stream

❯ go run ./contrib/decoder/decode_partition/ csfr6kfqmpl9eb7rqlt0
2024-10-28 15:52:49 +0000 UTC (asc)
3945-03-05 08:07:11 +0000 UTC (desc)

But does not decode the partition name ("desc"); not sure why not.

❯ go run ./contrib/decoder/decode_partition/ gcnl78vqmpl9eb7rqlr0
2039-09-29 13:48:19 +0000 UTC (asc)
3930-04-04 10:11:41 +0000 UTC (desc)

(Happy to modify if reviewer can explain this!)

Bug in xid!?!?! 😲😲😲😲 They use uint32 when dealing with golang Time objects instead of uint64 😢 This has major implications on us - we need to understand how to resolve this...

N-o-Z commented 2 weeks ago

@arielshaqed what would you like to do with this? I can approve it but given what we found this won't be very useful for partitions

N-o-Z commented 3 days ago

@arielshaqed closing - please reopen if relevant