Creating DeltaTable object slow

[braaannigan]

Environment

Delta-rs version: 0.17.4

Binding: python

Environment:

• Cloud provider: AWS
• OS:MacOS
• Other:

---

Bug

What happened: I have a DeltaTable on S3 partitioned by date with about 60 dates. The partitions have been compacted and vacuumed so have one file each. I append to this table 100 times a day so the transaction log has about 6000 json files.

When I try to create the DeltaTable object it takes 30 seconds.

What you expected to happen: I expected this operation to be faster but I'm not sure if that's a reasonable expectation?

How to reproduce it: No repro example, I'm just trying to establish if there is something unusual here

More details:

[ion-elgreco]

Did you checkpoint?

[braaannigan]

Did you checkpoint?

Haven't come across that before, how do I do it?

[ion-elgreco]

With the latest version it should automatically checkpount every 100 commits, but you can also manually do it by doing DeltaTable.create_checkpoint()

[PeterKeDer]

We're also seeing a similar issue where constructing the DeltaTable takes 20-30 seconds, which is longer than we expected.

For context, we're using version 0.17.1. Our table is on AWS S3 and has 20000 transaction logs. We have a very recent checkpoint (only 5 versions from the latest). The parquet is 8 MB in size and has 20000 rows.

I traced through the performance with some custom debug logs. Here's the operations that takes the most time:

• The commit jsons and checkpoint parquet are loaded twice during the instantiation
  • Is there a reason we're loading them two separate times? If we remove one, we could save 5-10s
• Replaying the transactions takes up to 10s
  • Are these performance numbers expected? I'm not too familiar with the exact operations, but this feels slow considering there's only 20000 rows

Edit: disregard these numbers, we were running on debug mode 😅

[braaannigan]

@ion-elgreco Shall I make a PR to document checkpointing a bit more?

[rtyler]

@braaannigan improving our documentation is always welcome! I'm going to close this in the meantime