ebyhr
commented
7 months ago cc: @findinpath @wendigo @raunaqmorarka
Closed sdaberdaku closed 1 month ago
ebyhr
commented
7 months ago cc: @findinpath @wendigo @raunaqmorarka
sdaberdaku
commented
7 months ago The problem persists also in Trino 445. We have two use cases where this issue emerges: In the first case we overwrite a Delta Table with Spark and Trino cannot read it any more. The issue is resolved if we restart the coordinator. In the second case we overwrite a Delta Table with DBT through Trino itself. Again, by restarting the coordinator the tables become readable.
jojovem
commented
3 months ago Hi all.
I´m having the same issue. Everytime i get these errors i just delete the cache data directly in the disk and it solves my problem.
My stack is: Trino (450) + DeltaLake Connector + AWS Glue
cache configuration for the catalog is pretty straighforward:
fs.cache.enabled=true
fs.cache.directories=/tmp/trino-cache/catalog
fs.cache.max-sizes=490GBWhen i get the snapshot errors now and then, i just do a
rm -rf /tmp/trino-cache/catalog
from within the coordinator pod and it works again
The disk is an EBS volume
raunaqmorarka
commented
3 months ago cc: @jkylling @Pluies
jkylling
commented
3 months ago Use cases where we can no longer assume that the commit and checkpoint files within the Delta log are immutable should likely just disable the cache on the coordinator. Having mutable commit and checkpoint files is not really addressed by the Delta protocol, and makes any concurrent access of Delta tables brittle. So it's better to avoid this, and use a unique locations every time a table is created.
sdaberdaku
commented
3 months ago Hello @jkylling,
I am definitely using the same unique and deterministic location for a given delta table (which simply depends on the delta table name). Maybe I misunderstood your response, could you please elaborate a bit more? Also, is it possible to disable caching only on the coordinator and keep it enabled on workers?
jkylling
commented
3 months ago Hello @jkylling,
I am definitely using the same unique and deterministic location for a given delta table (which simply depends on the delta table name). Maybe I misunderstood your response, could you please elaborate a bit more? Also, is it possible to disable caching only on the coordinator and keep it enabled on workers?
Sorry, I might be confusing this with another issue. Are you able to share a bit more about your environment? Which object store are you using? You mention that this happens when overwriting a table. What operation do you run on Spark or Trino when you do an overwrite? Can it ever happen that the content of a commit at path/to/my/table/_delta_log/00...00.json is different if I check it at different times?
Unfortunately, it looks like disabling the cache on the coordinator only is no longer possible.
sdaberdaku
commented
3 months ago I am running Trino in AWS EKS using EC2 nodes with NVMe support where I am mounting the cache folders (ec2 nodes are provisioned by Karpenter, and the NVMe disks are automatically mounted as RAID0 ephemeral storage).
The object store is S3 and I am using Glue as a Data Catalog.
With Spark I overwite the tables like so:
(df
.write
.format("delta")
.mode("overwrite")
.option("overwriteSchema", "true")
.save(path))Regarding the the content of a commit I really have no idea.
jojovem
commented
3 months ago @jkylling It seems that the stack @sdaberdaku is using is pretty similar to mine. In my case, this problem usually happens when i delete the entire s3 bucket (including the _delta_log) to start over with a new table. I also delete the table on aws glue and re-run the catalog crawler.
It seems the cache seems to think the previous created table is still valid, so it seems to keep searching for the snapshot that do not exist anymore. Maybe it should flush the cache when this kind of error happens?
I also tried with:
CALL system.flush_metadata_cache();
But it doesnt work, i have to manually delete the cache to make it work again.
jkylling
commented
3 months ago Discussed this issue with @raunaqmorarka and we will add a configuration option to disable caching of files with /_delta_log/ in their path to avoid issues with Delta tables with mutable commits.
That said, having mutable files within /_delta_log/ is an anti-pattern which is likely to bite you in other subtle ways. It's better to avoid it, for instance by adding a UUID to the location of every table, or by deleting a table while preserving its history and data.
sdaberdaku
commented
2 months ago Hello @jkylling, Any news on the configuration option to disable delta log file caching?
Thanks,
Sebastian
jkylling
commented
2 months ago Hi @sdaberdaku,
No one has had time to look into this yet. If you want to contribute this yourself I'd be happy to give some guidance.
Basically, a new https://github.com/trinodb/trino/blob/c8568a9ccfcf2876ef441588a6040270d82f95b2/plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/cache/DeltaLakeCacheKeyProvider.java must be added which returns an empty cache key for any file within /_delta_log/. Enabling of this CacheKeyProvider would be controlled by some new configuration option defined within https://github.com/trinodb/trino/blob/c8568a9ccfcf2876ef441588a6040270d82f95b2/plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeConfig.java#L45, and used to select the provider at https://github.com/trinodb/trino/blob/c8568a9ccfcf2876ef441588a6040270d82f95b2/plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeModule.java#L154
Pluies
commented
2 months ago Alternatively, you can reuse code from closed PR https://github.com/trinodb/trino/pull/21131 and use skip-paths to skip over file paths containing /_delta_log 👍
sdaberdaku
commented
2 months ago Thank you @jkylling and @Pluies for the valuable suggestions! I will look into this and try my best to submit a PR!
sdaberdaku
commented
2 months ago Hello @jkylling,
I implemented your suggestions and submitted a PR (I also submitted the cla). Would you be willing to be a reviewer?
Best,
Sebastian
In Trino 444 with Alluxio cache enabled, when dropping and then recreating a delta table, I occasionally get the following error:
Unfortunately the error is not easy to replicate, I can drop and recreate the same table without issues multiple times.
The actual delta log files in the object storage are not corrupted, they can be read by Spark and by Trino if the coordinator is restarted.