dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
10.69k stars 1.33k forks source link

dagster-deltalake S3Config allow_unsafe_rename parameter ignored #19889

Open plaanv opened 4 months ago

plaanv commented 4 months ago

https://github.com/dagster-io/dagster/blob/190175d2a795a212de95e9a7a1b6453e4f8f4743/python_modules/libraries/dagster-deltalake/dagster_deltalake/config.py#L114C5-L114C24

What happens is this parameter is ignored because it is discarded by both :

It can still be passed as an env var with AWS_S3_ALLOW_UNSAFE_RENAME as a workaround.

ion-elgreco commented 4 months ago

I am addressing that in a PR I've opened