Open ion-elgreco opened 4 months ago
I think you are printing the default values, not the configured values
I think you are printing the default values, not the configured values
@danielgafni with pydantic V1 it will show all the configured values.
With pydantic V2 it will only show the env vars, but I didn't add that in the example above.
Also it's just a way to show that non env vars disappear. You can see it as well in the UI
We suffer the same problem in our pipeline. Even though we pass the "aws_endpoint_url" storage_options to the io_manager, it overwrittes it as an empty dict and in the scan_parquet, it pass an empty dict as storage_options which does not use our desired "aws_endpoint_url"
The issue seems to be located in the ConfigurableResourceFactory
.
I can see that the values passed to the config have been modified between data_without_resources
and casted_data_without_resources
.
>>> data_without_resources
{'base_dir': 's3://my-bucket', 'storage_options': {'aws_access_key_id': 'minioadmin', 'aws_secret_access_key': 'minioadmin', 'aws_endpoint_url': 'http://localhost:32808'}}
>>> casted_data_without_resources
{'base_dir': 's3://my-bucket', 'storage_options': {}}
I believe that the pydantic aliases are not correctly taken into account.
Dagster version
1.7.1
What's the issue?
We noticed that values that we are passing dictionaries with non-env vars through a resource (PolarsParquetIOManager), causes them to get lost once they go through the Definitions object. Before that they are still there. See below:
Field(<dagster._config.field_utils.Shape object at 0x7fea437933a0>, default={'extension': '.parquet', 'base_dir': None, 'storage_options': {}}, is_required=False)
Fyi @danielgafni
What did you expect to happen?
No response
How to reproduce?
No response
Deployment type
None
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.