os.environ["P1__BUCKET_URL"] = "./data"
p1 = dlt.pipeline("p1", destination="filesystem")
p1.run([1,2,3], table_name="dave")
# this will fail
p1.sql_client()
The run command will work, as the resolving of the bucket_url correctly searches for the key "P1__BUCKET_URL". When accessing the destination_client directly outside of a load, it does not seem to have the pipeline name segment in the lookup path. It fails with:
File "/Volumes/dlthub/dlt/test_pipe.py", line 13, in <module>
p1.destination_client()
File "/Volumes/dlthub/dlt/dlt/pipeline/pipeline.py", line 1021, in destination_client
return self._get_destination_clients(schema)[0]
File "/Volumes/dlthub/dlt/dlt/pipeline/pipeline.py", line 1244, in _get_destination_clients
client = self.destination.client(schema, initial_config)
File "/Volumes/dlthub/dlt/dlt/common/destination/reference.py", line 750, in client
config = self.configuration(initial_config)
File "/Volumes/dlthub/dlt/dlt/common/destination/reference.py", line 736, in configuration
config = resolve_configuration(
File "/Volumes/dlthub/dlt/dlt/common/configuration/resolve.py", line 67, in resolve_configuration
return _resolve_configuration(config, sections, (), explicit_value, accept_partial)
File "/Volumes/dlthub/dlt/dlt/common/configuration/resolve.py", line 164, in _resolve_configuration
_resolve_config_fields(
File "/Volumes/dlthub/dlt/dlt/common/configuration/resolve.py", line 299, in _resolve_config_fields
raise ConfigFieldMissingException(type(config).__name__, unresolved_fields)
dlt.common.configuration.exceptions.ConfigFieldMissingException: Following fields are missing: ['bucket_url'] in configuration with spec FilesystemDestinationClientConfiguration
for field "bucket_url" config providers and keys were tried in following order:
In Environment Variables key DESTINATION__FILESYSTEM__BUCKET_URL was not found.
In Environment Variables key DESTINATION__BUCKET_URL was not found.
In Environment Variables key BUCKET_URL was not found.
In secrets.toml key destination.filesystem.bucket_url was not found.
In secrets.toml key destination.bucket_url was not found.
In secrets.toml key bucket_url was not found.
In Airflow Secrets TOML Provider key destination.filesystem.bucket_url was not found.
In Airflow Secrets TOML Provider key destination.bucket_url was not found.
In Airflow Secrets TOML Provider key bucket_url was not found.
Please refer to https://dlthub.com/docs/general-usage/credentials for more information
Given the folllowing pipeline:
The run command will work, as the resolving of the bucket_url correctly searches for the key "P1__BUCKET_URL". When accessing the destination_client directly outside of a load, it does not seem to have the pipeline name segment in the lookup path. It fails with: