I use s3 storage in xcom backend and it Error on xcom (Invalid Key), I find it because In _is_relative_to(ObjectStoragePath(data), p) return False because p from base_path that contain conn_id@... but data from the xcom (str(base_path.joinpath(dag_id, run_id, task_id, f"{uuid.uuid4()}{suffix}")) ) that did't contain conn_id anymore. The _is_relative_to check storage_options(conn_id) that make it return False.
What you think should happen instead?
It should return True because data come from p (serialize_value and deserialize_value on XCOM).
How to reproduce
Use XComObjectStorageBackend in xcombackend. And xcom_objectstorage_path = s3://conn_id@airflow-test/ with conn_id = amazon web services, endpoint = "https://example.com".
Operating System
docker airflow slim-2.9.1-python3.9 run on Ubuntu 22.04.4 LTS host.
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
Apache Airflow version
2.9.1
If "Other Airflow 2 version" selected, which one?
No response
What happened?
I use s3 storage in xcom backend and it Error on xcom (
Invalid Key
), I find it because In_is_relative_to(ObjectStoragePath(data), p)
return False because p frombase_path
that containconn_id@...
butdata
from the xcom (str(base_path.joinpath(dag_id, run_id, task_id, f"{uuid.uuid4()}{suffix}"))
) that did't contain conn_id anymore. The_is_relative_to
checkstorage_options
(conn_id
) that make it return False.What you think should happen instead?
It should return True because
data
come fromp
(serialize_value and deserialize_value on XCOM).How to reproduce
Use
XComObjectStorageBackend
inxcombackend
. Andxcom_objectstorage_path = s3://conn_id@airflow-test/
withconn_id
=amazon web services, endpoint = "https://example.com"
.Operating System
docker airflow slim-2.9.1-python3.9 run on Ubuntu 22.04.4 LTS host.
Versions of Apache Airflow Providers
apache-airflow-providers-common-io 1.3.1
apache-airflow-providers-amazon 8.19.0
Deployment
Docker-Compose
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct