apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
36.44k stars 14.11k forks source link

`XComObjectStorageBackend` can't `deserialize_value` on S3 path #39797

Closed Rmjp closed 4 months ago

Rmjp commented 4 months ago

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

I use s3 storage in xcom backend and it Error on xcom (Invalid Key), I find it because In _is_relative_to(ObjectStoragePath(data), p) return False because p from base_path that contain conn_id@... but data from the xcom (str(base_path.joinpath(dag_id, run_id, task_id, f"{uuid.uuid4()}{suffix}")) ) that did't contain conn_id anymore. The _is_relative_to check storage_options(conn_id) that make it return False.

What you think should happen instead?

It should return True because data come from p (serialize_value and deserialize_value on XCOM).

How to reproduce

Use XComObjectStorageBackend in xcombackend. And xcom_objectstorage_path = s3://conn_id@airflow-test/ with conn_id = amazon web services, endpoint = "https://example.com".

Operating System

docker airflow slim-2.9.1-python3.9 run on Ubuntu 22.04.4 LTS host.

Versions of Apache Airflow Providers

apache-airflow-providers-common-io 1.3.1 apache-airflow-providers-amazon 8.19.0

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 4 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Rmjp commented 4 months ago

Sorry, I didn't see #39602