Analyze best way for DAG steps to share data

[tloubrieu-jpl]

💡 Description

Options are EFS, S3, other databases

That could managed as:

• a constraint on the DAG steps implementation, e.g. must support S3 to read/write interfaces
• generic DAG's component running during the execution and made available, somehow to the other components
• specific DAG's running environments providing these specific component interfaces to the other components
• ...

A good place to start is to get inputs from PODAAC on how they do that in Cumulus.

[tloubrieu-jpl]

airflow has a feature called excom to share data between steps. @ramesh-maddegoda is investigating that.

[ramesh-maddegoda]

Evaluated the following options to share data between Airflow tasks:

1. EFS
2. S3
3. Databases
4. Airflow XComs (https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html)

EFS:

Pros:

• It is really easy to integrate EFS with ECS Tasks, because we can introduce the access points of EFS as volumes in an ECS Task definition. This helps to use PDS Docker images as it is without making any Nucleus specific changes.
• Higher performance compared to S3.

Cons:

• EFS is a higher cost option compared to storage options such as S3.

S3:

Pros:

• S3 is a lower cost option compared to EFS and Databases.

Cons:

• It is required to use a library like s3fs (3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem) in order for a Docker container to write/read files to/from an S3 bucket. That means we have to modify PDS Docker images to support S3.
• Lower performance compared to EFS and Databases.

Databases:

Pros: Higher performance compared to S3. Other features of databases such as transaction management, queries and security can be easily utilized.

Cons:

• EFS is a higher cost option compared to storage options such as S3.
• It is required to modify PDS Docker images to support database read/writes.

Airflow XComs:

Pros:

• XComs is an inbuilt feature of Airflow and can be easily integrated with DAGs.
• There is no additional cost to use XCom.

Cons:

• XComs are not recommended for large amounts of data.

Recommendations:

Considering the above pros and cons of each option, we can make following recommendations:

• Especially considering the lower cost, S3 can be used as a way to share data between tasks in an Airflow DAG.
• The Docker images of PDS components should be modified in a way to make the support for S3 optional. This can be achieved by passing additional parameters to each docker container and only use s3fs library to connect to S3 conditionally (The docker images can still be used without S3, if there are not S3 relate parameters being passed).
• To avoid overwriting data in S3 (when same DAG is executed multiple times in parallel), the data will be save in a directory uniquely named with the a combination of Airflow variables such as dag_id, job_id, run_id etc. (E.g: https://airflow.apache.org/docs/apache-airflow/1.10.4/_api/airflow/models/dagrun/index.html)
• Airflow XComs can still be used to pass above unique directory names between tasks.

References:

• Airflow Models (including variables) https://airflow.apache.org/docs/apache-airflow/1.10.4/_api/airflow/models/index.html
• Airflow XComs: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html
• Amazon S3 Pricing: https://aws.amazon.com/s3/pricing/
• Amazon Dynamo DB Pricing: https://aws.amazon.com/dynamodb/pricing/on-demand/
• Amazon RDS Pricing: https://aws.amazon.com/rds/pricing/
• S3FS: https://linux.die.net/man/1/s3fs
• How to Install s3fs to access s3 bucket from Docker container: https://maxcotec.com/learning/how-to-install-s3fs-to-access-s3-from-docker-container/

[ramesh-maddegoda]

Additional note on Cumulus:

Had a chat with a Cumulus user and got to know that they use S3 buckets to share data between tasks in a workflow. At the end of each task, the data is uploaded to an S# bucket and then the next task downloads data from the same S3 bucket.

[tloubrieu-jpl]

Agreed during the breakout to move forward with the S3fs solution.