Design requirement: Assurance of Data Segregation between workflows

[rhosking]

Capturing the overarching design requirement, though work may be spread across a number of other issues. The requirement is around an single airflow environment, supporting multiple workflows which unlike the existing cases, which use shared buckets and a single GCP BigQuery project, to allow for DAGs/SubDags to support explicitly segregating data into multiple projects

Roughly speaking, the following should be easily supported when writing new Telescopes:

• DAGs/Telescopes to be easily configured to use custom buckets
• DAGs/Telescopes to be easily configured to put data into a specific GCP BigQuery project
• DAGs/Telescopes to be easily configured to use specific VMs

[rhosking]

Digging into the design goals a little further, motivated by a few use cases that we broadly should support. Consider the following situation:

There is an airflow instance running, as it stands it can support a wide number of DAGs running on a schedule. Collectively, these DAGs all currently use the same bucket, VM (if required) and GCP project for loading BigQuery Datasets. For many public datasets this is perfectly fine. However, this same Airflow instance might be required to fetch both public and private datasets, with the private datasets subject to a number of restrictions (both commercially, ethically and from a range of data protection legislation).

The requirement is for DAG/Telescope authors, to have a clearly defined set of mechanisms that support writing data ingestion workflows that can optionally:

• Define a custom bucket to segregation data from that workflow from other public datasets
• Define a custom GCP project in order to load data in BigQuery
• Specify a specific VM for which any data processing must take place (away from the processing of other public datasets)
• For data endpoints that support multiple private (such as requiring a login, and the credentials used for that login dictates the specific set of data able to be downloaded), a telescope implementation should allow for defining multiple set of credentials, which importantly can each define separate buckets/VMs/GCP projects for processing and storing the data returned.

The overarching purpose behind these requirements is to support data segregation, and thus the ability to define access controls across each set of private data independently.

EDIT: Additionally, it might be worth considering building in functionality to specify data locations for the buckets/VMs/BigQuery datasets. it get complicated if you are looking to join datasets across regions though, hence why we have been trying to work in a single region/multi-region thus far

[jdddog]

To enable this we would need to create a REST API and database to store all of the information specific to a particular telescope and workflow, which allows:

• Create a project: e.g. create a GCP project, add permissions for that GCP project to the observatory service account, create buckets, create BigQuery datasets.
• Read project data: to be used by specific DAGs, so that they know what buckets / projects to save data to etc.

We may also want to update to Airflow 2.0, so that we can use TaskGroups instead of SubDags, because they have a better UI and overcome some of the limitations of SubDags: https://airflow.apache.org/docs/apache-airflow/stable/concepts.html#taskgroup. We might for instance, want to have a TaskGroup for each project that gets processed, like what Tuan did with the WoS and Scopus telescopes.

[rhosking]

I think that sounds sensible, subject to working through the details. The other two things that need considering is the work around telescope templates (#326 ) and the prototype work that is being done for an API (#331 )

[rhosking]

The following telescopes are currently impacted by, and are waiting for decisions made on this issue:

• ONIX SFTP #289 @jdddog
• Google Books #247 @aroelo
• Google Analytics #249 @aroelo
• IRUS-UK/OAPEN #244 / #345 @aroelo
• JSTOR #246 @aroelo