Create a Scaffold for a Daily Airflow DAG for retrieving dynamic list of MARC BIB records from various vendors

sul-dlss / libsys-airflow

Airflow DAGS for migrating and managing ILS data into FOLIO along with other LibSys workflows

Apache License 2.0

5 stars 0 forks source link

Create a Scaffold for a Daily Airflow DAG for retrieving dynamic list of MARC BIB records from various vendors #247

Closed jermnelson closed 1 year ago

jermnelson commented 1 year ago

Create an Airflow DAG that runs daily and queries the Dataloader Management App and possibly FOLIO Organizations for a dynamic list of vendors that includes connection information (FTP, SFTP, username, passwords) and Data Import Profile for MARC BIB records along with specific pre- and post- processing tasks to be preformed on these records.

This ticket should trigger secondary DAGs for each vendor load.

Specific tasks to be accomplished will be ticketed separately.

jmartin-sul commented 1 year ago

this ticket would be complete if a skeleton DAG with the right placeholder tasks is created, with task implementation TBD based on dataloader management app API.

jmartin-sul commented 1 year ago

on a daily basis, queries the data management app.
gets back a list of vendors
vendors have a list of interfaces (connection info and creds for e.g. FTP to remote system for data load files to retrieve)
this app retrieves those interfaces
for each interface, trigger a DAG that loads data from that vendor.

@jermnelson will spawn tickets from this

jermnelson commented 1 year ago

For this ticket, just create placeholder Tasks using the EmptyOperator that can be replaced with PythonOperators once those tickets are implemented.

jermnelson commented 1 year ago

Here are the follow-up tickets: