re-writes json intake driver and adds two collection catalogs

sul-dlss / dlme-airflow

This is a new repository to capture the work related to the DLME ETL Pipeline and establish airflow

Apache License 2.0

1 stars 0 forks source link

re-writes json intake driver and adds two collection catalogs #449

Closed jacobthill closed 9 months ago

jacobthill commented 10 months ago

The existing json driver was written for the Library of Congress data but it doesn't handle paging so it could not be used for that collection or any other. This driver handles paging by accepting a json path in the catolog to the next page or by iterating through the results and updating offset in the url.