search

sul-dlss-labs / rialto-airflow

Airflow for harvesting data for research intelligence and open access analysis
Apache License 2.0
1 stars 0 forks source link

Use JSON Lines #65

Closed edsu closed 2 months ago

edsu commented 2 months ago

This is an experiment in using JSON Lines for our harvested JSON data instead of CSV.

This commit moves our sul_pub, dimensions and openalex harvesters over to writing records as JSON Lines (jsonl). This means we will preserve the dictionary and list data structures we received from the APIs and will be able to use them for querying later, e.g. ORCID IDs embedded in author objects.

Reading in JSON-L can be achieved with Pandas like:

pandas.read_json('data.jsonl', orient='records', lines=True)

and Polars:

polars.read_ndjson('data.jsonl')

or lazily:

polars.scan_ndjson('data.jsonl')
edsu commented 2 months ago

This was a useful experiment to see how the structure of the harvested data could be preserved. But I don't think it's strictly necessary. Perhaps it's something to keep in mind for the future though?