project-lux / data-pipeline

Data pipeline to harvest, transform, reconcile, enrich and export Linked Art data for LUX (or other system)
Apache License 2.0
15 stars 1 forks source link

Add SNAC as a source #3

Closed azaroth42 closed 2 months ago

azaroth42 commented 8 months ago

We can get the links from wikidata (amongst other places) and there's interesting information in the records... but the permalink and the data export URI don't match up.

https://www.wikidata.org/wiki/Q727693 which has SNAC ARK ID https://snaccooperative.org/ark:/99166/w67d3fsz

But the JSON export is https://snaccooperative.org/download/8895729?type=constellation_json

So we need a way to resolve w67d3fsz into 8895729.

Thankfully there's an API ...https://snaccooperative.org/api_help

But it only supports PUT of JSON parameters, making the retrieval a little more complex.

SNAC openrefine endpoint mentioned here: https://portal.snaccooperative.org/node/489

azaroth42 commented 5 months ago

ahha!

https://snaccooperative.org/download?arkid=http://n2t.net/ark:/99166/w6jh3pk0&type=constellation_json :)

kkdavis14 commented 5 months ago

My config looks like this:

{
    "name": "snac",
    "type": "external",
    "namespace": "https://snaccooperative.org",
    "matches": ["http://n2t.net/ark:/99166/"],
    "fetch": "https://snaccooperative.org/download?arkid=http://n2t.net/ark:/99166/{identifier}&type=constellation_json",
    "merge_order": 95,

    "fetcherClass": "sources.archives.snac.fetcher.SNACFetcher",
    "mapperClass": "sources.archives.snac.mapper.SNACMapper"
}