search

GLAM-Workbench / glam-workbench.github.io

https://glam-workbench.github.io/
24 stars 5 forks source link

Harvesting data from NSW State Archives database #45

Open wragge opened 2 years ago

wragge commented 2 years ago

See: https://ozglam.chat/t/nsw-state-archive-record-series-without-index/648

My initial response:

It’s not an easy one. The Primo system that NSWSA now uses is not very friendly to scraping data. You can get data for individual items as XML once you know their id, but I can’t figure out how to get a list of results. The interface uses a Javascript framework to load the search results, so the results aren’t ‘in’ the HTML of the page. It’s so annoying (the new Trove interface does this too…).

I think the only way around this would be to use something like Selenium which mimics a web browser and would allow the Javascript details to load, or leverage Zotero’s Primo translator – that might at least get 50 results at a time…

wragge commented 2 weeks ago

Has this changed with the move of the Archives to whatever it is now? The indexes have changed and use a GraphQL API, haven't checked if this applies to other content.