Scraping SRO's digital objects collection

[keithamoss]

Is there an API? If not, can we get a data dump? If not, we'll scrape.

Will ask SRO.

Next: #11

[keithamoss]

If we are in want of an API...

Let's scrape the digital objects collection from SRO.

Process

• Scrape /digitalobject/browse
• Find all image preview tiles with links to /{item-name}
• Scrape /{item-name}
• For each item listed in the page, request its page
• Extract the fields in a meaningful and machine-readable fashion
• time.sleep(1) to be nice to SRO's servers
• Save to disk as a nice JSON blob
• time.sleep(1) to be nice to SRO's servers
• Iterate to next page

• Rinse and repeat until we've got all 6,600 objects.

Potential Challenges

• Can we reliably link items to the filenames we have on the harddrive?
• How many of the items in SRO's digital objects collection are not maps?
• Do we need the Dublin Core XML, or is scraping the HTML enough?

Test Data

Page https://archive.sro.wa.gov.au/index.php/a-c-gregory-bejoording-to-sturt-river-through-wongan-hills-021

Dublin Core XML https://archive.sro.wa.gov.au/index.php/a-c-gregory-bejoording-to-sturt-river-through-wongan-hills-021;dc?sf_format=xml

Dublin Core Metadata Schema http://www.openarchives.org/OAI/2.0/oai_dc.xsd

Page https://archive.sro.wa.gov.au/index.php/plan-showing-locations-for-the-salvation-army-north-north-east-of-collie-by-n-j-moore-fieldbooks-20-24-scale-20-chains-to-an-inch-wellington-160

Image https://archive.sro.wa.gov.au/uploads/r/srowa/3/2/32111b70dfa1d0759802feedc15efefc391474bb4509a1d02999ea229e8346be/Cons_3869_Wellington_160.jpg

Dublin Core XML https://archive.sro.wa.gov.au/index.php/plan-showing-locations-for-the-salvation-army-north-north-east-of-collie-by-n-j-moore-fieldbooks-20-24-scale-20-chains-to-an-inch-wellington-160;dc?sf_format=xml

[samwilson]

I guess it's unlikely that they'll upgrade to the latest AtoM, that has an API?

[keithamoss]

In the short term, certainly unlikely, @samwilson.

They've suggested they could send us a database dump in the next couple of weeks though - so we may not have to scrape at all!

[samwilson]

Oh that's cool! :)

[keithamoss]

Checkout https://github.com/geogeeks-au/maps-for-lost-towns/blob/master/scrapers/SRO%20Digital%20Objects%20Scraper.ipynb for a partially completed scraper.

[keithamoss]

The scraper has finished running against SRO! We've now got all 6,745 maps in the database in the new sro_digital_objects_collection table.

Code is here: https://github.com/geogeeks-au/maps-for-lost-towns/blob/master/scrapers/SRO%20Digital%20Objects%20Scraper.ipynb CSV dump is here: https://github.com/geogeeks-au/maps-for-lost-towns/blob/master/scrapers/sro_digital_objects_collection.csv