Knit together SRO's digital objects with our copies

[keithamoss]

Now let's stitch the two together so we can see if the files we have can be associated with SRO's digital objects collection

Dependencies: #10 Next: #9

Process

• Open the JSON blob of filenames, and the JSON blob of digital objects from SRO
• Iterate through each digital object searching for the value of "filename" field in the filenames JSON blob
• Print out the number of matches
• Hopefully we get a lot of matches and all is good!
  • If not, we'll have to dive into looking at why the non-matching ones aren't
  • If so, let's do a sanity check and also compare the filesizes we have to theirs to make sure we're using the highest quality scans.

    Potential Challenges

• The names of the files SRO uploaded may not match the names of the files they gave us 3 years ago.
• Parsing inconsistent filenames!

[keithamoss]

What else have they stored in the JPEG metadata?

At least resolution (PPI), but what else?!

[keithamoss]

Use SQLite/PostgreSQL rather than JSON to make querying easier.

[keithamoss]

Need to exclude fieldbooks

[keithamoss]

6090 rows matching

SELECT fields->'filename' as do_filename, fields->'filesize' as do_filesize, img.filename, img.filesize, img.id as img_id FROM sro_digital_objects_collection AS "do", sro_images AS "img" WHERE fields->>'filename' = replace(img.filename, ' ', '_')

655 DOs with no match

SELECT fields->'filename' as do_filename FROM sro_digital_objects_collection AS "do" WHERE NOT EXISTS 
(SELECT replace(filename, ' ', '_') FROM sro_images WHERE fields->>'filename' = replace(filename, ' ', '_'))

[keithamoss]

From a quick comparison of file sizes it looks like they match pretty closely. calling this done!

[keithamoss]

SELECT MIN(width), MIN(height), MAX(width), MAX(height), AVG(width), AVG(height), MEDIAN(width), MEDIAN(height) FROM sro_images

https://wiki.postgresql.org/wiki/Aggregate_Median