Closed bmschmidt closed 8 years ago
OK, I have figured this out.
It looks to me like in the dumps they send DPLA, someone is replacing multispace sequences with a single space. So for example, in HathiTrust record number 100308172, you get (from the url here)
the following field 008:
000502s1908 enkae |o|||| 001 0 eng d
But in the dumps, it looks like this:
000502s1908 enkae |o|||| 001 0 eng d
Since 008 is a fixed-width field, this is bad. Some piece of code, either at Hathi or inside DPLA, is over-aggressively cleaning. Without the spaces publication place and language data can't be extracted.
I've written DPLA about this to see if they know why it's happening. @jjett or @organisciak, do you have a contact at Hathi we could ask about this?
Actually, since 008 is a fixed width field, couldn't you just use a regular expression to cherry pick the publication place and language?
Yeah, we should be able to extract the bytes by position--but the issue is that some bytes are being thrown away unpredictably, so we can't do that in full assurance of the MARC fields still working.
We could use internal hathi records which aren't junked like the DPLA versions... But there are access issues.
Mark Matienzo writes back from DPLA that it's their fault, but that they're scaling back from offering full metadata anyway. I have found the original XML files and will work with those instead.
According to the LOC, the 3-character publication place information in field 008 should be in bytes 15 to 17. But in many of the records from DPLA, it's instead in bytes 12-14. Why?
https://github.com/Bookworm-project/Bookworm-MARC/blob/master/bookwormMARC/bookwormMARC.py#L59-L72