Open bmschmidt opened 7 years ago
organisciak
commented
7 years ago Could you sum of all the return values for contributing_library? e.g. cat test.json | jq '.contributing_library' | sort | uniq. I know NRLF is "University of California Northern Regional Library Facility", Eleanor should have info for others.
Also, send me the URL for the imperfect current version that you have hosted, I'll try to build it with the unigrams.
bmschmidt
commented
7 years ago I've fixed up the contributing libraries more recently by using the first few characters of the identifier instead of the contributing library code. That seems to do it.
I will look around for the data. I've noticed a few changes to the bookwormDB repo that need to be made. A few are already on my hosted version and I will push them this afternoon.
A few additional changes that need to happen I'm noting here, even though it's the wrong place, that will make the memory tables work better.
1. SET optimizer_search_depth=0;
2. Increase Memory Table limit by about 50%.
3. Ensure silencing of errors is working.
I want to do a status check--after the penultimate conference call, I think I was supposed to check in with @organisciak on the status of this.
I've built out the jsoncatalog.txt for the bookworm; it can be as the file "jsoncatalog_hathi.txt.gz" hosted on my personal web domain. (Not posting the URL because it's huge and I don't want robots to try and download it.)
There are still a few issues. The "contributing library" is sometimes a code, because it's looked up in a sub-optimal place.
There's not a full field_descriptions.txt, because it's not yet determined exactly which fields we want in the bookworm. (I plan to do some visualizations with the MARC field item, for instance; but I doubt anyone else cares about that.)
But I think it's necessary to get the code in shape to redo the metadata anyway (which mostly works at present) so it should be safe to build the bookworm with some subset of the data here.
Here are some random entries from the top of the first million or so items in the file; they are all from the non-PD portion of the collection, where we've looked less in the past.