search

srophe / syriaca-data

Repository for Syriaca.org TEI data, used by srophe-eXist-app.
4 stars 16 forks source link

Convert BHSE records to new NHSL format #648

Closed nathangibson closed 7 years ago

nathangibson commented 7 years ago

@wsalesky , we would like to convert the BHSE records to the revised NHSL data format. Could you please write a script to do this? I've written up here https://docs.google.com/document/d/1fFkachdqCg04J4aGTvvxWx_CVmyDNE8ByukibsRgdFI/edit?usp=sharing what the script would need to do.

We want to do this in the dev branch only, of course, and then work on getting the app visualization to show the new format correctly. You could first run it on a small sample of records or be prepared to roll back changes if I've missed something or something goes awry.

Let me know if you need clarification. Thanks!

cc: @davidamichelson

nathangibson commented 7 years ago

I forgot to include in the above doc that we should also add a changelog item describing the changes.

wsalesky commented 7 years ago

Okay, I will take a look.

wsalesky commented 7 years ago

I will be doing the developments on this branch: https://github.com/srophe/srophe-app-data/tree/issue648

nathangibson commented 7 years ago

Great, thanks!

wsalesky commented 7 years ago

First take: https://github.com/srophe/srophe-app-data/tree/issue648

Let me know how it looks. Also, I posted all the change queries here (a single file, although you have to run them one query at a time): https://github.com/srophe/srophe-xQueries/blob/master/nhsl-conversion.xql

nathangibson commented 7 years ago

This is looking really good for the first take!

A few things I see on a quick glance at https://github.com/srophe/srophe-app-data/blob/issue648/data/works/tei/1000.xml:

Thanks!

wsalesky commented 7 years ago

Okay newest version: https://github.com/srophe/srophe-app-data/tree/issue648-2

nathangibson commented 7 years ago

I hope to look at this more thoroughly tomorrow, but looking at work/1000 I notice a couple of issues for the moment:

  1. The first relation in bib1000-3 has a blank passive value. I think we decided to eliminate relations for items that do not indicate their witnesses (i.e., notes that have no @corresp).
  2. bib1000-4 (manuscript) does not have a @source. The @source should be transferred from the note containing it. (I may have overlooked this in the original instructions.)

Thanks!

wsalesky commented 7 years ago

Okay. I will take a look (but will wait to rerun the scripts until you get back to me tomorrow)

nathangibson commented 7 years ago

Other than the two things I mentioned above, I see the following

  1. We also need @source transferred from note to bibl on ModernTranslation and AncientVersion. (In other words, Editions are the only ones doing this correctly right now).
  2. We should add you to the new change element or include an additional change element for you.

Looks great otherwise. Thanks!

wsalesky commented 7 years ago

Okay, I will try to run this tomorrow. If I don't get to it, I can have it run by Monday.

nathangibson commented 7 years ago

Sounds great. The next step (once we verify the results) will be to tweak the visualization to accommodate this new format. We can open a new issue about that and go through it step-by-step.

wsalesky commented 7 years ago

Updated: https://github.com/srophe/srophe-app-data/tree/issueNHSL

nathangibson commented 7 years ago

This looks good except for 2 things which we could maybe fix with just a find/replace:

  1. For some reason the latest version removed xmlns:lawd="http://lawd.info/ontology/". Actually we should additionally include xmlns:dct="http://purl.org/dc/terms/".

  2. Did you not want to include a change element citing yourself in the @who?

Once these are fixed, this branch could be merged into dev so that we can work on visualization.

Thanks so much!

wsalesky commented 7 years ago

Ops! I actually just put the xml namespaces in with a find and replace on the root element. I will do that, and add a new change element... I will have it up by the end of the day.

wsalesky commented 7 years ago

Okay, I think it ready to go. Review and merge when you are ready. I will probably have to manually push them to the dev server.

nathangibson commented 7 years ago

@wsalesky Ya, these do need to be manually pushed to the dev server. Thanks!

wsalesky commented 7 years ago

Okay, I pushed them to dev. Looks like they broke the display. I will take a look at it.