Closed ebeshero closed 6 years ago
How to run chunks of Frankenstein through CollateX for TEI app-crit output:
1) Produce JSON input file (using XSLT or XQuery), holding base text of the witnesses 2) Download CollateX java jar locally from here: http://collatex.net/download/
3) At command line, in the CollateX directory, run:
java -jar -Xmx6g collatex-tools-1.7.1.jar ../GitHub/Pittsburgh_Frankenstein/CollateX/FrankenCollatV1ch1.json -f tei -o ../GitHub/Pittsburgh_Frankenstein/CollateX/collatedFrankV1Ch1.xml
This takes input JSON and outputs TEI.
-xml
, but we should probably wait to run that when we have better XML to work with (for stuff like italics, things missing in current texts).
*Note: -Xmx6g sets 6 GB of memory to work on this. Probably NOT necessary for processing one chapter at a time. This is better for chunking larger passages. @Rikkm An update on collation: @djbpitt and I worked on normalization and collation of a portion of all three texts yesterday, and we've got a Python script worked out for this on my branch that we'll keep refining. I can merge this with our Text_Processing branch, but I also wanted to check in with you as we're prepping for the Saturday meeting. We noticed a single-letter typo that I want to correct in Ch. 1 of the 1823 text (my error, I bet, b/c I was editing that part)--but I know you're working on the file. Is it safe for me to go in and push a tiny correction on the 1823 text over to Text_Processing, together with our new collation stuff?
II added some edits last night, they've been pushed up already, so you should be fine. I'll make sure to do a pull before I begin working on it again this evening.
On Thu, May 4, 2017 at 7:00 PM, Elisa Beshero-Bondar < notifications@github.com> wrote:
@Rikkm https://github.com/Rikkm An update on collation: @djbpitt https://github.com/djbpitt and I worked on normalization and collation of a portion of all three texts yesterday, and we've got a Python script worked out for this on my branch that we'll keep refining. I can merge this with our Text_Processing branch, but I also wanted to check in with you as we're prepping for the Saturday meeting. I noticed in a single-letter typo I want to correct in Ch. 1 of the 1823 text (my error I bet b/c I was editing that part)--but I know you're working on the file. Is it safe for me to go in a push a tiny correction on that text over to Text_Processing, together with our new collation stuff?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ebeshero/Pittsburgh_Frankenstein/issues/2#issuecomment-299332422, or mute the thread https://github.com/notifications/unsubscribe-auth/ASoxxmWg-hGC1CedsQnE6y63tMb1gg6oks5r2lidgaJpZM4KixIw .
@Rikkm Good--yes, I just pulled those in. I'll make the change and then merge my branch. I'm going to blitz through as much of 1831 as I can after that. Let me know how close you're getting to the finish of 1823!
Okay, just to get us ready for the next stage: We're going to need to do a little prep to help align the three texts for collation:
<milestone/>
markers--this doesn't need to be paragraph-by-paragraph; more like chapter-by-chapter, taking note of how much material tends to get moved around between chapters. It's best if we do this in small-ish units, but (for example), if we notice that some text moves from Chapter 15 in 1823 to Chapter 16 in 1831, we'll want to take units that hold the same material (and maybe make that a two-chapter unit). We'll use the <milestone/>
markers as signals for the Python script to "chunk" the three texts into units of comparison. @Rikkm See preliminary output here: https://github.com/ebeshero/Pittsburgh_Frankenstein/blob/ebeshero_Exp/collateXPrep/outputEBB.txt
I've just gotten home, so I'm eating then will be working on 1823 from 8-9 pm, then I have a call, and probably again from 10 - 11 or 12. I don't think I'll be able to finish 1823 tonight, but I'll try to finish it tomorrow morning or before noon.
I then have a big thing going on Monday, for which I am coordinating the group, so I don't think I'll have any more bandwidth Friday or Saturday after our meeting, but I'll update you Saturday.
On Thu, May 4, 2017 at 7:17 PM, Elisa Beshero-Bondar < notifications@github.com> wrote:
@Rikkm https://github.com/Rikkm Good--yes, I just pulled those in. I'll make the change and then merge my branch. I'm going to blitz through as much of 1831 as I can after that. Let me know how close you're getting to the finish of 1823!
Okay, just to get us ready for the next stage: We're going to need to do a little prep to help align the three texts for collation:
- One thing we worked out is that I'll do an up-conversion of all three documents to very simple XML before doing the collation (it's just easier to use Python's XML parser than to try to use its regular expression matching on our pseudo-markup).
- And we'll be processing the texts in segments that we need to define: The texts are too long to process with collateX all at once (that would introduce alignment errors), so we'll want to inspect the texts and identify some locations where they line up, where we can drop
markers--this doesn't need to be paragraph-by-paragraph; more like chapter-by-chapter, taking note of how much material tends to get moved around between chapters. It's best if we do this in small-ish units, but (for example), if we notice that some text moves from Chapter 15 in 1823 to Chapter 16 in 1831, we'll want to take units that hold the same material (and maybe make that a two-chapter unit). We'll use the markers as signals for the Python script to "chunk" the three texts into units of comparison. - Let's check in fairly frequently as we're getting close to the finish on checking, and as soon as we're ready, I can get started on the XML prep and looking for good alignment points.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ebeshero/Pittsburgh_Frankenstein/issues/2#issuecomment-299334930, or mute the thread https://github.com/notifications/unsubscribe-auth/ASoxxkoQukBfRPGRAELXvF73fsWpTEiRks5r2lyGgaJpZM4KixIw .
@Rikkm Understood. I've just pushed the small corrections (after checking against the photofacsimiles). Everything's up to date (and all three of our GitHub branches have all of our commits), so you should be ready to go to keep working on 1823.
I could meet with you sometime before the full meeting on Saturday if you like--just let me know.
Hi Elisa,
On the one hand, I have the conference room from 12 - 3:30 on Saturday; on the other, third report we are presenting to most of the library personnel in Monday morning is not yet complete. I'll know more by tonight, though my personal best for a working day tends to hit the wall at 12 hours, and I think I'll need the additional time tomorrow morning and afternoon to continue this work.
I do appreciate your flexibility and the offer, a lot.
Rikk
On Thu, May 4, 2017 at 7:57 PM, Elisa Beshero-Bondar < notifications@github.com> wrote:
@Rikkm https://github.com/Rikkm Understood. I've just pushed the small corrections (after checking against the photofacsimiles). Everything's up to date, so you should be ready to go to keep working on 1823.
I could meet with you sometime before the full meeting on Saturday if you like--just let me know.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ebeshero/Pittsburgh_Frankenstein/issues/2#issuecomment-299340754, or mute the thread https://github.com/notifications/unsubscribe-auth/ASoxxneBK8OnYtffxkjqsR0dKVOYxszDks5r2mX5gaJpZM4KixIw .
@raffazizzi Tutorial for working with CollateX locally with Python http://collatex.obdurodon.org/