TEI import stabilization

[rsimon]

The TEI import is still rather prototypical. When importing a large text, the following two aspects cause problems:

• Conversion from text to annotations+context can run out of memory. Needs to be made iterable/streamable somehow
• At the moment, all annotations on a single item are imported to ElasticSearch in one batch. That wasn't a problem so far, since a single item would usually have one, or at least just a handful, of annotations. With TEI documents, however, a single item (the doc) can have 1,000s of annotations. That exceeds what the ElasticSearch buffer will handle at a time. I.e. import of annotations needs to be sequentialized in the same way as everything else in the system.

[rsimon]

TEI conversion was now revised completely (commit 7727c9ee5db7a96493d2a6976b985ba69ffcd6f3). Annotation import I just tried with a 5k+ document (The Histories) and things worked fine. Need to check the code though to confirm this is actually safe, not just a luck with a freshly-started & otherwise idle instance of ElasticSearch.

[rsimon]

Checked the code. References are imported through the bulk API, plus the import is integrated with the standard failure/retry mechanism, i.e. this works:

https://github.com/pelagios/peripleo2/blob/master/app/services/item/importers/BaseImporter.scala#L92-L100

Earlier failures were actually a consequence of the broken TEI conversion, which was fixed in 7727c9ee5db7a96493d2a6976b985ba69ffcd6f3.