Closed simon-clematide closed 3 years ago
Programming the method to_json for NewspaperIssue and NewspaperPage: https://github.com/impresso/impresso-text-acquisition/blob/master/text_importer/importers/classes.py
Find actual article boundaries in case they end/start on the same page:
grep -C1 "ST" data_text/de/1849/**/*.text
Hi @aflueckiger ! great to see you are making progress on this. Just a heads-up to say that the library documentation (RTD-style) should be published by the end of today (and the GH repo will become public). That should be helpful to see how a TETML importer would fit into the overall library (but perhaps it's already clear).
As promised, the documentation is now live here. The Writing a new importer section may be especially useful to you.
Any feedback on the clarity of the docs and possible improvements is very welcome!
Thanks @mromanello. I will have a look at the documentation today and make up my mind about the implementation the TETML importer. Probably, I start programming next week.
Steps to get tetml import working:
[x] fork & branch repo impresso-text-acquisition
comply with canonical archiving
/mnt/storage/clwork/projects/climpresso/local_storages/karr/nzz/pdf2wordplus.mk
tet --tetml word --pageopt "tetml={elements={line} glyphdetails={all}} contentanalysis={dehyphenate=true keephyphenglyphs=true}" file.pdf
write code for generic tetml
in impresso-text-acquisition/text_importer/importers/
TetmlNewspaperPage
and TetmlNewspaperIssue
in tetml/classes.py
adapting from Olivetetmlimporter.py
in impresso-text-acquisition/text_importer/scripts/
test_tetml_importer.py
in tests/importers/
write custom Tetml importer for FedGaz
grep -C1 "ST" data_text/FedGazDe/1849/**/*.text
[x] clean up text (e.g. #ST, page numbers, "Schweizerisches Bundesblatt" headline, boilerplates)
[x] adapt to new schema standard
[x] treat multi-hyphenated word (>2 boxes) as a non-hyphenated word as it is delicate to assume a proper table segmentation because of the possibility of merged cells.
Checks:
Final Implementation:
Improvements:
questions:
Suggestions documentation:
git submodule update --init
suggestions coding:
Importer for TETML and FedGaz-TETML is implemented in the following repo: https://github.com/impresso/impresso-text-acquisition/tree/master/text_importer
Documentation for the FedGaz specificities can be found in the README of this repo.
conversion of TETXML to impresso canonical format
schemas for canonical format
The software should match the overall architecture of the impresso textimporters from various formats.
For instance, the one from mets_alto:
https://github.com/impresso/impresso-text-acquisition/tree/master/text_importer/importers/mets_alto