Text Importer for TETXML; mapping into canonical format

[simon-clematide]

conversion of TETXML to impresso canonical format

• deciding for the exact TETML format (including characters and der position, or just from words as defined by TETXML)
• dealing with coordinates transformation of PDF into IIIF coordinates (code exists for that) in /mnt/storage/clwork/projects/climpresso/local_storages/karr/nzz/lib/tetml2canonical.py
• dealing with the logical segmentation of FG in articles
• producing JSON

schemas for canonical format

• issues: https://github.com/impresso/impresso-schemas/blob/master/json/newspaper/issue.schema.json
• pages: https://github.com/impresso/impresso-schemas/blob/master/json/newspaper/page.schema.json

The software should match the overall architecture of the impresso textimporters from various formats.

For instance, the one from mets_alto:

https://github.com/impresso/impresso-text-acquisition/tree/master/text_importer/importers/mets_alto

[aflueckiger]

Programming the method to_json for NewspaperIssue and NewspaperPage: https://github.com/impresso/impresso-text-acquisition/blob/master/text_importer/importers/classes.py

• [x] understand code, make branch of text importer and mark everything that needs to be extended

Find actual article boundaries in case they end/start on the same page:

• [X] grep "#ST#" Do they correspond with article titles? --> Yes, indicating the start of title (the title may be spread across a few subsequent lines) grep -C1 "ST" data_text/de/1849/**/*.text
• [x] fuzzy match on article title for articles that end within page (edit distance 10)

[mromanello]

Hi @aflueckiger ! great to see you are making progress on this. Just a heads-up to say that the library documentation (RTD-style) should be published by the end of today (and the GH repo will become public). That should be helpful to see how a TETML importer would fit into the overall library (but perhaps it's already clear).

[mromanello]

As promised, the documentation is now live here. The Writing a new importer section may be especially useful to you.

Any feedback on the clarity of the docs and possible improvements is very welcome!

[aflueckiger]

Thanks @mromanello. I will have a look at the documentation today and make up my mind about the implementation the TETML importer. Probably, I start programming next week.

[aflueckiger]

Steps to get tetml import working:

• [x] fork & branch repo impresso-text-acquisition

• comply with canonical archiving

  • [x] adapt pipeline to make canonical subdirs (FedGazXX, year, month, day)
  • [x] define abbreviations (FedGazDe FedGazFr FedGazIt) in impresso_commons/path/path_fs.py
  • [x] produce new tetml to get lines as there are no lines, only paragraphs
  • template /mnt/storage/clwork/projects/climpresso/local_storages/karr/nzz/pdf2wordplus.mk
  • tet --tetml word --pageopt "tetml={elements={line} glyphdetails={all}} contentanalysis={dehyphenate=true keephyphenglyphs=true}" file.pdf

• write code for generic tetml in impresso-text-acquisition/text_importer/importers/

  • treat each tetml as separate article in generic version, assuming no doubled pages across articles
  • [x] write TetmlNewspaperPage and TetmlNewspaperIssue in tetml/classes.py adapting from Olive
  • [x] create tetmlimporter.py in impresso-text-acquisition/text_importer/scripts/
  • [x] create test_tetml_importer.py in tests/importers/
  • [x] test functionality of generic tetml importer

• write custom Tetml importer for FedGaz

  • [x] use metadata to lookup title and language
  • [x] overwrite start and end pages according to metadata
  • [x] logical article boundary detection as an ad hoc measure (within page segmentation)

  • ST#" indicates the start of a title (the title may be spread across a few subsequent lines)

    grep -C1 "ST" data_text/FedGazDe/1849/**/*.text

  • fuzzy match on article title for articles that end within page (edit distance 10)
  • [x] prune article content according to logical article boundaries
  • [x] extend page span when recomposing articles

• [x] clean up text (e.g. #ST, page numbers, "Schweizerisches Bundesblatt" headline, boilerplates)

  • filter only ST # before json creation
  • it doesn't make sense to reassign the issue headlines (other than article headlines) --> leave it for the time being

• [x] adapt to new schema standard

• [x] treat multi-hyphenated word (>2 boxes) as a non-hyphenated word as it is delicate to assume a proper table segmentation because of the possibility of merged cells.

• Checks:

  • [x] adapt script for coordinate conversion check
  • [x] test with more data
  • [x] refactoring code
  • [x] redownloading pdfs from Bundesarchiv
  • check issues with "random" empty pdfs. likely, that a missing/false action command cause this problem (needed to download big pdf)

• Final Implementation:

  • [x] make generic tetml-importer and tetml-importer-fedgaz that inherits from the generic one
  • [ ] write recursive make recursion when the pipeline and documentation is accepted

• Improvements:

  • [ ] improve the computation of the bounding box of a token by removing outliers in a line

questions:

• including the data ingestion into pipeline? if so, how to integrate the other impresso-text-aquisation repo
• removing page numbers in canonical data?
• remove tables or parse them separately?
• issues with "random" empty pdf --> does the missing action command cause this problem?
• does generic tetml importer need to support images? --> no

Suggestions documentation:

• text_importer/data/sample_data/ instead of text_importer/data/sample/
• add information about S3 credentials
• add information about initalization of submodule: git submodule update --init

suggestions coding:

• whitespace rules only for French as of yet
• reference master instead of specific commit to allow continuous updates of known journals
• disentangle S3 from importer: only require S3 if a bucket is provided when calling the generic CLI script

[aflueckiger]

Importer for TETML and FedGaz-TETML is implemented in the following repo: https://github.com/impresso/impresso-text-acquisition/tree/master/text_importer

Documentation for the FedGaz specificities can be found in the README of this repo.