Open mikegerber opened 2 years ago
While eynollah should handle this gracefully, we should also consider how to handle irrelevant images that are already marked as such in the METS structMap
. In this case possibly spine
and colour_checker
(could also be SBB defined types):
<mets:structMap TYPE="LOGICAL">
<mets:div ADMID="AMD" CONTENTIDS="http://resolver.staatsbibliothek-berlin.de/SBB000205BC00000000" DMDID="DMDLOG_0000" ID="LOG_0000" LABEL="Disputationum Medicarum Undecima, De Chirurgia" ORDERLABEL="Disputationum Medicarum Undecima, De Chirurgia" TYPE="monograph">
<mets:div ID="LOG_0001" TYPE="binding">
<mets:div ID="LOG_0002" TYPE="cover_front"/>
<mets:div ID="LOG_0003" TYPE="paste_down"/>
<mets:div ID="LOG_0004" TYPE="endsheet">
<mets:div ID="LOG_0005" TYPE="contents"/>
</mets:div>
</mets:div>
<mets:div ID="LOG_0006" TYPE="title_page"/>
<mets:div DMDID="DMDLOG_0001" ID="LOG_0007" LABEL="Quaestio Prima. [bis] 44." TYPE="section"/>
<mets:div ID="LOG_0008" TYPE="binding">
<mets:div ID="LOG_0009" TYPE="endsheet"/>
<mets:div ID="LOG_0010" TYPE="paste_down"/>
<mets:div ID="LOG_0011" TYPE="cover_back"/>
<mets:div ID="LOG_0012" TYPE="spine"/>
</mets:div>
<mets:div ID="LOG_0013" TYPE="colour_checker"/>
</mets:div>
(Full document: PPN894261851.zip)
@bertsky @kba @cneud What are your thoughts on this?
Yes, it should be possible to skip pages marked as certain types in the logical structmap – not just in any one processor, but as a general mechanism for workflows in OCR-D.
For the concrete set of supported page types, we should stick to DFG Strukturdatenset, which is strangely missing colour_checker
.
This set is also partially supported by ocrd-anybaseocr-layout-analysis
:
{'annotation': 0, 'binding': 1, 'chapter': 2, 'colour_checker': 3, 'contained_work': 4, 'contents': 5, 'cover': 6, 'edge': 7, 'endsheet': 8, 'epicedia': 9, 'illustration': 10, 'index': 11, 'musical_notation': 12, 'page': 13, 'paste_down': 14, 'preface': 15, 'provenance': 16, 'section': 17, 'sermon': 18, 'table': 19, 'title_page': 20}
For the general mechanism, I suggest something along the lines of our --page-id
CLI option's existing numerical range syntax, but more elaborate. For example, one could define filter operators that can look into the structmap, perhaps XPath expressions with predefined functions?
Yes, it should be possible to skip pages marked as certain types in the logical structmap – not just in any one processor, but as a general mechanism for workflows in OCR-D.
For the concrete set of supported page types, we should stick to DFG Strukturdatenset, which is strangely missing
colour_checker
.
100% agree! Should we take this to an OCR-D core or spec issue? I have some additional thoughts to discuss (like: What happens with skipped pages in the output?)
Should we take this to an OCR-D core or spec issue?
Yes, we should elevate this to OCR-D/spec.
I have some additional thoughts to discuss (like: What happens with skipped pages in the output?)
There is already some discussion on skip strategies for API changes in spec...
With the current version including https://github.com/qurator-spk/eynollah/issues/67 I was able to
FILE_0017_MAX.tif
successfully without memory explosion-di
flag without running into memory issuesIs there anything relevant from here that is still needed for https://github.com/OCR-D/spec/issues/172#issuecomment-693593327 or can we close this?
With the current version including #67 I was able to
* process `FILE_0017_MAX.tif` successfully without memory explosion7 * process the whole document PPN894261851 using the `-di` flag without running into memory issues
Is there anything relevant from here that is still needed for OCR-D/spec#172 (comment) or can we close this?
I wouldn't know, the current version is not working for OCR-D and so I can't reproduce until it's fixed. (Yes, there is a elaborate workaround but I am not willing to invest the time to reproduce with a lengthy changeset (#86) missing.)
With this document (PPN894261851.zip) we experienced an OOM error. Further investigation revealed this memory usage (measured using procpath):
The culprit seems to be this "page" from the document - an image of a book spine:
Relevant parts from the log output:
This log output is not from the OOM, but another run I did on a different machine to investigate the problem. If I interpret the
cont_page
part correctly, the image is blown up to[ 4404, 27685],
which would certainly explain the OOM error on the other machine.Reproduce with
ocrd-eynollah-segment -I MAX -O TEST-SEGMENT -P models /path/to/models
.