Automatize extraction of text analysis on specific dataset (stage 1)

[blcham]

The goal is to create scripts to extract HTML files from TermIt and process them to produce statistics.

We have the following artifacts:

• vocabulary
• datasets -- see "Documents" tab

Our goal is to create statistics such as this:

• termit-export folder in google drive

Terminology:

• training-dataset = manually annotated data ( = possibly automatically annotated but as well confirmed)
• full-dataset = manually annotated data + automatically annotated data

Additional notes:

• RDFa
• in best case scenario, each line in output file should produce one component and failure

Full task view:

• [x] stage 0 - create a minimal project that would not depend on TermIt (e.g. executed from Java tests)
• [x] stage 1 - we want to have a bash script that has a folder with dataset files as an input
• [ ] stage 2 - fine-tunning extraction
• [ ] stage 3 - have a module in SPipes that extracts dataset files from TermIt (note: TermIt is secured)

A/C = implementing stage 1:

• new module for extraction is created using maven scripts to create SPipes module (i suggest module "csat")
• there is a script in this repository that can extract CSV (excel) from files located in a folder (NOTE!!!: dataset files MUST not be committed in the repository, e.g. use a private folder)
• with the script, I would be able to do statistics (partially manually) such as in termit-export folder in google drive
• such statistics are exported in google drive to separate = new folder

[blcham]

See https://github.com/kbss-cvut/aircraft-maintenance-planning-system/blob/main/aircraft-maintenance-planning-model/data/termit/termit-nlp.sms.ttl

[blcham]

Algorithm for stage 2 (I would start without custom rule first):

input: vocabulary IRI + html document with table + custom extraction rules

1. convert html using tabular module (output is RDF representation of table)
2. take every cell value (literals) that has term occurences () and annotate the cell value with the occurences (create extract-term-occurences-module)
3. apply algorithm to select appropriate terms (terms with highest score + rules) (applyConstruct)
4. create different RDF representation of the table that is compliant with this output
5. convert RDF to CSV

---

Module extract-term-occurences-module:

• input: RDF containing literals which contains
• output: occurrences pointing to literals

Example of the module's inputs and outputs:

• input RDF:

  :x a csvw:row
  :csat-wo-tc "4339272" ;
  :tc-reference "52-610-00-04" ;
  :wo-text "<span about="_:a970-5" property="ddo:je-výskytem-termu" resource="http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/missing-part" typeof="ddo:výskyt-termu" score="0.5">finding</span> (nrc) taskcard 531900-03-1 (1.0) / item 1 <br> there were <span about="_:a970-6" property="ddo:je-výskytem-termu" resource="http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/missing-part" typeof="ddo:výskyt-termu" score="0.5">found</span> <span id="id33cs" about="_:33cs" property="http://onto.fel.cvut.cz/ontologies/application/termit/pojem/je-výskytem-termu" resource="http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/broken-part" typeof="http://onto.fel.cvut.cz/ontologies/application/termit/pojem/výskyt-termu" class="assigned-term-occurrence selected-occurrence">broken</span> <span id="idc342-14" about="_:c342-14" property="http://onto.fel.cvut.cz/ontologies/application/termit/pojem/je-výskytem-termu" resource="http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/drain-valve" typeof="http://onto.fel.cvut.cz/ontologies/application/termit/pojem/výskyt-termu" class="assigned-term-occurrence selected-occurrence">drain valves</span>" ;

• output RDF (see RDF4J of termit how it looks exactly):

  _:a970-5 a ddo:výskyt-termu ; 
  :score "0.5"^^integer ; 
  ddo:je-výskytem-termu "http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/missing-part"
  :references-annotation "<span about="_:a970-5" property="ddo:je-výskytem-termu" resource="http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/missing-part" typeof="ddo:výskyt-termu" score="0.5">finding</span> (nrc) taskcard 531900-03-1 (1.0) / item 1 <br> there were <span about="_:a970-6" property="ddo:je-výskytem-termu" resource="http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/missing-part" typeof="ddo:výskyt-termu" score="0.5">found</span> <span id="id33cs" about="_:33cs" property="http://onto.fel.cvut.cz/ontologies/application/termit/pojem/je-výskytem-termu" resource="http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/broken-part" typeof="http://onto.fel.cvut.cz/ontologies/application/termit/pojem/výskyt-termu" class="assigned-term-occurrence selected-occurrence">broken</span> <span id="idc342-14" about="_:c342-14" property="http://onto.fel.cvut.cz/ontologies/application/termit/pojem/je-výskytem-termu" resource="http://onto.fel.cvut.cz/ontologies/slovnik/slovnik-komponent-a-zavad---novy/pojem/drain-valve" typeof="http://onto.fel.cvut.cz/ontologies/application/termit/pojem/výskyt-termu" class="assigned-term-occurrence selected-occurrence">drain valves</span>" ;
  :references-text "finding (nrc) taskcard 531900-03-1 (1.0) / item 1 there were found broken drain valves1"
  :annotation-in-text-start "0"^^integer ;
  :annotation-in-text-end "7"^^integer ;
  .

---

Custom rule: if ?x part-of ?y then select ?x if there are multiple components

Instance of the rule: if "security seal" part-of "first aid kit" then pick "security seal"

You can find relevant detail in 8th line of 2020-02-18-termit-export-full-dataset

Application of the rule should be done in applyConstruct:

CONSTRUCT ...
WHERE {
    ?x :part-of ?y  .
    ?x a :Component .
    ?y a :Component .
}

Rule pattern: ?x :part-of ?y where ?x a :Component, ?y a :Component .

Rule instance (this should be ideally loaded from ttl file from within GIT repository): :security-seal :part-of :first-aid-kit .

---

Related resources in https://graphdb.onto.fel.cvut.cz/ termit-csat repository:

http://onto.fel.cvut.cz/ontologies/application/termit/pojem/cíl-souborového-výskytu/instance416404681
http://onto.fel.cvut.cz/ontologies/application/termit/pojem/má-selektor
http://onto.fel.cvut.cz/ontologies/application/termit/pojem/selektor-pozici-v-textu/instance625469017
http://onto.fel.cvut.cz/ontologies/application/termit/pojem/má-startovní-pozici
http://onto.fel.cvut.cz/ontologies/application/termit/pojem/má-koncovou-pozici

Related query:

select * where {     
     #<http://onto.fel.cvut.cz/ontologies/application/termit/pojem/selektor-text-quote/instance-1002686696> 
     ?s ?p ?stq .
     ?stq  <http://onto.fel.cvut.cz/ontologies/application/termit/pojem/má-přesný-text-quote> ?tQuote .
    filter(contains(str(?tQuote), "finding"))
} limit 100