Import geo-aware TEI dataset in ES, link to questionnaire data and augment with spatial shapes

ale0xb commented 6 years ago

This is the step 2 of #19

I'm opening this issue to track the progress of combining these two datasets (plus possibly a third one providing the geo-shapes) into a new one, which in turn will serve as an entry point for more sophisticated data visualisations.

Geo-aware hierarchy

In the past there was no way to link the dictionary sources to places in the map. However, recent advances on the data curation of the dboe dataset have produced a geo-aware hierarchy of sources based on alphanumeric strings ("sigles") present in the original data. An example hierarchy is as follows

<listPlace xml:id="sigle:1A.3c01">
    <place type="Bundesland">
      <placeName>STir.</placeName>
      <idno>1A</idno>
      <listPlace>
        <place type="Großregion">
          <placeName>mSTir.</placeName>
          <idno>1A.21A.3</idno>
          <listPlace>
            <place type="Kleinregion">
              <placeName>Jaufengeb.</placeName>
              <idno>1A.3c</idno>
              <listPlace>
                <place type="Gemeinde">
                  <placeName>Ratschings</placeName>
                  <idno/>
                  <listPlace>
                    <place type="Ort">
                      <placeName>Ratschings, Racines</placeName>
                      <idno>1A.3c01</idno>
                    </place>
                  </listPlace>
                </place>
              </listPlace>
            </place>
          </listPlace>
        </place>
      </listPlace>
    </place>
</listPlace>

In the excerpt a top-down hierarchy of places related to the sigle "1A.3c01" is presented. For example the place this sigle refers to can be found under the last placeName tag, Ratschings/Racines, a municipality in South Tyrol, Italy.

The rest of more general administrative divisions can be found in the enclosing tags, each one having shorter idno (sigles) prefixes as you go up the tree.

This information can be found under the "usg" tag on each record. Theoretically, I have been informed this information could be augmented with actual geo information extracted from MySQL and indexed using ES geo-shapes.

Link with questionnaire data

Linking with questionnaire is kinda of more cumbersome process and not all TEI records will have a direct correspondence to a question. There are current efforts to migrate the rest of these questionnaires but it hasn't happened yet. Maybe someone who knows this stuff can look into this @amelieacdh?

For example we can find tags like: (from db182_qdb-TEI-02.xml)

<ref type="fragebogenNummer">60F55: Holzstoß (Zain); Pl./Dem.; Füg./Ra.</ref>

(from I537_qdb-TEI-02.xml)

<ref type="fragebogenNummer">86B26 (I,II): Lavendel = Lavandula angustifolia</ref>

which can be mapped respectively to questions (inside frage-fragebogen-full-tgd01.xml):

<item n="F55" xml:id="Fb-60-F55">
               <label>Waldarbeit</label>
               <pc>:</pc>
               <seg ana="question">
                  <seg xml:id="d1e223582">Holzstoß</seg>
                  <pc>(</pc>
                  <seg xml:id="d1e223586">Zain</seg>
                  <pc>)</pc>
                  <pc>;</pc>
                  <seg xml:id="d1e223592">Pl.</seg>
                  <pc>/</pc>
                  <seg xml:id="d1e223596">Dem.</seg>
                  <pc>;</pc>
                  <seg xml:id="d1e223600">Ra.</seg>
               </seg>
            </item>

and

<item n="B26" xml:id="Fb-86-B26">
               <label>Gartenblume</label>
               <pc>:</pc>
               <seg ana="question">
                  <seg xml:id="d1e368579">Lavendel</seg>
               </seg>
</item>

Sum up

As I see it In order to have this all working we need:

[x] MySQL instance (already done) with dboe data loaded to extract geoshapes.
[x] ES instance setup and running.
[ ] ES mappings
[ ] Import scripts working and running in a separate container.

I will open sub-issues from here so we have everything nice and tidy @interrogator.