Implement methods for domains

DavidLeoni commented 7 years ago

Detecting domains

Following a review of how Wordnet 3.1 are converted to LMF, basically you can't directly state that a synset is a domain. You can know if a Synset is a domain if a) has associated at least one Sense that is linked in turn to a SemanticLabel of type ELabelTypeSemantics.domain, regionOfUsage or usage b) OR other synsets point to it with one of usage (pointer key ;r) , region (pointer key ;u) or topic (pointer key ;c) relations. c) OR establish a root domain synset (div_ss_n_domain, see #33) and make existing domains point to it. If a lexical resource has topics expressed only via b) method (like Wordnet 3.1) during import normalization substep we can automatically add edges pointing to the root marked with 'diversicon' provenance

So method c) should be the preferred one, as works even with synsets with nothing pointing at them and a root node can always come handy.

Domain hierarchy

Given that:

Wordnet partitions domain relations into usage (pointer key ;r) , region (pointer key ;u) or topic (pointer key ;c) relations.
None of these relations is hierarchical.
Only topic is transitive.
We need to express hierarchies
we prefer talking about 'domain' instead of topics
LMF converters use the generic word ELabelTypeSemantics.domain in SemanticLabel.type
Wordnet 3.1 domains look a bit confusing

We shall then:

1) introduce new domain and superDomain relations, plus the respective inverses domainOf and subDomain. This could be an example of usage:

ss_train domain ss_transportation superDomain ss_applied-sciences superDomain div_ss_domain

Note computation of transitive closure won't consider leaves, see #32

2) Introduce new div-upper lexical resource with root domain div_ss_n_domain

3) When importing:

preserve existing edges as usual
detect domain synsets and link them to root domain synset
normalize Wordnet topic relation into domain. If in input graph there are inverses but not canonical edges, add the canonical domain and superDomain
calculate transitive closure for superDomain

4) When exporting:

As usual, try to export a graph very similar to input one, so will avoid exporting computed domain edges.

Methods to implement

We should then implement the following methods:

List<Synset> getDomains() : returns all domains in the db, as synsets
boolean isDomain(Synset synset) : returns true if provided synset is a domain
Synset getDomain(String lemma, String lang) : returns all domains in the db, as synsets
isSubDomain(Synset a, Synset b) : returns true if a is a subdomain of b

DavidLeoni commented 7 years ago

Turned out that domain normalization the wa it is implemented at commit 532436db18 is a particularly expensive operation, importing Wordnet 3.1 the normalization takes 12 hours !!!!!!!

Log excerpt:

753451 core.Diversicon - Done loading LMF : /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml .
753451 core.Diversicon - 
753451 core.Diversicon - Executing post-import db validation ... 
753451 core.Diversicon - 
754310 core.Diversicon - DB is valid!
754311 core.Diversicon - 
754311 core.Diversicon -    Elapsed time: 0 secs
754311 core.Diversicon - 
754312 core.Diversicon - Normalizing SynsetRelations...
754651 core.Diversicon - 
Found 117792 synsets.

=====  REDACTED  =======
45309745 core.Diversicon - SynsetRelation normalization - processed synsets: 110,000
46116243 core.Diversicon - 
46116243 core.Diversicon - Done normalizing SynsetRelations.
46116243 core.Diversicon - 
46116244 core.Diversicon -    Max level:      0
   Initial edges:  285,668
   Inserted edges: 20,213
        domain:   11,761
        superDomain:   8,452

46116244 core.Diversicon -    Elapsed time: 12hours, 36mins, 1sec
46116244 core.Diversicon - 
46116244 core.Diversicon - Computing transitive closure for SynsetRelations (may take some minutes) ...
46116311 core.Diversicon - 
   Found 305,881 synset relations.

46213839 core.Diversicon - 
   Elapsed time: 1min, 37secs

46213840 core.Diversicon - 
Going to write closure into the db...

46221630 core.Diversicon - SynsetRelation transitive closure - written edges: 10,000
====== REDACTED =========
46399058 core.Diversicon - SynsetRelation transitive closure - written edges: 680,000
46403945 core.Diversicon - 
46403945 core.Diversicon - Done writing transitive closure for SynsetRelations.
46403945 core.Diversicon - 
46403946 core.Diversicon -    Max level:      19
   Initial edges:  305,881
   Inserted edges: 687,348
        hypernym:   661,083
        holonymPart:   26,265

46403946 core.Diversicon - Total elapsed time:  4mins, 47secs
46403946 core.Diversicon - 
46403947 core.Diversicon - Elapsed time: 12hours, 49mins, 31secs   Started: 2017-02-01 09:47:45   Ended: 2017-02-02 10:37:16
46403947 core.Diversicon - 
46403947 core.Diversicon - 
46403947 core.Diversicon - 
46403947 core.Diversicon - Done importing 1 LMF by import author David Leoni
46403947 core.Diversicon - 
46403947 core.Diversicon - Imported lexical resources: 
46403947 core.Diversicon - 
46403947 core.Diversicon -     div-wn31    from    /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml

Long log (still some parts are redacted)

[INFO] Scanning for projects...
[WARNING] The POM for eu.trentorise.opendata:josman-maven-plugin:jar:0.8.0-SNAPSHOT is missing, no dependency information available
[WARNING] Failed to retrieve plugin descriptor for eu.trentorise.opendata:josman-maven-plugin:0.8.0-SNAPSHOT: Plugin eu.trentorise.opendata:josman-maven-plugin:0.8.0-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact eu.trentorise.opendata:josman-maven-plugin:jar:0.8.0-SNAPSHOT
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building DivMaker 0.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for eu.trentorise.opendata:josman-maven-plugin:jar:0.8.0-SNAPSHOT is missing, no dependency information available
[WARNING] Failed to retrieve plugin descriptor for eu.trentorise.opendata:josman-maven-plugin:0.8.0-SNAPSHOT: Plugin eu.trentorise.opendata:josman-maven-plugin:0.8.0-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact eu.trentorise.opendata:josman-maven-plugin:jar:0.8.0-SNAPSHOT
[INFO] 
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) > validate @ divmaker >>>
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (enforce) @ divmaker ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (enforce-maven) @ divmaker ---
[INFO] 
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) < validate @ divmaker <<<
[INFO] 
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ divmaker ---
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/david.leoni/.m2/repository/org/slf4j/slf4j-simple/1.6.1/slf4j-simple-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/david.leoni/.m2/repository/ch/qos/logback/logback-classic/1.0.13/logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
237 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
238 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
238 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - ****   Going to create:
LexicalResourcePackage:
  name         = div-wn31
  label        = Diversicon WordNet 3.1
  prefix       = wn31
  version      = 0.1.0-SNAPSHOT
  namespaces   =
                 wn31:https://github.com/diversicon-kb/diversicon-wordnet-3.1
  sampleXmlUri = classpath:/div-wn31-sample.xml
  xmlUri       = classpath:/div-wn31.xml.xz
  sqlUri       = classpath:/div-wn31.sql.xz
  h2DbUri      = classpath:/div-wn31.h2.db.xz

238 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
238 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
Feb 01, 2017 9:43:55 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Started converting WordNet to LMF...
Feb 01, 2017 9:43:55 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SubcategorizationFrameExtractor parseSubcatMappings
INFO: Parsing SubcatMappings...
Feb 01, 2017 9:43:55 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SubcategorizationFrameExtractor parseSubcatMappings
INFO: done
Feb 01, 2017 9:43:55 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating Synsets...
Feb 01, 2017 9:43:55 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SynsetGenerator initialize
INFO: processing noun
Feb 01, 2017 9:43:58 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SynsetGenerator initialize
INFO: processing verb
Feb 01, 2017 9:44:00 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SynsetGenerator initialize
INFO: processing adjective
Feb 01, 2017 9:44:01 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SynsetGenerator initialize
INFO: processing adverb
Feb 01, 2017 9:44:01 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating Synsets done
Feb 01, 2017 9:44:01 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating LexicalEntries...
Feb 01, 2017 9:44:01 PM de.tudarmstadt.ukp.lmf.transform.wordnet.util.WNConvUtil buildExceptionMap
INFO: Skipped 1288 WordForm exceptions because their baseform was not defined in Wordnet. This is a Wordnet issue, for details see https://github.com/DavidLeoni/dkpro-uby/issues/4 
Feb 01, 2017 9:44:01 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator groupLexemes
INFO:  grouping lexemes...
Feb 01, 2017 9:44:01 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator groupLexemes
INFO: 0%
Feb 01, 2017 9:44:06 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator groupLexemes
INFO: 25%
Feb 01, 2017 9:44:07 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator groupLexemes
INFO: 50%
Feb 01, 2017 9:44:08 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator groupLexemes
INFO: 75%
Feb 01, 2017 9:44:08 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator groupLexemes
INFO: 100%
Feb 01, 2017 9:44:09 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: transforming lexeme groups... 0% 
Feb 01, 2017 9:44:10 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SenseGenerator generateSenses
WARNING: IndexSenseReader did not provide sense number for senseKey ddc%1:06:01::
adding a dummy value of sense number
Feb 01, 2017 9:44:10 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SenseGenerator generateSenses
WARNING: IndexSenseReader did not provide sense number for senseKey ddi%1:06:01::
adding a dummy value of sense number
Feb 01, 2017 9:44:11 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 10%
Feb 01, 2017 9:44:11 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SenseGenerator generateSenses
WARNING: IndexSenseReader did not provide sense number for senseKey sun%1:17:02::
adding a dummy value of sense number
Feb 01, 2017 9:44:11 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 20%
Feb 01, 2017 9:44:12 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 30%
Feb 01, 2017 9:44:13 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SenseGenerator generateSenses
WARNING: IndexSenseReader did not provide sense number for senseKey earth%1:15:01::
adding a dummy value of sense number
Feb 01, 2017 9:44:13 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SenseGenerator generateSenses
WARNING: IndexSenseReader did not provide sense number for senseKey earth%1:17:02::
adding a dummy value of sense number
Feb 01, 2017 9:44:13 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 40%
Feb 01, 2017 9:44:14 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SenseGenerator generateSenses
WARNING: IndexSenseReader did not provide sense number for senseKey kb%1:23:01::
adding a dummy value of sense number
Feb 01, 2017 9:44:14 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SenseGenerator generateSenses
WARNING: IndexSenseReader did not provide sense number for senseKey kb%1:23:03::
adding a dummy value of sense number
Feb 01, 2017 9:44:14 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 50%
Feb 01, 2017 9:44:15 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SenseGenerator generateSenses
WARNING: IndexSenseReader did not provide sense number for senseKey moon%1:17:03::
adding a dummy value of sense number
Feb 01, 2017 9:44:15 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 60%
Feb 01, 2017 9:44:16 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 70%
Feb 01, 2017 9:44:18 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 80%
Feb 01, 2017 9:44:21 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 90%
Feb 01, 2017 9:44:27 PM de.tudarmstadt.ukp.lmf.transform.wordnet.LexicalEntryGenerator createLexicalEntries
INFO: 100%
Feb 01, 2017 9:44:27 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating LexicalEntries done
Feb 01, 2017 9:44:27 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating SynsetRelations...
Feb 01, 2017 9:44:27 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SynsetRelationGenerator updateSynsetRelations
INFO: Skipping adding relation to synset [POS: adjective]-1124528 because of non-synset pointer type [PointerType: [Label: antonym] [Key: !] Applies To: noun, verb, adjective, adverb] , see https://github.com/DavidLeoni/dkpro-uby/issues/3
Feb 01, 2017 9:44:27 PM de.tudarmstadt.ukp.lmf.transform.wordnet.SynsetRelationGenerator updateSynsetRelations

===============   REDACTED several thousands logs like 'Skipping adding relation to synset ......  ========================================================

Feb 01, 2017 9:44:39 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating SynsetRelations done
Feb 01, 2017 9:44:39 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating RelatedForms...
Feb 01, 2017 9:44:44 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating RelatedForms done
Feb 01, 2017 9:44:44 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating SenseRelations...
Feb 01, 2017 9:44:54 PM de.tudarmstadt.ukp.lmf.transform.wordnet.WNConverter toLMF
INFO: Generating SenseRelations done
62106 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - Created lexical resource in memory!
62122 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - Creating dumps/diversicon   ...
62195 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - ****  Going to create XML to /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml
62195 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - ****                   (make take several minutes...)
78135 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - ****  Done creating /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml
78135 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
78135 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Creating /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml.xz   ....
213865 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Created /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml.xz
213865 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
213865 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Creating /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml.zip   ....
213888 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Created /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml.xz
222623 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicons - Creating tables in database  jdbc:h2:file:dumps/diversicon/div-wn31 ...
user=root, password=****, characterEncoding=UTF-8, charSet=UTF-8}
Wed Feb 01 21:47:43 CET 2017: COMMIT 0
Wed Feb 01 21:47:43 CET 2017: COMMIT 4
TOTAL TIME: 221
NUM ENTRIES: 4
231024 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done loading LMF : classpath:/div-upper.xml .
231024 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231024 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Executing post-import db validation ... 
231024 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231064 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - DB is valid!
231064 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231065 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -    Elapsed time: 0 secs
231065 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231068 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Normalizing SynsetRelations...
231099 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
Found 1 synsets.

231116 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231116 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done normalizing SynsetRelations.
231116 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231116 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -    No edges were added to the 0 existing ones. 

231116 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -    Elapsed time: 0 secs
231117 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231117 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Computing transitive closure for SynsetRelations (may take some minutes) ...
231123 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
   Found 0 synset relations.

231156 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
   Elapsed time: 0 secs

231156 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
Going to write closure into the db...

231169 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231169 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done writing transitive closure for SynsetRelations.
231169 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231169 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -    No edges were added to the 0 existing ones. 

231169 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Total elapsed time:  0 secs
231169 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Elapsed time: 3secs   Started: 2017-02-01 09:47:40   Ended: 2017-02-01 09:47:43
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done importing 1 LMF by import author Diversicon
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Imported lexical resources: 
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -     div-upper    from    classpath:/div-upper.xml
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicons - Done creating tables in database  jdbc:h2:file:dumps/diversicon/div-wn31
231170 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Connecting to database   jdbc:h2:file:dumps/diversicon/div-wn31

232843 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Connected!
232843 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Going to import 1 files by import author David Leoni...
232854 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Loading LMF : /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml ...
232857 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
232857 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Validating XML Schema of /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml   ...
232857 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
249273 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - XML is valid!
249273 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
249273 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Checking XML external references against the database ...
249273 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
249275 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.DivXmlValidator - Validating import data against the db....
252296 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - XML can be merged!
252296 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
252296 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
252296 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Starting import...
252296 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 

Wed Feb 01 21:48:05 CET 2017: COMMIT 0
Wed Feb 01 21:48:08 CET 2017: COMMIT 1000
Wed Feb 01 21:48:14 CET 2017: COMMIT 2000
Wed Feb 01 21:48:16 CET 2017: COMMIT 3000
Wed Feb 01 21:48:18 CET 2017: COMMIT 4000
Wed Feb 01 21:48:20 CET 2017: COMMIT 5000
Wed Feb 01 21:48:21 CET 2017: COMMIT 6000
Wed Feb 01 21:48:23 CET 2017: COMMIT 7000
Wed Feb 01 21:48:24 CET 2017: COMMIT 8000
Wed Feb 01 21:48:25 CET 2017: COMMIT 9000
Wed Feb 01 21:48:27 CET 2017: COMMIT 10000
Wed Feb 01 21:48:28 CET 2017: COMMIT 11000
Wed Feb 01 21:48:29 CET 2017: COMMIT 12000
Wed Feb 01 21:48:30 CET 2017: COMMIT 13000
Wed Feb 01 21:48:31 CET 2017: COMMIT 14000
Wed Feb 01 21:48:32 CET 2017: COMMIT 15000
Wed Feb 01 21:48:34 CET 2017: COMMIT 16000
Wed Feb 01 21:48:35 CET 2017: COMMIT 17000
Wed Feb 01 21:48:36 CET 2017: COMMIT 18000
Wed Feb 01 21:48:37 CET 2017: COMMIT 19000
Wed Feb 01 21:48:39 CET 2017: COMMIT 20000
Wed Feb 01 21:48:40 CET 2017: COMMIT 21000
Wed Feb 01 21:48:41 CET 2017: COMMIT 22000
Wed Feb 01 21:48:42 CET 2017: COMMIT 23000
Wed Feb 01 21:48:43 CET 2017: COMMIT 24000
Wed Feb 01 21:48:44 CET 2017: COMMIT 25000
Wed Feb 01 21:48:45 CET 2017: COMMIT 26000
Wed Feb 01 21:48:46 CET 2017: COMMIT 27000
Wed Feb 01 21:48:47 CET 2017: COMMIT 28000
Wed Feb 01 21:48:48 CET 2017: COMMIT 29000
Wed Feb 01 21:48:49 CET 2017: COMMIT 30000
Wed Feb 01 21:48:50 CET 2017: COMMIT 31000
Wed Feb 01 21:48:51 CET 2017: COMMIT 32000
Wed Feb 01 21:48:52 CET 2017: COMMIT 33000
Wed Feb 01 21:48:53 CET 2017: COMMIT 34000
Wed Feb 01 21:48:54 CET 2017: COMMIT 35000
Wed Feb 01 21:48:56 CET 2017: COMMIT 36000
Wed Feb 01 21:48:57 CET 2017: COMMIT 37000
Wed Feb 01 21:48:58 CET 2017: COMMIT 38000
Wed Feb 01 21:49:00 CET 2017: COMMIT 39000
Wed Feb 01 21:49:01 CET 2017: COMMIT 40000
Wed Feb 01 21:49:02 CET 2017: COMMIT 41000
Wed Feb 01 21:49:03 CET 2017: COMMIT 42000
Wed Feb 01 21:49:04 CET 2017: COMMIT 43000
Wed Feb 01 21:49:06 CET 2017: COMMIT 44000
Wed Feb 01 21:49:07 CET 2017: COMMIT 45000
Wed Feb 01 21:49:08 CET 2017: COMMIT 46000
Wed Feb 01 21:49:09 CET 2017: COMMIT 47000
Wed Feb 01 21:49:10 CET 2017: COMMIT 48000
Wed Feb 01 21:49:12 CET 2017: COMMIT 49000
Wed Feb 01 21:49:13 CET 2017: COMMIT 50000
Wed Feb 01 21:49:14 CET 2017: COMMIT 51000
Wed Feb 01 21:49:16 CET 2017: COMMIT 52000
Wed Feb 01 21:49:18 CET 2017: COMMIT 53000
Wed Feb 01 21:49:19 CET 2017: COMMIT 54000
Wed Feb 01 21:49:20 CET 2017: COMMIT 55000
Wed Feb 01 21:49:22 CET 2017: COMMIT 56000
Wed Feb 01 21:49:23 CET 2017: COMMIT 57000
Wed Feb 01 21:49:24 CET 2017: COMMIT 58000
Wed Feb 01 21:49:25 CET 2017: COMMIT 59000
Wed Feb 01 21:49:27 CET 2017: COMMIT 60000
Wed Feb 01 21:49:28 CET 2017: COMMIT 61000
Wed Feb 01 21:49:29 CET 2017: COMMIT 62000
Wed Feb 01 21:49:30 CET 2017: COMMIT 63000
Wed Feb 01 21:49:32 CET 2017: COMMIT 64000
Wed Feb 01 21:49:33 CET 2017: COMMIT 65000
Wed Feb 01 21:49:34 CET 2017: COMMIT 66000
Wed Feb 01 21:49:35 CET 2017: COMMIT 67000
Wed Feb 01 21:49:36 CET 2017: COMMIT 68000
Wed Feb 01 21:49:37 CET 2017: COMMIT 69000
Wed Feb 01 21:49:39 CET 2017: COMMIT 70000
Wed Feb 01 21:49:40 CET 2017: COMMIT 71000
Wed Feb 01 21:49:41 CET 2017: COMMIT 72000
Wed Feb 01 21:49:42 CET 2017: COMMIT 73000
Wed Feb 01 21:49:44 CET 2017: COMMIT 74000
Wed Feb 01 21:49:46 CET 2017: COMMIT 75000
Wed Feb 01 21:49:47 CET 2017: COMMIT 76000
Wed Feb 01 21:49:48 CET 2017: COMMIT 77000
Wed Feb 01 21:49:50 CET 2017: COMMIT 78000
Wed Feb 01 21:49:51 CET 2017: COMMIT 79000
Wed Feb 01 21:49:52 CET 2017: COMMIT 80000
Wed Feb 01 21:49:54 CET 2017: COMMIT 81000
Wed Feb 01 21:49:55 CET 2017: COMMIT 82000
Wed Feb 01 21:49:56 CET 2017: COMMIT 83000
Wed Feb 01 21:49:58 CET 2017: COMMIT 84000
Wed Feb 01 21:49:59 CET 2017: COMMIT 85000
Wed Feb 01 21:50:00 CET 2017: COMMIT 86000
Wed Feb 01 21:50:02 CET 2017: COMMIT 87000
Wed Feb 01 21:50:03 CET 2017: COMMIT 88000
Wed Feb 01 21:50:05 CET 2017: COMMIT 89000
Wed Feb 01 21:50:06 CET 2017: COMMIT 90000
Wed Feb 01 21:50:07 CET 2017: COMMIT 91000
Wed Feb 01 21:50:09 CET 2017: COMMIT 92000
Wed Feb 01 21:50:10 CET 2017: COMMIT 93000
Wed Feb 01 21:50:11 CET 2017: COMMIT 94000
Wed Feb 01 21:50:14 CET 2017: COMMIT 95000
Wed Feb 01 21:50:15 CET 2017: COMMIT 96000
Wed Feb 01 21:50:17 CET 2017: COMMIT 97000
Wed Feb 01 21:50:18 CET 2017: COMMIT 98000
Wed Feb 01 21:50:20 CET 2017: COMMIT 99000
Wed Feb 01 21:50:21 CET 2017: COMMIT 100000
Wed Feb 01 21:50:22 CET 2017: COMMIT 101000
Wed Feb 01 21:50:24 CET 2017: COMMIT 102000
Wed Feb 01 21:50:25 CET 2017: COMMIT 103000
Wed Feb 01 21:50:27 CET 2017: COMMIT 104000
Wed Feb 01 21:50:28 CET 2017: COMMIT 105000
Wed Feb 01 21:50:30 CET 2017: COMMIT 106000
Wed Feb 01 21:50:31 CET 2017: COMMIT 107000
Wed Feb 01 21:50:32 CET 2017: COMMIT 108000
Wed Feb 01 21:50:34 CET 2017: COMMIT 109000
Wed Feb 01 21:50:36 CET 2017: COMMIT 110000
Wed Feb 01 21:50:37 CET 2017: COMMIT 111000
Wed Feb 01 21:50:39 CET 2017: COMMIT 112000
Wed Feb 01 21:50:40 CET 2017: COMMIT 113000
Wed Feb 01 21:50:42 CET 2017: COMMIT 114000
Wed Feb 01 21:50:44 CET 2017: COMMIT 115000
Wed Feb 01 21:50:45 CET 2017: COMMIT 116000
Wed Feb 01 21:50:47 CET 2017: COMMIT 117000
Wed Feb 01 21:50:48 CET 2017: COMMIT 118000
Wed Feb 01 21:50:50 CET 2017: COMMIT 119000
Wed Feb 01 21:50:52 CET 2017: COMMIT 120000
Wed Feb 01 21:50:56 CET 2017: COMMIT 121000
Wed Feb 01 21:51:00 CET 2017: COMMIT 122000
Wed Feb 01 21:51:04 CET 2017: COMMIT 123000
Wed Feb 01 21:51:09 CET 2017: COMMIT 124000
Wed Feb 01 21:51:13 CET 2017: COMMIT 125000
Wed Feb 01 21:51:17 CET 2017: COMMIT 126000
Wed Feb 01 21:51:21 CET 2017: COMMIT 127000
Wed Feb 01 21:51:25 CET 2017: COMMIT 128000
Wed Feb 01 21:51:30 CET 2017: COMMIT 129000
Wed Feb 01 21:51:34 CET 2017: COMMIT 130000
Wed Feb 01 21:51:38 CET 2017: COMMIT 131000
Wed Feb 01 21:51:41 CET 2017: COMMIT 132000
Wed Feb 01 21:51:43 CET 2017: COMMIT 133000
Wed Feb 01 21:51:45 CET 2017: COMMIT 134000
Wed Feb 01 21:51:47 CET 2017: COMMIT 135000
Wed Feb 01 21:51:50 CET 2017: COMMIT 136000
Wed Feb 01 21:51:52 CET 2017: COMMIT 137000
Wed Feb 01 21:51:54 CET 2017: COMMIT 138000
Wed Feb 01 21:51:57 CET 2017: COMMIT 139000
Wed Feb 01 21:51:59 CET 2017: COMMIT 140000
Wed Feb 01 21:52:02 CET 2017: COMMIT 141000
Wed Feb 01 21:52:04 CET 2017: COMMIT 142000
Wed Feb 01 21:52:06 CET 2017: COMMIT 143000
Wed Feb 01 21:52:08 CET 2017: COMMIT 144000
Wed Feb 01 21:52:11 CET 2017: COMMIT 145000
Wed Feb 01 21:52:13 CET 2017: COMMIT 146000
Wed Feb 01 21:52:15 CET 2017: COMMIT 147000
Wed Feb 01 21:52:17 CET 2017: COMMIT 148000
Wed Feb 01 21:52:20 CET 2017: COMMIT 149000
Wed Feb 01 21:52:23 CET 2017: COMMIT 150000
Wed Feb 01 21:52:25 CET 2017: COMMIT 151000
Wed Feb 01 21:52:27 CET 2017: COMMIT 152000
Wed Feb 01 21:52:29 CET 2017: COMMIT 153000
Wed Feb 01 21:52:31 CET 2017: COMMIT 154000
Wed Feb 01 21:52:33 CET 2017: COMMIT 155000
Wed Feb 01 21:52:36 CET 2017: COMMIT 156000
Wed Feb 01 21:52:40 CET 2017: COMMIT 157000
Wed Feb 01 21:52:42 CET 2017: COMMIT 158000
Wed Feb 01 21:52:46 CET 2017: COMMIT 159000
Wed Feb 01 21:52:48 CET 2017: COMMIT 160000
Wed Feb 01 21:52:50 CET 2017: COMMIT 161000
Wed Feb 01 21:52:52 CET 2017: COMMIT 162000
Wed Feb 01 21:52:54 CET 2017: COMMIT 163000
Wed Feb 01 21:52:55 CET 2017: COMMIT 164000
Wed Feb 01 21:52:58 CET 2017: COMMIT 165000
Wed Feb 01 21:53:00 CET 2017: COMMIT 166000
Wed Feb 01 21:53:02 CET 2017: COMMIT 167000
Wed Feb 01 21:53:04 CET 2017: COMMIT 168000
Wed Feb 01 21:53:06 CET 2017: COMMIT 169000
Wed Feb 01 21:53:08 CET 2017: COMMIT 170000
Wed Feb 01 21:53:10 CET 2017: COMMIT 171000
Wed Feb 01 21:53:11 CET 2017: COMMIT 172000
Wed Feb 01 21:53:14 CET 2017: COMMIT 173000
Wed Feb 01 21:53:15 CET 2017: COMMIT 174000
Wed Feb 01 21:53:17 CET 2017: COMMIT 175000
Wed Feb 01 21:53:19 CET 2017: COMMIT 176000
Wed Feb 01 21:53:21 CET 2017: COMMIT 177000
Wed Feb 01 21:53:23 CET 2017: COMMIT 178000
Wed Feb 01 21:53:25 CET 2017: COMMIT 179000
Wed Feb 01 21:53:27 CET 2017: COMMIT 180000
Wed Feb 01 21:53:29 CET 2017: COMMIT 181000
Wed Feb 01 21:53:31 CET 2017: COMMIT 182000
Wed Feb 01 21:53:32 CET 2017: COMMIT 183000
Wed Feb 01 21:53:34 CET 2017: COMMIT 184000
Wed Feb 01 21:53:36 CET 2017: COMMIT 185000
Wed Feb 01 21:53:38 CET 2017: COMMIT 186000
Wed Feb 01 21:53:40 CET 2017: COMMIT 187000
Wed Feb 01 21:53:41 CET 2017: COMMIT 188000
Wed Feb 01 21:53:43 CET 2017: COMMIT 189000
Wed Feb 01 21:53:45 CET 2017: COMMIT 190000
Wed Feb 01 21:53:47 CET 2017: COMMIT 191000
Wed Feb 01 21:53:49 CET 2017: COMMIT 192000
Wed Feb 01 21:53:51 CET 2017: COMMIT 193000
Wed Feb 01 21:53:53 CET 2017: COMMIT 194000
Wed Feb 01 21:53:55 CET 2017: COMMIT 195000
Wed Feb 01 21:53:57 CET 2017: COMMIT 196000
Wed Feb 01 21:53:58 CET 2017: COMMIT 197000
Wed Feb 01 21:54:00 CET 2017: COMMIT 198000
Wed Feb 01 21:54:03 CET 2017: COMMIT 199000
Wed Feb 01 21:54:04 CET 2017: COMMIT 200000
Wed Feb 01 21:54:06 CET 2017: COMMIT 201000
Wed Feb 01 21:54:08 CET 2017: COMMIT 202000
Wed Feb 01 21:54:11 CET 2017: COMMIT 203000
Wed Feb 01 21:54:13 CET 2017: COMMIT 204000
Wed Feb 01 21:54:15 CET 2017: COMMIT 205000
Wed Feb 01 21:54:18 CET 2017: COMMIT 206000
Wed Feb 01 21:54:20 CET 2017: COMMIT 207000
Wed Feb 01 21:54:22 CET 2017: COMMIT 208000
Wed Feb 01 21:54:24 CET 2017: COMMIT 209000
Wed Feb 01 21:54:26 CET 2017: COMMIT 210000
Wed Feb 01 21:54:28 CET 2017: COMMIT 211000
Wed Feb 01 21:54:30 CET 2017: COMMIT 212000
Wed Feb 01 21:54:32 CET 2017: COMMIT 213000
Wed Feb 01 21:54:34 CET 2017: COMMIT 214000
Wed Feb 01 21:54:36 CET 2017: COMMIT 215000
Wed Feb 01 21:54:38 CET 2017: COMMIT 216000
Wed Feb 01 21:54:39 CET 2017: COMMIT 217000
Wed Feb 01 21:54:41 CET 2017: COMMIT 218000
Wed Feb 01 21:54:42 CET 2017: COMMIT 219000
Wed Feb 01 21:54:45 CET 2017: COMMIT 220000
Wed Feb 01 21:54:47 CET 2017: COMMIT 221000
Wed Feb 01 21:54:49 CET 2017: COMMIT 222000
Wed Feb 01 21:54:51 CET 2017: COMMIT 223000
Wed Feb 01 21:54:53 CET 2017: COMMIT 224000
Wed Feb 01 21:54:55 CET 2017: COMMIT 225000
Wed Feb 01 21:54:57 CET 2017: COMMIT 226000
Wed Feb 01 21:55:00 CET 2017: COMMIT 227000
Wed Feb 01 21:55:02 CET 2017: COMMIT 228000
Wed Feb 01 21:55:04 CET 2017: COMMIT 229000
Wed Feb 01 21:55:06 CET 2017: COMMIT 230000
Wed Feb 01 21:55:07 CET 2017: COMMIT 231000
Wed Feb 01 21:55:09 CET 2017: COMMIT 232000
Wed Feb 01 21:55:11 CET 2017: COMMIT 233000
Wed Feb 01 21:55:13 CET 2017: COMMIT 234000
Wed Feb 01 21:55:15 CET 2017: COMMIT 235000
Wed Feb 01 21:55:17 CET 2017: COMMIT 236000
Wed Feb 01 21:55:19 CET 2017: COMMIT 237000
Wed Feb 01 21:55:21 CET 2017: COMMIT 238000
Wed Feb 01 21:55:23 CET 2017: COMMIT 239000
Wed Feb 01 21:55:26 CET 2017: COMMIT 240000
Wed Feb 01 21:55:28 CET 2017: COMMIT 241000
Wed Feb 01 21:55:29 CET 2017: COMMIT 242000
Wed Feb 01 21:55:31 CET 2017: COMMIT 243000
Wed Feb 01 21:55:33 CET 2017: COMMIT 244000
Wed Feb 01 21:55:35 CET 2017: COMMIT 245000
Wed Feb 01 21:55:37 CET 2017: COMMIT 246000
Wed Feb 01 21:55:39 CET 2017: COMMIT 247000
Wed Feb 01 21:55:40 CET 2017: COMMIT 248000
Wed Feb 01 21:55:42 CET 2017: COMMIT 249000
Wed Feb 01 21:55:44 CET 2017: COMMIT 250000
Wed Feb 01 21:55:46 CET 2017: COMMIT 251000
Wed Feb 01 21:55:48 CET 2017: COMMIT 252000
Wed Feb 01 21:55:50 CET 2017: COMMIT 253000
Wed Feb 01 21:55:51 CET 2017: COMMIT 254000
Wed Feb 01 21:55:53 CET 2017: COMMIT 255000
Wed Feb 01 21:55:55 CET 2017: COMMIT 256000
Wed Feb 01 21:55:57 CET 2017: COMMIT 257000
Wed Feb 01 21:55:59 CET 2017: COMMIT 258000
Wed Feb 01 21:56:01 CET 2017: COMMIT 259000
Wed Feb 01 21:56:03 CET 2017: COMMIT 260000
Wed Feb 01 21:56:05 CET 2017: COMMIT 261000
Wed Feb 01 21:56:07 CET 2017: COMMIT 262000
Wed Feb 01 21:56:09 CET 2017: COMMIT 263000
Wed Feb 01 21:56:10 CET 2017: COMMIT 264000
Wed Feb 01 21:56:12 CET 2017: COMMIT 265000
Wed Feb 01 21:56:14 CET 2017: COMMIT 266000
Wed Feb 01 21:56:16 CET 2017: COMMIT 267000
Wed Feb 01 21:56:18 CET 2017: COMMIT 268000
Wed Feb 01 21:56:19 CET 2017: COMMIT 269000
Wed Feb 01 21:56:20 CET 2017: COMMIT 270000
Wed Feb 01 21:56:21 CET 2017: COMMIT 271000
Wed Feb 01 21:56:22 CET 2017: COMMIT 272000
Wed Feb 01 21:56:24 CET 2017: COMMIT 273000
Wed Feb 01 21:56:25 CET 2017: COMMIT 274000
Wed Feb 01 21:56:26 CET 2017: COMMIT 274652
TOTAL TIME: 500312
NUM ENTRIES: 274652
753451 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done loading LMF : /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml .
753451 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
753451 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Executing post-import db validation ... 
753451 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
754310 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - DB is valid!
754311 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
754311 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -    Elapsed time: 0 secs
754311 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
754312 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Normalizing SynsetRelations...
754651 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
Found 117792 synsets.

5803856 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 10,000
9533082 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 20,000
13599743 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 30,000
18676691 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 40,000
23796697 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 50,000
28413698 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 60,000
32006432 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 70,000
36428706 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 80,000
39982616 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 90,000
42957275 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 100,000
45309745 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation normalization - processed synsets: 110,000
46116243 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46116243 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done normalizing SynsetRelations.
46116243 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46116244 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -    Max level:      0
   Initial edges:  285,668
   Inserted edges: 20,213
        domain:   11,761
        superDomain:   8,452

46116244 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -    Elapsed time: 12hours, 36mins, 1sec
46116244 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46116244 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Computing transitive closure for SynsetRelations (may take some minutes) ...
46116311 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
   Found 305,881 synset relations.

46213839 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
   Elapsed time: 1min, 37secs

46213840 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
Going to write closure into the db...

46221630 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 10,000
46227195 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 30,000
46232405 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 50,000
46237461 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 70,000
46242817 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 90,000
46247996 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 110,000
46253189 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 130,000
46261245 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 160,000
46266670 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 180,000
46271775 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 200,000
46279445 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 230,000
46286711 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 260,000
46291742 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 280,000
46298981 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 310,000
46304305 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 330,000
46310156 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 350,000
46315493 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 370,000
46322621 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 400,000
46328005 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 420,000
46333119 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 440,000
46339386 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 460,000
46344445 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 480,000
46349653 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 500,000
46354783 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 520,000
46360429 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 540,000
46366616 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 560,000
46373949 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 590,000
46379228 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 610,000
46384782 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 630,000
46391795 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 660,000
46399058 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - SynsetRelation transitive closure - written edges: 680,000
46403945 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46403945 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done writing transitive closure for SynsetRelations.
46403945 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46403946 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -    Max level:      19
   Initial edges:  305,881
   Inserted edges: 687,348
        hypernym:   661,083
        holonymPart:   26,265

46403946 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Total elapsed time:  4mins, 47secs
46403946 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Elapsed time: 12hours, 49mins, 31secs   Started: 2017-02-01 09:47:45   Ended: 2017-02-02 10:37:16
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done importing 1 LMF by import author David Leoni
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Imported lexical resources: 
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - 
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon -     div-wn31    from    /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - ****  Exporting to SQL /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.sql
46403947 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Backing up database to /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.sql  ...
46439259 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.core.Diversicon - Done backing up database to /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.sql
46439260 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
46439260 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Creating /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.sql.xz   ....
46780249 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Created /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.sql.xz
46780249 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
46780249 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Creating /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.sql.zip   ....
46780262 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Created /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.sql.xz
46780262 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - ****  Compacting db
46800434 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - ****  Done compacting db.
46800434 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
46800434 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Creating /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.h2.db.xz   ....
47371853 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Created /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.h2.db.xz
47371853 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
47371853 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Creating /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.h2.db.zip   ....
47371854 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - **** Created /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.h2.db.xz
47371854 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
47371854 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
47371854 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - ****  DONE!!!   ALLELUJA!!!  
47371854 [eu.kidf.diversicon.maker.DivMaker.main()] INFO eu.kidf.diversicon.maker.DivMaker - 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:09 h
[INFO] Finished at: 2017-02-02T10:53:24+01:00
[INFO] Final Memory: 74M/2552M
[INFO] ------------------------------------------------------------------------

DavidLeoni commented 7 years ago

Just factorized some code so to have three big queries instead of one per synset , now normalization takes only 57 secs instead of 12 hours!!

Total Wordnet import time is ~14 mins, which is acceptable (probably we could do faster writes).

Log excerpt:

NUM ENTRIES: 274652
757756 core.Diversicon - Done loading LMF : /home/david.leoni/Da/prj/divmaker/dumps/diversicon/div-wn31.xml .
757756 core.Diversicon - 
757756 core.Diversicon - Executing post-import db validation ... 
757756 core.Diversicon - 
758562 core.Diversicon - DB is valid!
758562 core.Diversicon - 
758562 core.Diversicon -    Elapsed time: 0 secs
758562 core.Diversicon - 
758564 core.Diversicon - Normalizing SynsetRelations...
763330 core.Diversicon - 
Found 117792 synsets.

773099 core.Diversicon - SynsetRelation normalization - processed synsets: 10,000
781459 core.Diversicon - SynsetRelation normalization - processed synsets: 30,000
789787 core.Diversicon - SynsetRelation normalization - processed synsets: 40,000
795372 core.Diversicon - SynsetRelation normalization - processed synsets: 50,000
801837 core.Diversicon - SynsetRelation normalization - processed synsets: 70,000
808738 core.Diversicon - SynsetRelation normalization - processed synsets: 90,000
814917 core.Diversicon - SynsetRelation normalization - processed synsets: 110,000
815850 core.Diversicon - 
815850 core.Diversicon - Done normalizing SynsetRelations.
815850 core.Diversicon - 
815850 core.Diversicon -    Max level:      0
   Initial edges:  285,668
   Inserted edges: 20,213
        domain:   11,761
        superDomain:   8,452

815851 core.Diversicon -    Elapsed time: 57secs
815851 core.Diversicon - 
815851 core.Diversicon - Computing transitive closure for SynsetRelations (may take some minutes) ...
815889 core.Diversicon - 
   Found 305,881 synset relations.

917715 core.Diversicon - 
   Elapsed time: 1min, 41secs

917715 core.Diversicon - 
Going to write closure into the db...

922297 core.Diversicon - SynsetRelation transitive closure - written edges: 10,000
927675 core.Diversicon - SynsetRelation transitive closure - written edges: 30,000
934843 core.Diversicon - SynsetRelation transitive closure - written edges: 60,000
939964 core.Diversicon - SynsetRelation transitive closure - written edges: 80,000
948173 core.Diversicon - SynsetRelation transitive closure - written edges: 110,000
954064 core.Diversicon - SynsetRelation transitive closure - written edges: 130,000
959352 core.Diversicon - SynsetRelation transitive closure - written edges: 150,000
964927 core.Diversicon - SynsetRelation transitive closure - written edges: 170,000
970157 core.Diversicon - SynsetRelation transitive closure - written edges: 190,000
975188 core.Diversicon - SynsetRelation transitive closure - written edges: 210,000
980313 core.Diversicon - SynsetRelation transitive closure - written edges: 230,000
985488 core.Diversicon - SynsetRelation transitive closure - written edges: 250,000
990873 core.Diversicon - SynsetRelation transitive closure - written edges: 270,000
998594 core.Diversicon - SynsetRelation transitive closure - written edges: 300,000
1006397 core.Diversicon - SynsetRelation transitive closure - written edges: 330,000
1011878 core.Diversicon - SynsetRelation transitive closure - written edges: 350,000
1017050 core.Diversicon - SynsetRelation transitive closure - written edges: 370,000
1022607 core.Diversicon - SynsetRelation transitive closure - written edges: 390,000
1027898 core.Diversicon - SynsetRelation transitive closure - written edges: 410,000
1034593 core.Diversicon - SynsetRelation transitive closure - written edges: 430,000
1039931 core.Diversicon - SynsetRelation transitive closure - written edges: 450,000
1047153 core.Diversicon - SynsetRelation transitive closure - written edges: 480,000
1052458 core.Diversicon - SynsetRelation transitive closure - written edges: 500,000
1057499 core.Diversicon - SynsetRelation transitive closure - written edges: 520,000
1062534 core.Diversicon - SynsetRelation transitive closure - written edges: 540,000
1068270 core.Diversicon - SynsetRelation transitive closure - written edges: 560,000
1075448 core.Diversicon - SynsetRelation transitive closure - written edges: 590,000
1081187 core.Diversicon - SynsetRelation transitive closure - written edges: 610,000
1088765 core.Diversicon - SynsetRelation transitive closure - written edges: 640,000
1097127 core.Diversicon - SynsetRelation transitive closure - written edges: 670,000
1105746 core.Diversicon - 
1105746 core.Diversicon - Done writing transitive closure for SynsetRelations.
1105746 core.Diversicon - 
1105747 core.Diversicon -    Max level:      19
   Initial edges:  305,881
   Inserted edges: 687,348
        hypernym:   661,083
        holonymPart:   26,265

1105747 core.Diversicon - Total elapsed time:  4mins, 49secs
1105747 core.Diversicon - 
1105748 core.Diversicon - Elapsed time: 14mins, 36secs   Started: 2017-02-02 06:29:39   Ended: 2017-02-02 06:44:16

diversicon-kb / diversicon-core