Open TobiasNx opened 3 years ago
I don't think I can confirm.
First of all, AlmaMarc21XmlToLobidJsonTest.setup()
sets GENERATE_TESTDATA = true
, so the test doesn't actually test anything (AFAIUI).
But even after removing that line, executing mvn clean test-compile failsafe:integration-test -Dit.test=AlmaMarc21XmlToLobidJsonTest
always yields the exact same comparison failure (see below). No erratic behaviour observed here.
Can you be more specific as to what should be the expected outcome?
expected:<...subject" : [ {
"[label" : "Berlin",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"label" : "Bremen",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"label" : "Hamburg",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"label" : "Nordrhein-Westfalen",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"label" : "Niedersachsen",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"label" : "Rheinland-Pfalz",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"label" : "Saarland",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {]
"label" : "Schl...> but was:<...subject" : [ {
"[type" : [ "Keyword" ],
"label" : "Berlin",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],
"label" : "Bremen",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],
"label" : "Hamburg",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],
"label" : "Nordrhein-Westfalen",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],
"label" : "Niedersachsen",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],
"label" : "Rheinland-Pfalz",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],
"label" : "Saarland",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],]
"label" : "Schl...>
at org.lobid.resources.AlmaMarc21XmlToLobidJsonTest.lambda$transformFiles$1(AlmaMarc21XmlToLobidJsonTest.java:174)
at org.lobid.resources.AlmaMarc21XmlToLobidJsonTest.transformFiles(AlmaMarc21XmlToLobidJsonTest.java:137)
There's almost definitely something fishy going on... When shuffling the file list, catching the AssertionError
and then just printing the failing file names, it becomes apparent that the failures are order-dependent:
(DE-605)HT003176544
(DE-605)TT050421649
(DE-605)HT015671602
(DE-605)HT017411546
(DE-605)HT017411546
(DE-605)HT003176544
(DE-605)HT003176544
(DE-605)HT020202475
(DE-605)HT015671602
(DE-605)HT017411546
(DE-605)HT017411546
(DE-605)HT020202475
(DE-605)HT003176544
(DE-605)TT050421649
(DE-605)TT050421649
(DE-605)HT017411546
(DE-605)TT050421649
(DE-605)HT020202475
(DE-605)HT017411546
While they're stable when iterating in fixed (directory) order:
(DE-605)HT003176544
(DE-605)HT017411546
(DE-605)HT003176544
(DE-605)HT017411546
(DE-605)HT003176544
(DE-605)HT017411546
(DE-605)HT003176544
(DE-605)HT017411546
It's also not limited to the <choose>
elements in subjects.xml
:
(DE-605)HT017411546 expected:<..." : "Münster",
"[]publicationHistory" ...> but was:<..." : "Münster",
"[location" : "Münster",
"]publicationHistory" ...>
Which originates from a - seemingly - inconspicuous <data>
element (via macro regex-del-punctuation-end
) in titleRelatedFields.xml
.
Can you be more specific as to what should be the expected outcome?
It is expected that 610 and 650 get a "type" : [ "Keyword" ] or "type" : [ "Concept" ] based on the specific second indicator of that field.
the internal inconsistency is shown in this two cases:
(DE-605)HT003176544.json:
"label" : "Schleswig-Holstein",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],
"label" : "Hotel- und Gaststättengewerbe",
"source" : {
"label" : "Freie Verschlagwortung",
The 610 and the 650 even if they are the same workflow 610 in this example provides the type and 650 does not.
We only have one 610 testfile.
Unfortunately not shown in the the catched example 650 also does not work always and even can give different results in different testfiles . It then can happen that HT003176544 and HT020202475 have different results with this workflow. We saw the case that: While HT003176544: "label" : "Hotel- und Gaststättengewerbe
" has no type HT020202475: "label" : "Critical theory"
and "label" : "Electronic books
" have or the other way around.
(DE-605)HT020202475.json
"subject" : [ {
"type" : [ "Keyword" ],
"label" : "Critical theory",
"source" : {
"label" : "Freie Verschlagwortung",
"id" : "https://www.wikidata.org/wiki/Q47524318"
}
}, {
"type" : [ "Keyword" ],
"label" : "Electronic books",
"source" : {
"label" : "Freie Verschlagwortung",
Your second case:
It's also not limited to the
elements in subjects.xml: (DE-605)HT017411546 expected:<..." : "Münster", "[]publicationHistory" ...> but was:<..." : "Münster", "[location" : "Münster", "]publicationHistory" ...>
Which originates from a - seemingly - inconspicuous element (via macro regex-del-punctuation-end) in titleRelatedFields.xml.
Could this due to the fact that the source data has duplicates?
<datafield tag="264" ind1="3" ind2="1">
<subfield code="a">Münster</subfield>
<subfield code="b">Regensberg</subfield>
<subfield code="a">Münster</subfield>
<subfield code="b">Historische Kommission, Landschaftsverband Westfalen-Lippe</subfield>
<subfield code="c">1951-2004</subfield>
<subfield code="3">1951-2004</subfield>
</datafield>
<datafield tag="264" ind1="2" ind2="1">
<subfield code="a">Münster</subfield>
<subfield code="b">Regensberg</subfield>
<subfield code="3">1951-1977</subfield>
</datafield>
Münster appears multiple times.
Sorry for intervening late: the tests are not working and we have an issue for that. For the rest of the problem I can't say because I haven't checked that on my own yet.
the tests are not working
I suspect both issues have exactly the same root cause. Your tests don't give reliable results (seems to be the location
issue mentioned above), because of the inconsistency between generating the test data and testing them (as well as between different test runs).
I was able to reproduce the erratic behaviour with a reduced test case (HT020391499
and HT003176544
, in this order):
Possible outcomes for HT003176544
(as of hbz/lobid-resources@ccc34fc):
610
fields gain type
.650
field loses type
.Test scenarios:
Control test with only HT003176544
:
Which leads to the following questions:
So here's a heavily reduced test case that hopefully still catches the essence of this issue (at least w.r.t. questions 2 and 4, maybe 3). You can run it with the following Gradle invocation from the metamorph
subdirectory:
../gradlew --rerun-tasks test --tests org.metafacture.metamorph.Issue366Test -Dissue366.verbose=true -Dissue366.iter=100
It shows that option 0 (the issue at hand) fails 50% of the time (misses k
) and that options 1-3 (small modifications to the morph) always pass (while option 1 only differs from options 2-3 in that it yields a different order in the output - as expected).
Input:
{"10012":{"a":"V"}}
Option 0:
<metamorph xmlns="http://www.culturegraph.org/metamorph" version="1">
<rules>
<entity name="r" flushWith="100??">
<choose flushWith="100??.a">
<data name="k" source="100?2.a">
<constant value="K"/>
</data>
</choose>
<data name="v" source="100??.a" />
</entity>
</rules>
</metamorph>
Output 0:
{"r":{"v":"V"}} // 47/100
{"r":{"k":"K","v":"V"}} // 53/100
Option 1:
<metamorph xmlns="http://www.culturegraph.org/metamorph" version="1">
<rules>
<entity name="r" flushWith="100??">
<choose flushWith="100??">
<data name="k" source="100?2.a">
<constant value="K"/>
</data>
</choose>
<data name="v" source="100??.a" />
</entity>
</rules>
</metamorph>
Diff 1:
--- option0.xml
+++ option1.xml
@@ -1,7 +1,7 @@
<metamorph xmlns="http://www.culturegraph.org/metamorph" version="1">
<rules>
<entity name="r" flushWith="100??">
- <choose flushWith="100??.a">
+ <choose flushWith="100??">
<data name="k" source="100?2.a">
<constant value="K"/>
</data>
Output 1:
{"r":{"v":"V","k":"K"}}
Option 2:
<metamorph xmlns="http://www.culturegraph.org/metamorph" version="1">
<rules>
<entity name="r" flushWith="100??">
<choose flushWith="100??.a">
<data name="k" source="100??.a">
<constant value="K"/>
</data>
</choose>
<data name="v" source="100??.a" />
</entity>
</rules>
</metamorph>
Diff 2:
--- option0.xml
+++ option2.xml
@@ -2,7 +2,7 @@
<rules>
<entity name="r" flushWith="100??">
<choose flushWith="100??.a">
- <data name="k" source="100?2.a">
+ <data name="k" source="100??.a">
<constant value="K"/>
</data>
</choose>
Output 2:
{"r":{"k":"K","v":"V"}}
Option 3:
<metamorph xmlns="http://www.culturegraph.org/metamorph" version="1">
<rules>
<entity name="r" flushWith="100??">
<choose flushWith="100?2.a">
<data name="k" source="100?2.a">
<constant value="K"/>
</data>
</choose>
<data name="v" source="100?2.a" />
</entity>
</rules>
</metamorph>
Diff 3:
--- option0.xml
+++ option3.xml
@@ -1,12 +1,12 @@
<metamorph xmlns="http://www.culturegraph.org/metamorph" version="1">
<rules>
<entity name="r" flushWith="100??">
- <choose flushWith="100??.a">
+ <choose flushWith="100?2.a">
<data name="k" source="100?2.a">
<constant value="K"/>
</data>
</choose>
- <data name="v" source="100??.a" />
+ <data name="v" source="100?2.a" />
</entity>
</rules>
</metamorph>
Output 3:
{"r":{"k":"K","v":"V"}}
Working with lobid-resources I found a bug.
https://github.com/hbz/lobid-resources/blob/ccc34fc8590340c6fb72d4d9c884746090b7fbf8/src/main/resources/alma/common/subjects.xml#L4-L60
To add a type specific to the second indicator of a marc field in 610 and 650 I have introduced a
choose
collector withsameEntity="true" reset="true" flushWith="650??.a"
(or 610??.a). The wrapping mother entity also has flushWith conditions. Both flushWith as well al the source data refrence are using wild cards.When I run the lobid morph to test with:
mvn failsafe:integration-test -Dit.test=AlmaMarc21XmlToLobidJsonTest
Results for this specific transformation process seem to be erratic. They are inconsistent internally as well as between different test runs. But is does not always create these inconsistencies between runs. https://github.com/hbz/lobid-resources/commit/ccc34fc8590340c6fb72d4d9c884746090b7fbf8
@dr0i saw it live. When we changed the
flushWith
-attribute ofchoose
to the entity level (610?? instead of 610??.a). The bug does not appear.