Error while parsing dump

[goramartin]

Hello, yesterday I downloaded a new wikidata json dump. But during the first phase of the build-tree I have received an error. I am still running on version before dep. updates.

2024/02/24 23:57:50 Processed 78530000 entities
2024/02/24 23:57:51 Processed 78540000 entities
2024/02/24 23:57:51 Something went wrong while processing.. cannot decode json: calendar model: invalid value
panic: Something went wrong while processing.. cannot decode json: calendar model: invalid value

goroutine 13 [running]:
log.Panicln({0xc000b46fb0?, 0xc0000380b0?, 0xc0000ce4b0?})
        /usr/local/go/src/log/log.go:398 +0x65
RecommenderServer/transactions.WikidataDumpTransactionSource.func1.1()
        /home/gora/RecommenderServer/transactions/createTransactions.go:57 +0xd5
created by RecommenderServer/transactions.WikidataDumpTransactionSource.func1
        /home/gora/RecommenderServer/transactions/createTransactions.go:24 +0x8d

[goramartin]

I tried it with the new dependencies update, just in case, and I am getting the same error.

[miselico]

From parallel conversation by email:

• The dump used is wikidata-20240219-all.json.bz2
• The spec defines a field with key "calendarmodel", what is encountered is "calendar model", with a space. I checked for that string in the dump, but could not find it, except for one occurrence in a literal value.

The entity number indicated above is not necessarily indicative because the reading an processing happen in parallel.

[miselico]

Now also reported here https://gitlab.com/tozd/go/mediawiki/-/issues/1 . It appears to be an issue in the parser which assumes one of two calendar types, while apparently also another on is used.

[mitar]

BTW, I suggest you print out errors with "% -+#.1v" as the format string because that will dump out also the problematic JSON and/or value itself. So we will be able to see what is the calendar type value used.

[mitar]

So I found it. The value is https://www.wikidata.org/wiki/Q12138, which should not be possible, because it should be https://www.wikidata.org/wiki/Q1985727, not sure how that value for the model happened.

[mitar]

Example: https://www.wikidata.org/wiki/Q105958428

[mitar]

Version v0.14.1 of gitlab.com/tozd/go/mediawiki has been released which should allow parsing those non-standard calendar models as well.

[goramartin]

After updating the dependency locally, it works again. Thank you very much.

[miselico]

Thank you both for fixing and checking this. @mitar could you elaborate what the format string "% -+#.1v" means? I cannot find this syntax in the fmt documentation. I found: %#v a Go-syntax representation of the value, but not what the other parts would mean. Is it an option to just log errE.Details() ?

[mitar]

% -+#.1v is syntax for formatting errors from gitlab.com/tozd/go/errors package which more or less says "print out everything you have" (stack trace, recursive wrapped/joined errors, details, etc.). You can see documentation here. This package is used by the gitlab.com/tozd/go/mediawiki. See also this issue.

%#v would format only any additional details.