Closed nichtich closed 4 years ago
My current use case is to serialize filtered Wikidata dumps in QuickStatements syntax, so I'm starting to implemented at https://github.com/nichtich/wikidata-sdk/tree/quickstatements
what do you do with a QuickStatements serialized dump then? (trying to understand the use case :))
@nichtich I have interest in this for large-volume updates (eg 1.7M claims, 82Mb QS CSV), since I thought QuickStatements is not suited for that: https://www.wikidata.org/wiki/Topic:Vfnr6v8lfqagpi7g.
@magnusmanske advised me to use the QS Batch tool (source https://github.com/magnusmanske/quickstatements_rs, deployed at https://tools.wmflabs.org/quickstatements/#/user).
I currently see reliability problems with QS Interactive:
Hopefully QS Batch won't have these problems; but if it has then I'll try using wikibase-cli.
@maxlath The current version of wikibase-cli would need some adaptations to make it suited for large-volume updates:
--quiet
option to reduce output to only error messagesSo far no luck with the QS Batch tool (see https://www.wikidata.org/wiki/Topic:Vfnr6v8lfqagpi7g)
not to say that it can't be improved, but a lot of what you ask is already possible in some way (out of the QS syntax)
option to take commands from file: I think the overhead of starting wd for every operation would be significant
there is certainly an overhead, but in my experience, it has been acceptable: the bottleneck remains Wikibase API
some way to add refs/qualifiers together with the claim, rather than the 2-step process described at https://github.com/maxlath/wikibase-cli/blob/master/docs/write_operations.md#with-a-reference
it's already possible using wb edit-entity
, I especially recommand you to have a look at the dynamic template, which allow to edit any part of an entity in a single edit, and with extreme flexibility (see this example)
a --quiet option to reduce output to only error messages
errors are written on the process stderr
, so redirecting stdout
to /dev/null
should work.
ex: wd sl Q4115189 foo bar > /dev/null
some reporting at which line the error occurred (so I can restart from there) option to stop at first error, or to continue
errors make the process exit with a non-zero code, so if you where to have a script like
set -e
wd sl Q4115189 foo bar
wd sl Q4115189 nl bar
It would print an error to stderr and exit the script thank to the set -e
bash parameter. You could then edit your script to fix the error and remove all the lines prior to the erronous command so that they aren't run again
FWIW, I fixed the QS (OAuth) issue. If you are logged into QS, you can find instructions here.
it would be too much work to add support for QS syntax, especially to maintain, as that would require staying up to date with QS evolutions, but:
wikibase-cli
, or more probably directly wikibase-edit
, would be welcome, and I could even help with one or the other's APIwikibase-edit
API in a newline-delimited JSON syntax (one operation per line), but I don't have the time to work on that nowI gave a shot to a batch mode, feedback welcome
@maxlath thanks a lot, seems perfect! After my QS batches finish (recently tge has been a speed up due to a fixed deadlock issue), I'll collect the leftover (which will be in the 100,000s) and run with your batch mode.
https://github.com/maxlath/wikidata-scripting/blob/master/worldcat_identities_ids/README.md is also great. Can't one use nohup
to ensure the script runs uninterrupted?
A QuickStatements issue has been re-opened on wikibase-edit repo https://github.com/maxlath/wikibase-edit/issues/64
The JSON format of wdEdit.entity.edit seems to be mappable from QuickStatements syntax and vice versa. I don't know where the mapping might best be place (
wikidata-sdk
?) but in the end it should be provided as alternative format tocreate-item
/edit-item
: if the first character isn't{
, try parsing QuickStatements syntax.