Support QuickStatements syntax

nichtich commented 5 years ago

The JSON format of wdEdit.entity.edit seems to be mappable from QuickStatements syntax and vice versa. I don't know where the mapping might best be place (wikidata-sdk?) but in the end it should be provided as alternative format to create-item/edit-item: if the first character isn't {, try parsing QuickStatements syntax.

nichtich commented 5 years ago

My current use case is to serialize filtered Wikidata dumps in QuickStatements syntax, so I'm starting to implemented at https://github.com/nichtich/wikidata-sdk/tree/quickstatements

maxlath commented 5 years ago

what do you do with a QuickStatements serialized dump then? (trying to understand the use case :))

VladimirAlexiev commented 4 years ago

@nichtich I have interest in this for large-volume updates (eg 1.7M claims, 82Mb QS CSV), since I thought QuickStatements is not suited for that: https://www.wikidata.org/wiki/Topic:Vfnr6v8lfqagpi7g.

@magnusmanske advised me to use the QS Batch tool (source https://github.com/magnusmanske/quickstatements_rs, deployed at https://tools.wmflabs.org/quickstatements/#/user).

I currently see reliability problems with QS Interactive:

Hopefully QS Batch won't have these problems; but if it has then I'll try using wikibase-cli.

@maxlath The current version of wikibase-cli would need some adaptations to make it suited for large-volume updates:

option to take commands from file: I think the overhead of starting wd for every operation would be significant
some way to add refs/qualifiers together with the claim, rather than the 2-step process described at https://github.com/maxlath/wikibase-cli/blob/master/docs/write_operations.md#with-a-reference
a --quiet option to reduce output to only error messages
some reporting at which line the error occurred (so I can restart from there)
option to stop at first error, or to continue

VladimirAlexiev commented 4 years ago

So far no luck with the QS Batch tool (see https://www.wikidata.org/wiki/Topic:Vfnr6v8lfqagpi7g)

maxlath commented 4 years ago

not to say that it can't be improved, but a lot of what you ask is already possible in some way (out of the QS syntax)

option to take commands from file: I think the overhead of starting wd for every operation would be significant

there is certainly an overhead, but in my experience, it has been acceptable: the bottleneck remains Wikibase API

some way to add refs/qualifiers together with the claim, rather than the 2-step process described at https://github.com/maxlath/wikibase-cli/blob/master/docs/write_operations.md#with-a-reference

it's already possible using wb edit-entity, I especially recommand you to have a look at the dynamic template, which allow to edit any part of an entity in a single edit, and with extreme flexibility (see this example)

a --quiet option to reduce output to only error messages

errors are written on the process stderr, so redirecting stdout to /dev/null should work. ex: wd sl Q4115189 foo bar > /dev/null

some reporting at which line the error occurred (so I can restart from there) option to stop at first error, or to continue

errors make the process exit with a non-zero code, so if you where to have a script like

set -e
wd sl Q4115189 foo bar
wd sl Q4115189 nl bar

It would print an error to stderr and exit the script thank to the set -e bash parameter. You could then edit your script to fix the error and remove all the lines prior to the erronous command so that they aren't run again

magnusmanske commented 4 years ago

FWIW, I fixed the QS (OAuth) issue. If you are logged into QS, you can find instructions here.

maxlath commented 4 years ago

it would be too much work to add support for QS syntax, especially to maintain, as that would require staying up to date with QS evolutions, but:

for most cases, it should be fairly easy to convert a file from the QS syntax to wikibase-cli commands syntax (ping me for help if needed)
an external lib re-using wikibase-cli, or more probably directly wikibase-edit, would be welcome, and I could even help with one or the other's API
in some no too distant future, we could still have our own mass import syntax, probably just mimicking the wikibase-edit API in a newline-delimited JSON syntax (one operation per line), but I don't have the time to work on that now

maxlath commented 4 years ago

I gave a shot to a batch mode, feedback welcome

VladimirAlexiev commented 4 years ago

@maxlath thanks a lot, seems perfect! After my QS batches finish (recently tge has been a speed up due to a fixed deadlock issue), I'll collect the leftover (which will be in the 100,000s) and run with your batch mode.

https://github.com/maxlath/wikidata-scripting/blob/master/worldcat_identities_ids/README.md is also great. Can't one use nohup to ensure the script runs uninterrupted?

maxlath commented 3 years ago

A QuickStatements issue has been re-opened on wikibase-edit repo https://github.com/maxlath/wikibase-edit/issues/64

maxlath / wikibase-cli

Support QuickStatements syntax #62