[QUESTION] Is it possible to run 'action=cargoquery' queries?

[BrunoBlanes]

I want to connect to a service that uses Cargo on their databases. Can I use your library to run queries directly on those databases without having to manually type out the query through this method?

WikiSite.InvokeMediaWikiApiAsync(WikiRequestMessage, CancellationToken)

[CXuesong]

Can I use your library to run queries directly on those databases without having to manually type out the query through this method?

Sure thing. But I'm not sure what did you mean by "manually type out the query"… I've checked the API documentation of cargoquery, but there is no much example there.

Anyway, generally speaking, you can check out the usage of this method inside WCL and it may offer you some context. You can raise your questions here if any.

[BrunoBlanes]

I mean that cargoquery supports a few SQL clauses, is it possible to not type that out and instead use something like LINQ? I mean something like EF Core, a wrapper that makes you not have to type the query yourself.

[CXuesong]

I see… That will be something to implement.

• Cargo uses a SQL-based query syntax: mw:Extension:Cargo/Querying data. For calling cargoquery API, I assume we'll need similar syntax to fill in the parameters.
• It would be nice (and interesting!) to have a complete custom EF provider (with some limitation posed, such as disabling all the EF editing API) based on Cargo API. But this does need some investigation.
• Another approach could be us writing some simple and sweet Expression-based query string generator instead of leveraging EF, but I'm a bit worry about whether I need to dig down into processing Expression tree by myself (this does needs some work).

Unfortunately I'm not familiar with this extension, nor I do have sufficient time to play with it at the moment (We used SMW a bit, before switching to Wikibase). Do you have some example Special:ApiSandbox links that I can get familiar with it quickly?

[BrunoBlanes]

I am working on this Special:ApiSandbox, here are the tables. I have started working on a wrapper for that API specifically using the latter approach with Expression, but then I found your library.

[CXuesong]

Okay I'll try with custom EF provider. I'll need some time to investigate. Then my question is: what is your priority? Like is it okay (i.e. helps most of your scenario) if we only have tables, select and where implemented in the MVP?

[BrunoBlanes]

order_by would be nice, but those are indeed sufficient.

[CXuesong]

Got it.

btw I've tried a simple query just now and I got MWException: https://lol.gamepedia.com/Special:ApiSandbox#action=cargoquery&format=json&tables=Skins&fields=Page%2CName&where=DATEDIFF(ReleaseDate%2C%20%7Bd'2010-1-1'%7D)%20%3C%200

[CXuesong]

Okay I've made it right now: https://lol.gamepedia.com/Special:ApiSandbox#action=cargoquery&format=json&tables=Skins&fields=_pageName%3DPage%2CName&where=DATEDIFF(ReleaseDate%2C%20%7Bd'2010-1-1'%7D)%20%3C%200

So I cannot really have an exception telling me exactly which part in my input is incorrect, right?

[BrunoBlanes]

Yes, so when there's an error in your query, the response message isn't particularly useful, I don't know why that is. Also, about the use of _pageName, one of the devs of Leguepedia wrote this article that I did not dig into just yet, but when querying tables through Cargo, I usually assign them a variable instead.

[CXuesong]

And now I'm starting to think leveraging EF Core is not an option as it does not support breaking down the built SQL query in to clauses (tables, fields, where, order_by, etc.). This is essential for sending cargoquery API request.

DefaultQuerySqlGenerator looks like the class responsible for building query clauses but we do not have enough information (dependency) to instantiate it.

Then we'll need to parse into Expression tree and build SQL clauses by ourselves. This looks like a pretty isolated problem to solve. Care to take a look at it?

I have started working on a wrapper for that API specifically using the latter approach with Expression, but then I found your library.

I also have interest in solving this problem. Perhaps I'll start a repo on it some day 😂 what we need is some Expression-based light-weight SQL clause generator without the dependency of EF Core.

That being said, I'll consider create a new WCL package for sending cargoquery request with given API parameters.

[BrunoBlanes]

So I cannot really have an exception telling me exactly which part in my input is incorrect, right?

Apparently when you run the query on the wiki it does return a pretty error message:

{{#cargo_query:tables=Skins
|fields=Page,Name
|where=DATEDIFF(ReleaseDate, {d'2010-1-1'}) < 0
}}

Error: No field named "Page" found for any of the specified database tables.

[BrunoBlanes]

Also, here's the complete documentation for creating queries using Cargo.

[CXuesong]

Let's give this PR a couple of weeks to get done 😂

[BrunoBlanes]

Wish I was able to code like that so that I could help, but my skills are currently comparable to those of a cat trying to catch a laser light point.

[CXuesong]

Nah frankly speaking Expression-related techniques are esoteric indeed. And my LOL skill is also nil

So there is still a lot to learn!

[BrunoBlanes]

Hope you don't mind, but I've shared the PR with the Leaguepedia community, and the main dev has looked at it and might leave some reviews, she is also trying to help. Not trying to pressure you, so, take your time and the reviews as you'd like.

[RheingoldRiver]

Yes, so when there's an error in your query, the response message isn't particularly useful, I don't know why that is.

For security reasons it's not wanted to print the raw error message without any parsing at all, because that could expose non-Cargo errors from the server, and not a lot of work was done to determine and return safe error messages in the cargoquery method. So if you want to figure out what's wrong with your query you have to avoid the API and enter your query into the wiki via another method (the parser function or Special:CargoQuery, I prefer doing the former on a sandbox page (that page will resolve to your sandbox page if you're logged in).

Unfortunately this leads to now a lot of duplicated effort in a lot of API libraries to do this error handling for the user....

One thing that might be useful is a method to print/log the wikitext parser function version of the fully-constructed query so that the user is able to debug on their wiki. The documentation for that is located here. I do this in my Lua Cargo wrapper on Leaguepedia (wikitextQuery / logQuery), it's the most useful logging ever.

Hopefully this helps, I'm happy to talk about Cargo any time! 😄

[CXuesong]

Okay so I found some details about Cargo to ask. Let's switch to #79 to discuss about it. @BrunoBlanes @RheingoldRiver

[CXuesong]

Released WikiClientLibrary.Cargo v0.7.5.