Create a searchable list of available gqueries for API users

[redekok]

Our API documentation contains a non-working link referring to the list of available gqueries. Multiple people using our API have yet pointed that out to me. Would it be possible to have some kind of autogeneration script that creates a list of available gqueries (with description) based on the gqueries in ETSource? @antw Do you have any thoughts on this?

(Not sure what the best repository is to post this issue btw..)

[noracato]

I created a simple GQuery API in quintel/etengine@c24722a2a826079d68dfa3dc0665c3ef71c4cb68 that retrieves all possible queries. As we have almost 5000 queries, we need a way to categorise them in order to be usefull to our users.

We thought about using the ETSource path for this. So for example costs_of_insulation will be categorised as General -> Costs, as this query is located on ETSource in these (sub)folders. With this we can create a small interface in the documentation (with some nicer css of course):

In order to build this we should:

• Extend the basic GQuery API to accept parameters indicating a category and subcategory (or section)
• Create a component in the documentation that either (1) sends a request to the API every time the form changes or (2) already has an (internal) repesentation of all possible queries from which it retrieves the correct queries (retrieves this list form the API at a set time/on new build, etc)

I also thought of a search bar, but I think having a search bar in the documentation website forces us to go in the direction of option (2). Another posibility is to host this table on ETEngine instead of at the documentation. However, I feel it would be nice to keep everything about the API in one place. Also we'd need to create a (separate) non-scenario-specific, non-login-wall part of ETEngine just for this purpose.

[antw]

I also thought of a search bar

A search feature would be nice although, as you say, it means either the browser fetching all 5000 queries (which might be a very large payload?) and then filtering client-side, or sending the search query to ETEngine and doing the filtering there. I expect we'll at least have to fetch a list of categories from ETEngine to populate the drop-downs?

(retrieves this list form the API at a set time/on new build, etc)

Creating the list at Docusaurus build-time means it may become out-of-sync with the current state of ETEngine. It would be unfortunate to have to remember to rebuild Docusaurus whenever a commit is imported in ETEngine, though it may be possible to automate this.

[noracato]

Good points!

Yes, the search feature would be very nice! I think filtering in ETEngine might be slightly faster. The payload of 5000 queries and their descriptions is around 675 KB of transferred data (not that large, but also not small).

Then: should we go for only a search bar, or combine the dropdowns with a search bar? If we would only go for a search bar, the Gquery API will not need an extra endpoint for categories, nor category parameters.

[antw]

675 KB isn't as bad as I expected (especially if the data is gzipped), since it won't be used that frequently.

Losing categories would be a pity since it might help people discover queries they might not otherwise find, but I don't think it's a dealbreaker.