Skip the level selection step

[antoine2711]

Description

The step where you choose the JSON import level can be skipped, it's always the same. The column name should come from the JSON fields name of the first record.

So the project name and tags could/should be asked the step before.

Screenshot

[wetneb]

Yes. I would expect this extension not to rely in the json importer at all actually, even if it retrieves the results in json internally

[WaltonG]

Sure. I think i get the logic

[WaltonG]

Hi @wetneb I am working on a custom json parser but am having challenges in understanding how the parsePreview and TabularImportingParserBase.readTable methods in the existing extensions work.

1. Should the raw data be passed to the initialize-parser-ui sub command?
2. How is the raw data handled by the parsePreview and TabularImportingParserBase.readTable methods?
3. How is the raw data then returned to the frontend for preview?

I have gone through several extensions but didnt understand how the raw data gets parsed before being displayed at the preview page

[wetneb]

Hi @WaltonG, The raw data is generally read from the backend directly. That means, in your case I would expect that the SPARQL request is not done by the frontend (as is currently the case) but rather by the backend.

To get a simple example, have you looked at the CommonsExtension? It is very similar, because it also lets people create projects from an API. In particular, the ImportingController should be interesting for you. I think it should answer your second question.

I would expect the general process to look like this.

• the frontend lets the user type in a SPARQL query (in the CommonsExtension: from a list of Wikimedia Commons categories)
• the user validates the form, which calls the preview sub-command of the importing controller defined by the extension;
• inside the server (in Java), this preview command calls the SPARQL endpoint (in CommonsExtension: the MediaWiki API) to execute the query, and transforms the query results into an OpenRefine project.
• the frontend displays this previewed project to the user (by retrieving the rows from the temporary project created by the preview command)
• when the user creates the project, that triggers again a request to the server, which then performs the SPARQL query again and creates the project out of it.

The initialize-parser-ui sub-command does not do any data processing itself, it just returns the default values for the options displayed in the preview stage.

As you can see from the CommonsExtension, the preview and project creation stage share a lot of the same code: the only difference between the two is the limit on the number of rows. In your case it could make sense to include this limit in the SPARQL query, if you have a safe way to do so (it might require parsing the SPARQL query).

One thing I would encourage you to get an overview of the requests that are made, is to try out the CommonsExtension with your web developer panel open, so that you can see the network requests made by the frontend to the backend (and inspect their parameters and responses).

Let me know if any of this is unclear.

[WaltonG]

Hi @wetneb I now have an understanding of what is expected and how to achieve that. In case of any clarifications I will reach out

[WaltonG]

@wetneb What could be the reason behind switching from Apache HttpClient in main OpenRefine to OkHttpClient in the CommonsExtension?

[wetneb]

I am not sure why @j-sal used one or the other, you could ask her directly. I think it would work with both.