QGIS Processing standalone executable

[nyalldawson]

QGIS Enhancement: QGIS Processing standalone executable

Date 2019/02/27

Author Nyall Dawson (@nyalldawson)

Contact nyall dot dawson at gmail dot com

maintainer @nyalldawson

Version >= QGIS 3.8

Summary

QGIS Processing offers a rich and expandable set of algorithms which can operate on map layers. This proposal covers creation of a new command-line executable which would allow for execution of processing algorithms outside of the QGIS GUI application, allowing for easy scripting of data manipulation via traditional batch files, shell scripts, and via operating system tools such as task scheduler.

Additionally, this command-line tool would make it easier for 3rd party applications to call and utilise Processing algorithms without the need to create custom PyQGIS scripts to achieve this.

Proposed Solution

UPDATE: the final tool is called qgis_process and has subcommands, all examples here have been updated to reflect the changes A new executable will be built alongside QGIS, tentatively named "qgis_transform" "qgis_process". The executable will be built whenever a new cmake setting is enabled (defaults to on):

SET (WITH_QGIS_PROCESS TRUE CACHE BOOL "Determines whether the standalone QGIS transform tool should be built")

The executable will be linked to the core, analysis and qgispython libraries only (i.e. no GUI component). If the WITH_3D setting is enabled, the 3d library will also be linked (in order to utilise the native 3D algorithms processing provider).

(It is undecided whether the executable will require WITH_BINDINGS to be enabled -- without the Python bindings the tool would be of limited use, as only native c++ algorithms would be available (and the Python-based model provider would not be available).)

On Windows, the qgis_process executable will utilise the same approach as the main qgis application executable to handle automatic setting of environment variables, allowing it to be run directly from a standard command line without any manual path or environment manipulation.

Tool usage

Running the qgis_process command without any additional arguments will output a basic help text, describing the available commands and use of the tool.

Listing algorithms

Running qgis_process list will output a complete list of all available algorithms, grouped by provider. The list will output the algorithm ID followed by it's display name, e.g.

QGIS (native c++)

native:buffer  Buffer
native:centroids Centroids
native:collect Collect geometries

QGIS (3D)

3d:tesselate Tesselate

Algorithm help

Running qgis_process help with a single argument representing an algorithm id will output the help text for that algorithm. Eg qgis_process help native:buffer will output:

This output mostly follows the output of the existing Python processing.algorithmHelp function, with the exception that the "Accepted data types" will not be included (it is Python specific), and will be replaced with a new "Acceptable values" section. This section will be populated by retrieving parameter metadata via QgsApplication::processingRegistry()::parameterType() and a new virtual method acceptableCommandLineValues method to be added to the QgsProcessingParameterType class. E.g. for QgsProcessingParameterVectorLayer parameters this will output a string "full path to a vector layer file", for QgsProcessingParameterBoolean parameters it will output "1 for true, 0 for false", etc.

Running an algorithm

To run an algorithm, the qgis_process run tool will be run with the first argument specifying a valid algorithm id, followed by a set of --parameter=value arguments.

E.g., to buffer a layer:

qgis_process run native:buffer --INPUT=/home/me/my.shp --DISTANCE=20 --OUTPUT=/home/me/buffered.shp

A new QgsProcessingFeedback subclass QgsConsoleFeedback will be used to directly output all messages from the algorithm to the console, and provide a text-based progress bar as the algorithm executes.

If the algorithm execution reports success the executable will return 0, else 1 (and any reported errors will be output to the console).

If algorithm execution is successful, the tool will output a text-based list of the algorithm results. (Possibly in future this list could be exported in a machine-readable format such as JSON, to make it easier for external tools calling qgis_process to handle the results).

Running a model or script

A potential future inclusion could be the ability to directly run a model or script based algorithm from a file, e.g.

qgis_process /home/me/my_models/do_stuff.model3 --INPUT=/home/me/my.shp  --OUTPUT=/home/me/buffered.shp

or

qgis_process /home/me/my_scripts/do_stuff.py --INPUT=/home/me/my.shp  --OUTPUT=/home/me/buffered.shp

Performance Implications

N/A

Further Considerations/Improvements

One important consideration is how to handle custom python plugins which expose Processing algorithms. Because the tool will NOT use a gui application, any attempt to create a QWidget in the plugin will cause a crash.

Possibly a new method will be created for plugin interface to "createAlgorithms", which will be called by the executable instead of the standard plugin hooks. Plugins would need to ensure that no GUI operations are performed by the createAlgorithms hook, or in the default plugin constructor. A blacklist of plugins which caused crashes at previous use of the tool could be used to avoid loading plugins which previously crashed due to widget creation issues.

Backwards Compatibility

N/A

Issue Tracking ID(s)

https://issues.qgis.org/issues/8955

Votes

(required)

[NathanW2]

I haven't read all this yet but pretty much going to be a massive +1 from me on this

[nirvn]

+1,000 on the concept, hoping we'll find a better executable file name by the time it's implemented :)

qgis_cli ? qgis_process?

Also will it support in place editing?

[nirvn]

qgis_toolbox? :)

[nyalldawson]

Also will it support in place editing?

No plans to - I don't think there's a strong use case for that (e.g. the gdal command line tools don't support this)

qgis_toolbox

I'm after a "verb" based name so that it's more descriptive of the actually functionality. qgis_process isn't bad though!

[eurojam]

+1: a brilliant idea, which will offer a bunch of new possibilities for ggis. My first thought was also qgis_process or qgis_cmd.
Stefan

[pcav]

+1, excellent idea - we'll occupy yet another GIS niche

[pcav]

Why not qgis_processing? I think Processing is a well established name for us.

[rduivenvoorde]

+10, was thinking about this too, when visiting a city where they use fme batch scripts A LOT [0]

But is it really needed to create another exe? qgis is nice and short :-) Or else qgis_run?

Another argument to not create a separate exe was that in my view it would be ubercool if you could create a QGIS-project with a model in it and then run the model from the command line.

So (in future) we could create models in gui, save a project and then run it as batchfiles. I think we use our strength (being a user friendly gui) then (I also use this argument to 'promote' QGIS-server).

Or an easier use-case: I create a project, and use qgis_run -p world.qgz -bbox 4,50,6,53 to create a map?

Or run the model on the data in the given project to create a nightly dump of some transformation: qgis_run -p nl.qgz -o fresh_dump.gpkg

Anyway, (my) idea's are cheap, go with whatever you think is doable!

[0] https://knowledge.safe.com/articles/975/batch-processing-method-1-command-line-or-batch-fi.html

[nyalldawson]

I create a project, and use qgis_run -p world.qgz -bbox 4,50,6,53 to create a map?

That's already supported by the main QGIS executable -- see https://docs.qgis.org/2.18/en/docs/user_manual/introduction/getting_started.html#command-line-options

:wink:

But is it really needed to create another exe?

Yes - because the intention here is to allow it to be built and installed as a separate package, independent from all GUI components. So e.g. on a server you could install without requiring an xserver, etc.

Another argument to not create a separate exe was that in my view it would be ubercool if you could create a QGIS-project with a model in it and then run the model from the command line.

Well, the separate executable still allows you to create models and scripts from the gui, and then "publish" them for use where only the qgis_transform (or whatever it's called) qgis_process executable is available. I think initially this will only be supported via the

qgis_process /home/me/my_models/do_stuff.model3 --INPUT=/home/me/my.shp  --OUTPUT=/home/me/buffered.shp

type syntax, but i'm sure there's a way to allow execution of models embedded in projects also.

[nyalldawson]

Why not qgis_processing? I think Processing is a well established name for us.

I'm after a "verb" here, so qgis_process would work better. (or qgis_exec, qgis_run, something along those lines)

[andreasneumann]

or we could save a bit of typing and call it "qprocess" (loosing the marketing effect towards QGIS, however)

[anvpires]

+1, excellent idea!

[rduivenvoorde]

qgisx ? Still have marketing but just 5 chars, AND using the x which always stays on your scrabble plate :-)

[luipir]

yes please :)

about the name... I would leave qgis in the name for marketing and to distinguish among other CLI commands... qtransform o qprocess can be manage using aliases.

[andreasneumann]

qgisx ? Still have marketing but just 5 chars, AND using the x which always stays on your scrabble plate :-)

I like "qgisx" - short, contains qgis and sound very mysterious ;-)

[roya0045]

Qgis_processor?

If you need a hand I'd be glad to assist however I can.

[nyalldawson]

qgisx

I'm not personally such a fan of this -- it sounds more like some "x windows" related variant of qgis, and won't be very obvious in package lists (I intend on this being a standalone package in linux distros -- e.g. "qgis-process")

[PedroVenancio]

A huge +1 from me!

Outstanding idea!

About the name, qgis_process looks good to me.

Thanks @nyalldawson !!

[roya0045]

Just a quick question, I wanted to implement an iterator in the modeler, would it be a good idea to make an iterator in qgis for things such as features, layers, files, int, list of other things) for this standalone and for the modeler?

This is an advantage in the commercial software that would be useful in QGIS for the standalone and the modeler.

Any opinion on this Mr. Dawson?

[nyalldawson]

@roya0045 sweet! Have you seen https://github.com/qgis/QGIS-Enhancement-Proposals/issues/108 ? I'd suggest resurrecting that discussion (ideally by starting with a survey of how ArcGIS/FME models handle iteration for comparison purposes).

[timlinux]

Huge +1 from me on this too!

[giohappy]

Same from me! +1

[dmarteau]

So e.g. on a server you could install without requiring an xserver, etc.

Some kind of headless X (Xvfb) rendering may be needed if your algorithm is going to render raster.

[dmarteau]

@nyalldawson why not build such tool in python ? Most of the processing ecosystem is in python.

[roya0045]

@dmarteau things are being ported to c++ quite expediently.

[nyalldawson]

why not build such tool in python ? Most of the processing ecosystem is in python

And one thing that's taught us is that python is great for small plugins and scripts, but just not a reliable choice for a large complex project like qgis!

[dmarteau]

And one thing that's taught us is that python is great for small plugins and scripts, but just not a reliable choice for a large complex project like qgis!

mmh, this is not exactly about rewriting qgis in python :-).

From my experience with qgis processing in python, it does not involve a large amount of code to handle headless processing algorithms in python - thanks to the current C++ API - and most of the processing managment is already written in python.

My POV it is that Qgis expose a very large and complete python API, and using it to write some useful qgis toolbox utilities is appealing. I was Just thinking about the most efficient way to get things done.

[nyalldawson]

This is actually all already written in c++. I just need funding to add some unit tests and some fixes for windows to auto-detect the qgis environment variables. See https://github.com/nyalldawson/QGIS/commit/631460cd1b8481cb118b7df2afacc5c3df40b3ad

One benefit of not doing this in python is that it doesn't even require a python to run. You can build it with python disabled and still run all the native C++ algorithms.

But on a more general note

most of the processing managment is already written in python

And it's the most fragile of the remaining processing code. Python just doesn't scale for bug projects like qgis. It's the lack of compile time checks which means that maintenance becomes impossible, and that's why the python code is a constant source of regressions. More unit tests of the python code would help, but honestly, compile time checks are worth 1000 manually added unit tests.