search

DHI-GRAS / qgis-processing-gpf

QGIS Processing provider for GPF-based algorithms (BEAM and Sentinel-1 Toolbox)
GNU General Public License v3.0
2 stars 5 forks source link

LinearToFromdB still causing S1_preprocessing.xml to fail #11

Closed j08lue closed 8 years ago

j08lue commented 8 years ago

The LinearToFromdB operator name change is still causing problems, even after @radosuav's recent fix. Processing the S1_preprocessing.xml GPF XML file fails at QGIS startup:

untitled

I can see that there is a text file that still has the old name: https://github.com/DHI-GRAS/processing_gpf/blob/e540cdf48b4196a106170875c06034290819955f/s1tbx_description/LinearTodB.txt but that is probably not the reason for the failure?

j08lue commented 8 years ago

OK, I tried renaming that QGIS header .txt file and this was not it.

Is it intentional that it is not renamed and this line still reads s1tbx_LinearTodB?

radosuav commented 8 years ago

Are you sure that it fails? Check if the model is present in the Processing toolbox. Those messages are sometimes displayed because the models might be loaded before the other algorithms, but at the end the models should be reloaded and everything should work.

As far as I remember the filename is not important. And I'll have to check the code to see what the second line of the description file does.

j08lue commented 8 years ago

Thanks for getting back so quickly, @radosuav!

Telling from that description file, the model should end up under SNAP Toolbox > Raster, right? It does not and searching for Linear... gives nothing.

image

I checked which algorithms/operators are in self.algs in getAlgorithmFromOperator and these were

['BandMaths', 'Read', 'Reproject', 'Subset', 'Write']

I can see that self.algs gets populated here and that loadGpfModels should add an error message to the log if GPFModelerAlgorithm.fromFile fails, which is does not.

But we must be processing S1_preprocessing.xml, because the above models do get loaded.

Very puzzling...

j08lue commented 8 years ago

OK, I now understand what you mean by some algorithms get loaded earlier than others. Algorithms actually get loaded many times from many places, which also explains why this error comes up four times. This is terrible...

So, the algorithms that do get loaded come from here, which seems to scan the snap_generic_description folder only.

Why does it not go on to the others, i.e. why are ProcessingConfig.getSetting(GPFUtils.S1TBX_ACTIVATE) and ProcessingConfig.getSetting(GPFUtils.S2TBX_ACTIVATE) never True in _loadAlgorithms...? But that is a different story, I guess.

But self.loadGpfModels(GPFUtils.modelsFolder()) does get called, but it fails at GPFModelerAlgorithm.fromFile() which raises a WrongModelException which gets caught and logged.

A nightmare debugging! :-O Errors are caught and forwarded from everywhere and functions call each other in loops...

So, final question for tonight :-), isn't GPFModelerAlgorithm.fromFile the method that should parse and register the LinearFromTodB operator in the first place? Why does it attempt to load the same operator (at which point it fails)?

j08lue commented 8 years ago

So this is a problem of the relative dependencies of the algorithms. If one requires the other, that other has to be loaded already? But dependencies have not changed here, as far as I can see. So why does this fail now?

j08lue commented 8 years ago

Well, I guess I have been looking the wrong place all the time. I can see that LinearToFromdB should be provided by the S1tbx, which is not activated in my QGIS. So no wonder the operator cannot be found. When I try to activate S1tbx, I get

image

So maybe this is related to the old GRASS problem from ESA_processing https://github.com/DHI-GRAS/ESA_Processing/issues/11?

j08lue commented 8 years ago

Wait, I just tested on a clean installation and S1tbx and S2tbx can be activated without error. See https://github.com/TIGER-NET/WOIS_Installer/issues/6 .

So this was not an issue after all. Sorry @radosuav and @ctottrup for the 500 notifications you must have gotten... ;-)

radosuav commented 8 years ago

The situation with loading of algorithms and what happens where is a bit confusing so I'll try to clarify.

Firstly, the only algorithms which depend on others are the ones in modeler, workflows and gpf modeler. This is because they are composed of those other algorithms. Therefore, those other algorithms must already be loaded before models/workflows are loaded. So, https://github.com/DHI-GRAS/processing_gpf/blob/GW-A_dev/GPFModelerAlgorithm.py#L198 does not load any algorithms (they are loaded in https://github.com/DHI-GRAS/processing_gpf/blob/GW-A_dev/SNAPAlgorithmProvider.py#L93 which is called from https://github.com/DHI-GRAS/ESA_Processing/blob/master/core/Processing.py#L173), but just tries to find a pointer to an already loaded one using https://github.com/DHI-GRAS/processing_gpf/blob/GW-A_dev/SNAPAlgorithmProvider.py#L87. If this algorithm does not exist, then the model cannot be created and an exception is raised.

The situation with models composed just of algorithms coming from core providers (i.e. providers loaded here https://github.com/DHI-GRAS/ESA_Processing/blob/master/core/Processing.py#L127) and with gpf models is simple because it is easy to ensure that the models are loaded last, after all other algorithms have already been loaded (e.g. https://github.com/DHI-GRAS/processing_gpf/blob/GW-A_dev/SNAPAlgorithmProvider.py#L105).

Non-core algorithm providers get added after the models are loaded. Therefore, they are loaded with updateList=True which reloads all of the previous providers and updates the models (follow the calling chain from here https://github.com/DHI-GRAS/ESA_Processing/blob/master/core/Processing.py#L91). This means that algorithms are loaded multiple times and that's why multiple messages are recorded in the log.

Similarly, workflow provider can be loaded before all non-core providers are loaded. However, it is not possible to add the workflow update function directly to Processing. Therefore WorkflowAlgListListener is used (https://github.com/DHI-GRAS/processing_workflow/blob/GW-A_dev/WorkflowAlgListListener.py), and it gets triggered from Processing (https://github.com/DHI-GRAS/ESA_Processing/blob/master/core/Processing.py#L160) whenever algorithm list gets updated.

There was code cleanup recently to simply this situation (see https://github.com/qgis/QGIS/commit/e925382292e3927fd881890beacb484ee3d82452 and subsequent commits in https://github.com/qgis/QGIS/commits/master/python/plugins/processing/core) but those changes are not in LTR so will not affect WOIS for some time.