Add modifier parameter to r.series

[landam]

Reported by pvanbosgeo on 14 Jan 2016 23:17 UTC An useful enhancement of r.series would be to add a "modifier" (or similar name) parameter to r.series which would allow to calculate the statistic defined by "method" on a modified version of the maps (e.g. squared, square root, log, sin, cos, etc.).

Attached patch is by Moritz Lennert, see also the email thread https://lists.osgeo.org/pipermail/grass-dev/2016-January/078470.html.

Migrated-From: https://trac.osgeo.org/grass/ticket/2867

[landam]

Attachment from pvanbosgeo on 14 Jan 2016 23:18 UTC patch to add modifier to r.series https://trac.osgeo.org/grass/attachment/ticket/2867/r_series_modifier.diff

[landam]

Modified by @landam on 16 Jan 2016 09:22 UTC

[landam]

Comment by glynn on 18 Jan 2016 22:06 UTC Replying to [ticket:2867 pvanbosgeo]:

An useful enhancement of r.series would be to add a "modifier" (or similar name) parameter to r.series which would allow to calculate the statistic defined by "method" on a modified version of the maps (e.g. squared, square root, log, sin, cos, etc.).

My main concern is that this is going to open the door to a never-ending stream of requests to add more modifiers or more features (e.g. parametric modifiers, or selecting modifiers on a per-output bases) until we end up having to embed either r.mapcalc or Python into r.series.

OTOH, I'm aware that the usual solution (use in conjunction with r.mapcalc) has a significant overhead when you're potentially dealing with hundreds of input maps. Even so, I'd be inclined to limit the available modifiers to those which seem likely to be particularly useful (only sqrt/x^2 and log/exp spring to mind); there are a lot of real->real functions in the standard math library.

As for the patch itself, I'd suggest:

1. Determining the function during initialisation and setting a function pointer, rather than performing the test in the inner loop. Multiple strcmp()s inside a triple-nested loop (rowscolumnsinputs) are likely to add significant overhead.

2. Vectorising the operations so that the dispatch is only done once per row, not per value. Even an indirect call via a function pointer may be more expensive than the function itself (some math functions may compile to a single FPU instruction). The simplest way to do this would be to modify inputs[].buf[] in-place after reading.

In that regard, maybe we should try to extract the relevant portions of r.mapcalc to a separate library so that they can be re-used here?

Or maybe a better option would be to just package up the raster-I/O loops as a Python module so that the actual processing can be implemented as Python functions which convert a 2D (inputs*columns) array into a set of 1D arrays (one per output).

[landam]

Comment by mlennert on 19 Jan 2016 10:45 UTC Replying to [comment:2 glynn]:

Replying to [ticket:2867 pvanbosgeo]:

An useful enhancement of r.series would be to add a "modifier" (or similar name) parameter to r.series which would allow to calculate the statistic defined by "method" on a modified version of the maps (e.g. squared, square root, log, sin, cos, etc.).

My main concern is that this is going to open the door to a never-ending stream of requests to add more modifiers or more features (e.g. parametric modifiers, or selecting modifiers on a per-output bases) until we end up having to embed either r.mapcalc or Python into r.series.

OTOH, I'm aware that the usual solution (use in conjunction with r.mapcalc) has a significant overhead when you're potentially dealing with hundreds of input maps. Even so, I'd be inclined to limit the available modifiers to those which seem likely to be particularly useful (only sqrt/x^2 and log/exp spring to mind);

+1

As for the patch itself, I'd suggest:

1. Determining the function during initialisation and setting a function pointer, rather than performing the test in the inner loop. Multiple strcmp()s inside a triple-nested loop (rowscolumnsinputs) are likely to add significant overhead.

+1

That's why I said in the mail accompanying the patch:

"Try the attached patch, which is just a brute-force, proof-of-concept. It would be nicer to solve this with function pointers, but I'm not at ease with that, so I'll leave that to others."

;-)

2. Vectorising the operations so that the dispatch is only done once per row, not per value. Even an indirect call via a function pointer may be more expensive than the function itself (some math functions may compile to a single FPU instruction). The simplest way to do this would be to modify inputs[].buf[] in-place after reading.

In that regard, maybe we should try to extract the relevant portions of r.mapcalc to a separate library so that they can be re-used here?

If that is not too much work, then this would probably be a good thing. But if it's only for adding a few modifiers to r.series is it worth the effort ?

Or maybe a better option would be to just package up the raster-I/O loops as a Python module so that the actual processing can be implemented as Python functions which convert a 2D (inputs*columns) array into a set of 1D arrays (one per output).

You've lost me there.

[landam]

Comment by pvanbosgeo on 19 Jan 2016 12:16 UTC Replying to [comment:2 glynn]:

Replying to [ticket:2867 pvanbosgeo]:

An useful enhancement of r.series would be to add a "modifier" (or similar name) parameter to r.series which would allow to calculate the statistic defined by "method" on a modified version of the maps (e.g. squared, square root, log, sin, cos, etc.).

My main concern is that this is going to open the door to a never-ending stream of requests to add more modifiers or more features (e.g. parametric modifiers, or selecting modifiers on a per-output bases) until we end up having to embed either r.mapcalc or Python into r.series.

OTOH, I'm aware that the usual solution (use in conjunction with r.mapcalc) has a significant overhead when you're potentially dealing with hundreds of input maps. Even so, I'd be inclined to limit the available modifiers to those which seem likely to be particularly useful (only sqrt/x^2 and log/exp spring to mind); there are a lot of real->real functions in the standard math library.

Completely get your point.. but if I may still add a possibly very particularly useful one: pow(). It could possible be included instead of sqrt and x^2.

As for the patch itself, I'd suggest:

1. Determining the function during initialisation and setting a function pointer, rather than performing the test in the inner loop. Multiple strcmp()s inside a triple-nested loop (rowscolumnsinputs) are likely to add significant overhead.

2. Vectorising the operations so that the dispatch is only done once per row, not per value. Even an indirect call via a function pointer may be more expensive than the function itself (some math functions may compile to a single FPU instruction). The simplest way to do this would be to modify inputs[].buf[] in-place after reading.

In that regard, maybe we should try to extract the relevant portions of r.mapcalc to a separate library so that they can be re-used here?

Or maybe a better option would be to just package up the raster-I/O loops as a Python module so that the actual processing can be implemented as Python functions which convert a 2D (inputs*columns) array into a set of 1D arrays (one per output).

[landam]

Comment by mlennert on 19 Jan 2016 13:33 UTC Replying to [comment:4 pvanbosgeo]:

Replying to [comment:2 glynn]:

Replying to [ticket:2867 pvanbosgeo]:

An useful enhancement of r.series would be to add a "modifier" (or similar name) parameter to r.series which would allow to calculate the statistic defined by "method" on a modified version of the maps (e.g. squared, square root, log, sin, cos, etc.).

My main concern is that this is going to open the door to a never-ending stream of requests to add more modifiers or more features (e.g. parametric modifiers, or selecting modifiers on a per-output bases) until we end up having to embed either r.mapcalc or Python into r.series.

OTOH, I'm aware that the usual solution (use in conjunction with r.mapcalc) has a significant overhead when you're potentially dealing with hundreds of input maps. Even so, I'd be inclined to limit the available modifiers to those which seem likely to be particularly useful (only sqrt/x^2 and log/exp spring to mind); there are a lot of real->real functions in the standard math library.

Completely get your point.. but if I may still add a possibly very particularly useful one: pow(). It could possible be included instead of sqrt and x^2.

The problem with pow() is that it needs to parameters and this would complicate things further.

Another option, maybe more in line with Glynn's suggestion of extracting some or r.mapcalc into a library would be to create a modification_expression parameter which would take any arbitrary formula using r.mapcalc syntax (e.g. pow(x, 4.5) or cos(x)/sqrt(x), etc) where x represents the map input to r.series. But IIUC that would +/- mean recreating r.mapcalc within r.series...

[landam]

Comment by glynn on 25 Jan 2016 09:30 UTC Replying to [comment:2 glynn]:

In that regard, maybe we should try to extract the relevant portions of r.mapcalc to a separate library so that they can be re-used here?

https://trac.osgeo.org/grass/changeset/67658 moves most of the r.mapcalc functions into lib/calc.

[landam]

Modified by @landam on 12 May 2016 06:37 UTC

[landam]

Modified by @landam on 25 Aug 2016 15:51 UTC

[landam]

Comment by @landam on 27 Aug 2016 13:42 UTC Milestone renamed

[landam]

Comment by neteler on 26 Jan 2018 11:40 UTC Ticket retargeted after milestone closed

[landam]

Modified by neteler on 12 Jun 2018 20:48 UTC

[landam]

Comment by @landam on 25 Sep 2018 16:52 UTC All enhancement tickets should be assigned to 7.6 milestone.

[landam]

Comment by @landam on 25 Jan 2019 21:07 UTC Ticket retargeted after milestone closed