r.connectivity.distance: implement walking distance and parallel processing

[ninsbl]

Include also #1091

[ecodiv]

@ninsbl I tested it using the example from the manual page, resulting in the following error:

gs.run_command(
...     "r.connectivity.distance",
...     flags="k",
...     input="patches_1ha",
...     pop_proxy="area_ha",
...     costs="costs",
...     prefix="hws_connectivity2",
...     cutoff=4500,
...     border_dist=18,
...     conefor_dir="./conefor",
...     nprocs=10,
... )
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1225, in <module>
    sys.exit(main())
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1142, in main
    pool.starmap(compute_distances_parallel, cat_chunks)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/utils.py", line 154, in __getattr__
    return self[key]
KeyError: '__getstate__'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 487, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 366, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
  File "/usr/local/grassmaster/grass85/etc/python/grass/exceptions/__init__.py", line 87, in __init__
    err = _("See errors above the traceback or in the error output.")
TypeError: 'int' object is not callable

[ninsbl]

Hm... The eror you see is likely related to the number of processes (nprocs). Could you try with nprocs=4 for example? With 10 cores I get a different error message (with GRASS 8.3) . But with 4 cores it runs fine...

[ecodiv]

Hm... The eror you see is likely related to the number of processes (nprocs). Could you try with nprocs=4 for example? With 10 cores I get a different error message (with GRASS 8.3) . But with 4 cores it runs fine...

Unfortunately, for me it doesn't work with 2 or 4 cores either. One core does work.

>>> gs.run_command(
...     "r.connectivity.distance",
...     flags="pk",
...     input="patches_1ha",
...     pop_proxy="area_ha",
...     costs="costs",
...     prefix="hws_connectivity4",
...     cutoff=1000,
...     border_dist=18,
...     nprocs=2
... )
Traceback (most recent call last):
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1225, in <module>
    sys.exit(main())
  File "/home/paulo/.grass8/addons/scripts/r.connectivity.distance", line 1142, in main
    pool.starmap(compute_distances_parallel, cat_chunks)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/utils.py", line 154, in __getattr__
    return self[key]
KeyError: '__getstate__'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 487, in run_command
    return handle_errors(returncode, result=None, args=args, kwargs=kwargs)
  File "/usr/local/grassmaster/grass85/etc/python/grass/script/core.py", line 366, in handle_errors
    raise CalledModuleError(module=module, code=code, returncode=returncode)
grass.exceptions.CalledModuleError: Module run `r.connectivity.distance -pk input=patches_1ha pop_proxy=area_ha costs=costs prefix=hws_connectivity4 cutoff=1000 border_dist=18 nprocs=2` ended with an error.
The subprocess ended with a non-zero return code: 1. See errors above the traceback or in the error output.

[echoix]

Is one of you using a platform other than Linux by any chance? If so, maybe the use of multiprocessing.Pool is using the start method other than fork, like spawn, that requires that what is ran to be Pickle-able. Sometimes the file needs to be reorganized, or even the part that is ran by multiple processes be specially designed and extracted in another file to be executed correctly.

The traceback shown doesn't properly mention the real error for me.

Alternatively, is there any chance that the Python version is 3.12 running on Linux, and using a multiprocessing function using the fork startup method also calls a sub process, such as it raises a DepreciationWarning (it can cause deadlocks and the default startup method will change to spawn in 3.14), and that this depreciationwarning is saved to a results file, making it invalid?

[echoix]

Using multiprocessing correctly, especially in a cross platform manner, is a little bit trickier than one can imagine

[ecodiv]

Alternatively, is there any chance that the Python version is 3.12 running on Linux,

I am running version 3.10.12, the default on Ubuntu 22.04

[ecodiv]

@echoix @ninsbl I don't know if it is helpful, but the attached log.txt shows the full debug output when setting debug to level 5.