Add Windows compatibility

[agrenott]

New multiprocessing usage is not compatible with native Windows Python.

Current work-around is to rely on Python inside WSL.

[treee111]

I also had this error now while trying your fork in Windows due to the linked issue. As reference for others, this is how it looks like when crashing:

  File "C:\Users\<username>\AppData\Local\anaconda3\envs\gdal-dev-pyh\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\<username>\AppData\Local\anaconda3\envs\gdal-dev-pyh\lib\runpy.py", line 86, in _run_code     
    exec(code, run_globals)
  File "C:\Users\<username>\AppData\Local\anaconda3\envs\gdal-dev-pyh\Scripts\pyhgtmap.exe\__main__.py", line 4, in <module>
  File "C:\Users\<username>\AppData\Local\anaconda3\envs\gdal-dev-pyh\lib\site-packages\pyhgtmap\main.py", line 12, in <module>
    from pyhgtmap.hgt.processor import HgtFilesProcessor
  File "C:\Users\<username>\AppData\Local\anaconda3\envs\gdal-dev-pyh\lib\site-packages\pyhgtmap\hgt\processor.py", line 3, in <module>
    from multiprocessing.context import ForkProcess
ImportError: cannot import name 'ForkProcess' from 'multiprocessing.context' (C:\Users\<username>\AppData\Local\anaconda3\envs\gdal-dev-pyh\lib\multiprocessing\context.py)

[bmffb]

I have installed pyhgtmap on a Windows 11 PC using the wheel provided on pypi.org. The installation is confirmed (by using pyhgtmap -h) and works using a Windows batch script to run (with input variables) :

call pyhgtmap ^
--jobs=1 ^
--polygon=%POLY_DATA% ^
--step=%ELE_STEP% ^
--line-cat=%ELE_MED_LRG% ^
--source=%DATA_SOURCE% ^
--no-zero-contour ^
--start-node-id=2000000000 ^
--start-way-id=2000000000 ^
--max-nodes-per-tile=0 ^
--max-nodes-per-way=250 ^
--simplifyContoursEpsilon=0.00002 ^
--hgtdir=%DIR_HGT% ^
--pbf ^
--output-prefix=%MAP_NAME%_ele_data

BUT: Using a start-node-id or a start-way-id with more than 10 digits I get the following error

Traceback (most recent call last):
  File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python39\Scripts\pyhgtmap.exe\__main__.py", line 7, in <module>
  File "C:\Python39\lib\site-packages\pyhgtmap\main.py", line 66, in main
    main_internal(sys.argv[1:])
  File "C:\Python39\lib\site-packages\pyhgtmap\main.py", line 58, in main_internal
    HgtFilesProcessor(opts.nJobs, opts.startId, opts.startWayId, opts).process_files(
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 281, in process_files
    self.process_file(file_name, check_poly)
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 239, in process_file
    self.process_tile(file_name, tile)
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 212, in process_tile
    self.process_tile_internal(file_name, tile)
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 150, in process_tile_internal
    new_start_id, ways = osm_output.write_nodes(
  File "C:\Python39\lib\site-packages\pyhgtmap\output\pbfUtil.py", line 148, in write_nodes
    return next_node_id, pyhgtmap.output.build_efficient_ways(ways)
  File "C:\Python39\lib\site-packages\pyhgtmap\output\__init__.py", line 143, in build_efficient_ways
    return numpy.array(
OverflowError: Python int too large to convert to C long

As I need at least 11 digit numbers for elevation nodes and ways (to not get into conflict with osm-based nodes and ways in my map generation), this is a kind of show-stopper for me. It sounds like the reason could be a 32-bit numpy array limit (I have numpy 1.26.4 installed on a 64-bit Windows 11 PC). A workaround would be needed.

[bmffb]

Addition: An error occurs also with any negative Ids.

[agrenott]

Hi, I'm actually surprised it reaches this far before crashing, as I don't expect pyhgtmap to be compatible with windows sur to the multiprocessing limitation.

Anyway, for your case I suspect a 32 bits version of python/numpy. You may check with https://stackoverflow.com/questions/1405913/how-do-i-determine-if-my-python-shell-is-executing-in-32bit-or-64bit#1405971

[bmffb]

I checked my version: Python runs in 64 bit. For numpy I tried to install the newest 64 bit version (after download of the wheel file) https://files.pythonhosted.org/packages/5f/9f/fe311331410759da4d441d6d08dd54b80065f4946374e817611f4f9c527f/numpy-2.0.0-pp39-pypy39_pp73-win_amd64.whl but when I try to install it using pip I get an error:

C:\Python39>py -m pip install numpy-2.0.0-pp39-pypy39_pp73-win_amd64.whl
ERROR: numpy-2.0.0-pp39-pypy39_pp73-win_amd64.whl is not a supported wheel on this platform.

Strange, as I am running 64 bit python on a 64 bit Windows PC. Are there other installation sources?

[agrenott]

pp39 denotes wheel built for PyPy, no CPython. You should try with numpy-2.0.0-cp39-cp39-win_amd64.whl instead - but I understand pip should already take the 64 bits wheel for a 64 bits python by default.

[bmffb]

Thanks for the hint. I was now able to install numpy-2.0.0 successfully. But, my attempt running pyhgtmap 3.7 with numpy 2.0.0 ended in this error message:

Traceback (most recent call last):
  File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python39\Scripts\pyhgtmap.exe\__main__.py", line 4, in <module>
  File "C:\Python39\lib\site-packages\pyhgtmap\main.py", line 13, in <module>
    from pyhgtmap.hgt.processor import HgtFilesProcessor
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 10, in <module>
    from pyhgtmap.output.factory import get_osm_output
  File "C:\Python39\lib\site-packages\pyhgtmap\output\__init__.py", line 7, in <module>
    from nptyping import NDArray, Structure
  File "C:\Python39\lib\site-packages\nptyping\__init__.py", line 32, in <module>
    from nptyping.ndarray import NDArray
  File "C:\Python39\lib\site-packages\nptyping\ndarray.py", line 40, in <module>
    from nptyping.shape import Shape
  File "C:\Python39\lib\site-packages\nptyping\shape.py", line 29, in <module>
    from nptyping.shape_expression import (
  File "C:\Python39\lib\site-packages\nptyping\shape_expression.py", line 36, in <module>
    from nptyping.typing_ import ShapeExpression, ShapeTuple
  File "C:\Python39\lib\site-packages\nptyping\typing_.py", line 51, in <module>
    Bool8 = np.bool8
  File "C:\Python39\lib\site-packages\numpy\__init__.py", line 439, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'bool8'

Is this an issue of pyhgtmap in general? Or something Windows specific?

[agrenott]

Indeed, looks like bool8 has been removed in numpy 2. You have to install 1.x. or upgrade other packages accordingly (nptyping here).

[bmffb]

I checked and I have nptyping installed in the newest version (2.5.0).

[agrenott]

There's no release of nptyping compatible with numpy 2.0 yet. https://github.com/ramonhagenaars/nptyping/pull/114

[bmffb]

OK. I went down to numpy 1.26.4 and get now this error message with pyhgtmap:

Traceback (most recent call last):
  File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python39\Scripts\pyhgtmap.exe\__main__.py", line 7, in <module>
  File "C:\Python39\lib\site-packages\pyhgtmap\main.py", line 66, in main
    main_internal(sys.argv[1:])
  File "C:\Python39\lib\site-packages\pyhgtmap\main.py", line 58, in main_internal
    HgtFilesProcessor(opts.nJobs, opts.startId, opts.startWayId, opts).process_files(
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 281, in process_files
    self.process_file(file_name, check_poly)
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 239, in process_file
    self.process_tile(file_name, tile)
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 212, in process_tile
    self.process_tile_internal(file_name, tile)
  File "C:\Python39\lib\site-packages\pyhgtmap\hgt\processor.py", line 150, in process_tile_internal
    new_start_id, ways = osm_output.write_nodes(
  File "C:\Python39\lib\site-packages\pyhgtmap\output\o5mUtil.py", line 271, in write_nodes
    return writeNodes(self, tile_contours, timestamp_string, start_node_id)
  File "C:\Python39\lib\site-packages\pyhgtmap\output\o5mUtil.py", line 304, in writeNodes
    return newId, pyhgtmap.output.build_efficient_ways(ways)
  File "C:\Python39\lib\site-packages\pyhgtmap\output\__init__.py", line 143, in build_efficient_ways
    return numpy.array(
OverflowError: Python int too large to convert to C long

There is a file generated, but it contains only the nodes of contours. The ways are not written. So I am back to the start of the problem.

[agrenott]

This is indeed a Windows specific issue, as on Windows numpy.int isn't 64 bits by default, while it is on other platforms: https://stackoverflow.com/a/57828594/23420157