Linux Installation Instructions

[AustinTSchaffer]

This repository has a requirements-linux.txt but doesn't seem to have explicit installation instructions for Linux. Please see below for the list of commands that I needed to run in order to install the dependencies for this project on my laptop. I already had the necessary Nvidia drivers and other libraries installed, so this list of commands is incomplete.

# Install OS package dependencies (APT instructions, I'm using Ubuntu)
# (taken from https://github.com/SciTools/cartopy/blob/main/INSTALL)
sudo apt -y install libproj-dev libgeos-dev

# Python version check
python --version
# Python 3.8.10

# setup venv
python -m venv venv
source venv/bin/activate

# Install requirements
python -m pip install -r requirements-linux.txt

JOSS review thread: https://github.com/openjournals/joss-reviews/issues/4079

[AustinTSchaffer]

Would you be open to a pull request to get the project working on Linux? I'm looking to make some modifications to the setup.py.

Even with the commands above, I was still not able to install or run this project, due to the setup.py using the contents of the Windows requirements.txt. I had to pre-install the requirements, then edit the setup.py to make sure it didn't try to install any packages. We could use import platform; platform.system() to detect Windows vs Linux, and use that to load the associated requirements file.

It appears to be semi-working now, but there's clearly still some work to do.

uname -a
# Linux austin-ub-xps 5.13.0-25-generic #26-Ubuntu SMP Fri Jan 7 15:48:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

spgr --help
# Usage: spgr [OPTIONS] COMMAND [ARGS]...
# 
# Options:
#   --help  Show this message and exit.
# 
# Commands:
#   map   Generate various raster maps
#   path  Generate paths minimising risk measures

spgr map strike 10 90 10 90
# ... huge stack trace
# ModuleNotFoundError: No module named '_bz2'

[aliaksei135]

Thanks for reviewing!

Currently much of the explicit Windows/Linux split is due to underlying C/C++ dependencies like GEOS and GDAL that have significantly different installation requirements on different platforms, which aren't covered in the setup.py of the packages that depend on them (and in turn that this work depends upon).

I would very much be open to a PR that could get this working again on Linux, using platform to set the correct requirements.txt seems like the right way to go. I did at some point have this working on WSL Ubuntu, with the exception of the CUDA accelerated routines as CUDA and WSL don't get along.

For the specific case above, I suspect that sudo apt install libbz2-dev would do it, however there will likely be more system dependencies required.

[kylebeggs]

I reached where Austin stopped, installed libbz2-dev and tried to re-run and now get this

/home/kylebeggs/.local/lib/python3.8/site-packages/geopandas/_compat.py:106: UserWarning: The Shapely GEOS version (3.8.0-CAPI-1.13.1 ) is incompatible with the GEOS version PyGEOS was compiled with (3.8.1-CAPI-1.13.3). Conversions between both will be slow.
  warnings.warn(
Traceback (most recent call last):
  File "/home/kylebeggs/.local/bin/spgr", line 8, in <module>
    sys.exit(main())
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/seedpod_ground_risk/cli/spgr.py", line 88, in strike
    pop_grid = make_pop_grid(bounds, hour, resolution)
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/seedpod_ground_risk/api/api.py", line 90, in make_pop_grid
    raster_shape = reproj_bounds(bounds, proj, resolution)
  File "/home/kylebeggs/.local/lib/python3.8/site-packages/seedpod_ground_risk/core/utils.py", line 38, in reproj_bounds
    raster_width = int(abs(max_x - min_x) // raster_resolution_m)
ValueError: cannot convert float NaN to integer

Looks like the GEOS that is installed via Apt is not playing nice.

[AustinTSchaffer]

To add to this list, end users may need to sudo apt-get install libbz2-dev, possibly reinstall Python afterwards if using pyenv, and run git lfs fetch --all and/or git lfs checkout. After performing those additional setup steps, I was able to get this command running locally for some constrained Lat/Lon containing Dudley, UK (chosen at random):

spgr map pop-density 52.480246 52.5304 -2.109753 -2.025296

I can't make sense of the output since I'm not a SME in this area. I've attached a screenshot of the resulting TIF.

[aliaksei135]

This is an 'issue' with the image viewer not being able to pull out specific geospatial data from the file and happens if you open the file in Windows Photos as well.

The file is a geoTIFF format which shares the same file extension as the plain TIFF image format, so most non-GIS viewers will try and open it as the plain format.

Try opening the file in a web viewer like: http://app.geotiff.io/

There should be a single alpha channel indicating the heatmap values.

---

From: Austin T Schaffer @.> Sent: Monday, February 21, 2022 10:12:23 PM To: aliaksei135/seedpod_ground_risk @.> Cc: Aliaksei Pilko @.>; Comment @.> Subject: Re: [aliaksei135/seedpod_ground_risk] Linux Installation Instructions (Issue #100)

To add to this list, end users may need to sudo apt-get install libbz2-dev, possibly reinstall Python afterwards if using pyenv, and run git lfs fetch --all and/or git lfs checkout. After performing those additional setup steps, I was able to get this command running locally for some constrained Lat/Lon containing Dudley, UK (chosen at random):

spgr map pop-density 52.480246 52.5304 -2.109753 -2.025296

I can't make sense of the output since I'm not a SME in this area. I've attached a screenshot of the resulting TIF.

[image]https://user-images.githubusercontent.com/16184219/155031873-98c38c21-ba52-403f-af09-f2aa7e22241c.png

— Reply to this email directly, view it on GitHubhttps://github.com/aliaksei135/seedpod_ground_risk/issues/100#issuecomment-1047261794, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA6VC46MI6Y3ZCGZFFSYGFLU4K2EPANCNFSM5OQNHPCQ. You are receiving this because you commented.Message ID: @.***>

[AustinTSchaffer]

Pretty cool! I've managed to run all of the sub-commands under the spgr map command, results are below. Do you have any documentation for the spgr path command? I'm not really sure what to supply for the subcommand and args.

Usage: spgr path [OPTIONS] COMMAND [ARGS]...

Generate paths minimising risk measures

Options: --help Show this message and exit.

(click to expand) spgr map pop-density 52.480246 52.5304 -2.109753 -2.025296


(click to expand) spgr map fatality 52.480246 52.5304 -2.109753 -2.025296


(click to expand) spgr map strike 52.480246 52.5304 -2.109753 -2.025296


[aliaksei135]

Those look much better and like they should!

For spgr path, looks like I forgot to add the make subcommand to it. I will fix. Fixed in #103

Thanks @AustinTSchaffer for #102 , I have successfully built the wheels on a Linux box with an Xserver, instead of WSL. I will update the build instructions with more up to date commands. Fixed in #104

I have arrived at what I think is the list of dependencies that require installing from apt, at least for the wheels to build and CLI functionality:

sudo apt install libgeos-dev libgdal-dev proj-bin libbz2-dev

Let me know if this list is missing any, if not I will merge #104

[AustinTSchaffer]

@aliaksei135 I added a few comments to #104, but overall it looks good. Thanks for updating the documentation!

[kylebeggs]

@aliaksei135 I am still having trouble running a command. I seem to have installed as I can run spgr --help with success. However, when I run the same command as Austin I get this error now:

> spgr map pop-density 52.480246 52.5304 -2.109753 -2.025296  
/home/kylebeggs/spgr/lib/python3.8/site-packages/geopandas/_compat.py:106: UserWarning: The Shapely GEOS version (3.8.0-CAPI-1.13.1 ) is incompatible with the GEOS version PyGEOS was compiled with (3.8.1-CAPI-1.13.3). Conversions between both will be slow.
  warnings.warn(
Traceback (most recent call last):
  File "fiona/_shim.pyx", line 83, in fiona._shim.gdal_open_vector
  File "fiona/_err.pyx", line 291, in fiona._err.exc_wrap_pointer
fiona._err.CPLE_OpenFailedError: 'static_data/england_wa_2011_clipped.shp' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kylebeggs/spgr/bin/spgr", line 8, in <module>
    sys.exit(main())
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/seedpod_ground_risk/cli/spgr.py", line 46, in pop_density
    raster_grid = make_pop_grid(bounds, hour, resolution)
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/seedpod_ground_risk/api/api.py", line 92, in make_pop_grid
    layer.preload_data()
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/seedpod_ground_risk/layers/temporal_population_estimate_layer.py", line 33, in preload_data
    self._ingest_census_data()
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/seedpod_ground_risk/layers/temporal_population_estimate_layer.py", line 153, in _ingest_census_data
    census_wards_df = gpd.read_file(england_wa_2011_clipped_filepath()).drop(['altname', 'oldcode'], axis=1)
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/geopandas/io/file.py", line 160, in _read_file
    with reader(path_or_bytes, **kwargs) as features:
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/fiona/env.py", line 408, in wrapper
    return f(*args, **kwargs)
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/fiona/__init__.py", line 256, in open
    c = Collection(path, mode, driver=driver, encoding=encoding,
  File "/home/kylebeggs/spgr/lib/python3.8/site-packages/fiona/collection.py", line 162, in __init__
    self.session.start(self, **kwargs)
  File "fiona/ogrext.pyx", line 540, in fiona.ogrext.Session.start
  File "fiona/_shim.pyx", line 90, in fiona._shim.gdal_open_vector
fiona.errors.DriverError: 'static_data/england_wa_2011_clipped.shp' not recognized as a supported file format.

[AustinTSchaffer]

@kylebeggs you need to use git lfs to download the actual contents of a few files in the repo.

@aliaksei135 could you add that to the documentation along with the instructions for cloning the repo? Sorry, forgot about that part during the PR

[aliaksei135]

@kylebeggs we had this exact issue before, in fact @Zach10a asked this on SO here

Unfortunately a few of the data files are just above the limit, will add to build docs.

[kylebeggs]

@aliaksei135 ok great, that solved it!

[aliaksei135]

@AustinTSchaffer @kylebeggs Have all the issues in this thread been fixed? If so, I think we can close this :)

[AustinTSchaffer]

@aliaksei135 yep!