Closed AustinTSchaffer closed 2 years ago
Would you be open to a pull request to get the project working on Linux? I'm looking to make some modifications to the setup.py
.
Even with the commands above, I was still not able to install or run this project, due to the setup.py
using the contents of the Windows requirements.txt
. I had to pre-install the requirements, then edit the setup.py
to make sure it didn't try to install any packages. We could use import platform; platform.system()
to detect Windows vs Linux, and use that to load the associated requirements file.
It appears to be semi-working now, but there's clearly still some work to do.
uname -a
# Linux austin-ub-xps 5.13.0-25-generic #26-Ubuntu SMP Fri Jan 7 15:48:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
spgr --help
# Usage: spgr [OPTIONS] COMMAND [ARGS]...
#
# Options:
# --help Show this message and exit.
#
# Commands:
# map Generate various raster maps
# path Generate paths minimising risk measures
spgr map strike 10 90 10 90
# ... huge stack trace
# ModuleNotFoundError: No module named '_bz2'
Thanks for reviewing!
Currently much of the explicit Windows/Linux split is due to underlying C/C++ dependencies like GEOS
and GDAL
that have significantly different installation requirements on different platforms, which aren't covered in the setup.py
of the packages that depend on them (and in turn that this work depends upon).
I would very much be open to a PR that could get this working again on Linux, using platform
to set the correct requirements.txt
seems like the right way to go. I did at some point have this working on WSL Ubuntu, with the exception of the CUDA accelerated routines as CUDA and WSL don't get along.
For the specific case above, I suspect that sudo apt install libbz2-dev
would do it, however there will likely be more system dependencies required.
I reached where Austin stopped, installed libbz2-dev and tried to re-run and now get this
/home/kylebeggs/.local/lib/python3.8/site-packages/geopandas/_compat.py:106: UserWarning: The Shapely GEOS version (3.8.0-CAPI-1.13.1 ) is incompatible with the GEOS version PyGEOS was compiled with (3.8.1-CAPI-1.13.3). Conversions between both will be slow.
warnings.warn(
Traceback (most recent call last):
File "/home/kylebeggs/.local/bin/spgr", line 8, in <module>
sys.exit(main())
File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/kylebeggs/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/kylebeggs/.local/lib/python3.8/site-packages/seedpod_ground_risk/cli/spgr.py", line 88, in strike
pop_grid = make_pop_grid(bounds, hour, resolution)
File "/home/kylebeggs/.local/lib/python3.8/site-packages/seedpod_ground_risk/api/api.py", line 90, in make_pop_grid
raster_shape = reproj_bounds(bounds, proj, resolution)
File "/home/kylebeggs/.local/lib/python3.8/site-packages/seedpod_ground_risk/core/utils.py", line 38, in reproj_bounds
raster_width = int(abs(max_x - min_x) // raster_resolution_m)
ValueError: cannot convert float NaN to integer
Looks like the GEOS that is installed via Apt is not playing nice.
To add to this list, end users may need to sudo apt-get install libbz2-dev
, possibly reinstall Python afterwards if using pyenv, and run git lfs fetch --all
and/or git lfs checkout
. After performing those additional setup steps, I was able to get this command running locally for some constrained Lat/Lon containing Dudley, UK (chosen at random):
spgr map pop-density 52.480246 52.5304 -2.109753 -2.025296
I can't make sense of the output since I'm not a SME in this area. I've attached a screenshot of the resulting TIF.
This is an 'issue' with the image viewer not being able to pull out specific geospatial data from the file and happens if you open the file in Windows Photos as well.
The file is a geoTIFF format which shares the same file extension as the plain TIFF image format, so most non-GIS viewers will try and open it as the plain format.
Try opening the file in a web viewer like: http://app.geotiff.io/
There should be a single alpha channel indicating the heatmap values.
From: Austin T Schaffer @.> Sent: Monday, February 21, 2022 10:12:23 PM To: aliaksei135/seedpod_ground_risk @.> Cc: Aliaksei Pilko @.>; Comment @.> Subject: Re: [aliaksei135/seedpod_ground_risk] Linux Installation Instructions (Issue #100)
To add to this list, end users may need to sudo apt-get install libbz2-dev, possibly reinstall Python afterwards if using pyenv, and run git lfs fetch --all and/or git lfs checkout. After performing those additional setup steps, I was able to get this command running locally for some constrained Lat/Lon containing Dudley, UK (chosen at random):
spgr map pop-density 52.480246 52.5304 -2.109753 -2.025296
I can't make sense of the output since I'm not a SME in this area. I've attached a screenshot of the resulting TIF.
— Reply to this email directly, view it on GitHubhttps://github.com/aliaksei135/seedpod_ground_risk/issues/100#issuecomment-1047261794, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA6VC46MI6Y3ZCGZFFSYGFLU4K2EPANCNFSM5OQNHPCQ. You are receiving this because you commented.Message ID: @.***>
Pretty cool! I've managed to run all of the sub-commands under the spgr map
command, results are below. Do you have any documentation for the spgr path
command? I'm not really sure what to supply for the subcommand and args.
Usage: spgr path [OPTIONS] COMMAND [ARGS]...
Generate paths minimising risk measures
Options: --help Show this message and exit.
Those look much better and like they should!
For spgr path
, looks like I forgot to add the make
subcommand to it. I will fix. Fixed in #103
Thanks @AustinTSchaffer for #102 , I have successfully built the wheels on a Linux box with an Xserver, instead of WSL. I will update the build instructions with more up to date commands. Fixed in #104
I have arrived at what I think is the list of dependencies that require installing from apt
, at least for the wheels to build and CLI functionality:
sudo apt install libgeos-dev libgdal-dev proj-bin libbz2-dev
Let me know if this list is missing any, if not I will merge #104
@aliaksei135 I added a few comments to #104, but overall it looks good. Thanks for updating the documentation!
@aliaksei135 I am still having trouble running a command. I seem to have installed as I can run spgr --help
with success. However, when I run the same command as Austin I get this error now:
> spgr map pop-density 52.480246 52.5304 -2.109753 -2.025296
/home/kylebeggs/spgr/lib/python3.8/site-packages/geopandas/_compat.py:106: UserWarning: The Shapely GEOS version (3.8.0-CAPI-1.13.1 ) is incompatible with the GEOS version PyGEOS was compiled with (3.8.1-CAPI-1.13.3). Conversions between both will be slow.
warnings.warn(
Traceback (most recent call last):
File "fiona/_shim.pyx", line 83, in fiona._shim.gdal_open_vector
File "fiona/_err.pyx", line 291, in fiona._err.exc_wrap_pointer
fiona._err.CPLE_OpenFailedError: 'static_data/england_wa_2011_clipped.shp' not recognized as a supported file format.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/kylebeggs/spgr/bin/spgr", line 8, in <module>
sys.exit(main())
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/seedpod_ground_risk/cli/spgr.py", line 46, in pop_density
raster_grid = make_pop_grid(bounds, hour, resolution)
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/seedpod_ground_risk/api/api.py", line 92, in make_pop_grid
layer.preload_data()
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/seedpod_ground_risk/layers/temporal_population_estimate_layer.py", line 33, in preload_data
self._ingest_census_data()
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/seedpod_ground_risk/layers/temporal_population_estimate_layer.py", line 153, in _ingest_census_data
census_wards_df = gpd.read_file(england_wa_2011_clipped_filepath()).drop(['altname', 'oldcode'], axis=1)
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/geopandas/io/file.py", line 160, in _read_file
with reader(path_or_bytes, **kwargs) as features:
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/fiona/env.py", line 408, in wrapper
return f(*args, **kwargs)
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/fiona/__init__.py", line 256, in open
c = Collection(path, mode, driver=driver, encoding=encoding,
File "/home/kylebeggs/spgr/lib/python3.8/site-packages/fiona/collection.py", line 162, in __init__
self.session.start(self, **kwargs)
File "fiona/ogrext.pyx", line 540, in fiona.ogrext.Session.start
File "fiona/_shim.pyx", line 90, in fiona._shim.gdal_open_vector
fiona.errors.DriverError: 'static_data/england_wa_2011_clipped.shp' not recognized as a supported file format.
@kylebeggs you need to use git lfs
to download the actual contents of a few files in the repo.
@aliaksei135 could you add that to the documentation along with the instructions for cloning the repo? Sorry, forgot about that part during the PR
@kylebeggs we had this exact issue before, in fact @Zach10a asked this on SO here
Unfortunately a few of the data files are just above the limit, will add to build docs.
@aliaksei135 ok great, that solved it!
@AustinTSchaffer @kylebeggs Have all the issues in this thread been fixed? If so, I think we can close this :)
@aliaksei135 yep!
This repository has a
requirements-linux.txt
but doesn't seem to have explicit installation instructions for Linux. Please see below for the list of commands that I needed to run in order to install the dependencies for this project on my laptop. I already had the necessary Nvidia drivers and other libraries installed, so this list of commands is incomplete.JOSS review thread: https://github.com/openjournals/joss-reviews/issues/4079