Open znmeb opened 5 years ago
It looks like there's more stuff missing from the Clear Linux GIS stack. The workhorse of GDAL, ogr2ogr
, isn't linked with all the libraries it needs to have. For example, this fails on Clear Linux:
ogr2ogr -f geojson -t_srs EPSG:4326 tm_route_stops.geojson tm_route_stops.shp
ERROR 6: Unable to load PROJ.4 library (libproj.so), creation of OGRCoordinateTransformation failed.
ERROR 1: Failed to create coordinate transformation between the
following coordinate systems. This may be because they
are not transformable, or because projection services
(PROJ.4 DLL/.so) could not be loaded.
ERROR 1: Source:
PROJCS["NAD_1983_HARN_StatePlane_Oregon_North_FIPS_3601_Feet_Intl",
GEOGCS["GCS_North_American_1983_HARN",
DATUM["NAD83_High_Accuracy_Reference_Network",
SPHEROID["GRS_1980",6378137.0,298.257222101]],
PRIMEM["Greenwich",0.0],
UNIT["Degree",0.0174532925199433],
AUTHORITY["EPSG","4269"]],
PROJECTION["Lambert_Conformal_Conic_2SP"],
PARAMETER["False_Easting",8202099.737532808],
PARAMETER["False_Northing",0.0],
PARAMETER["Central_Meridian",-120.5],
PARAMETER["Standard_Parallel_1",44.33333333333334],
PARAMETER["Standard_Parallel_2",46.0],
PARAMETER["Latitude_Of_Origin",43.66666666666666],
UNIT["Foot",0.3048]]
ERROR 1: Target:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4326"]]
ERROR 1: Terminating translation prematurely after failed
translation of layer tm_route_stops (use -skipfailures to skip errors)
ldd
lists the libraries ogr2ogr
knows about:
$ ldd /usr/bin/ogr2ogr
linux-vdso.so.1 (0x00007ffce4bf2000)
libgdal.so.20 => /usr/lib64/libgdal.so.20 (0x00007f3f01c57000)
libstdc++.so.6 => /usr/lib64/haswell/libstdc++.so.6 (0x00007f3f01a4c000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f3f01a31000)
libc.so.6 => /usr/lib64/haswell/libc.so.6 (0x00007f3f01841000)
libodbc.so.2 => /usr/lib64/libodbc.so.2 (0x00007f3f017d0000)
libodbcinst.so.2 => /usr/lib64/libodbcinst.so.2 (0x00007f3f017b8000)
libhdf5.so.103 => /usr/lib64/libhdf5.so.103 (0x00007f3f0141c000)
libjpeg.so.62 => /usr/lib64/haswell/libjpeg.so.62 (0x00007f3f01376000)
libtiff.so.5 => /usr/lib64/libtiff.so.5 (0x00007f3f012e1000)
libpng16.so.16 => /usr/lib64/haswell/libpng16.so.16 (0x00007f3f01299000)
libz.so.1 => /usr/lib64/haswell/libz.so.1 (0x00007f3f01270000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f3f0124e000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f3f01246000)
libm.so.6 => /usr/lib64/haswell/libm.so.6 (0x00007f3f01102000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f3f02a48000)
libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00007f3f010f6000)
libwebp.so.7 => /usr/lib64/haswell/libwebp.so.7 (0x00007f3f01051000)
libzstd.so.1 => /usr/lib64/haswell/libzstd.so.1 (0x00007f3f00f96000)
liblzma.so.5 => /usr/lib64/haswell/liblzma.so.5 (0x00007f3f00f68000)
By contrast, on Debian stretch
, ogr2ogr
has many more libraries, including the one that's missing that caused the above failure:
$ ldd /usr/bin/ogr2ogr
linux-vdso.so.1 (0x00007ffd65df7000)
libgdal.so.20 => /usr/lib/libgdal.so.20 (0x00007f6581987000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f6581605000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6581301000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f65810ea000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6580ecd000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6580b2e000)
libarmadillo.so.7 => /usr/lib/libarmadillo.so.7 (0x00007f6580925000)
libproj.so.12 => /usr/lib/x86_64-linux-gnu/libproj.so.12 (0x00007f65806bb000)
libpoppler.so.64 => /usr/lib/x86_64-linux-gnu/libpoppler.so.64 (0x00007f6580225000)
libjson-c.so.3 => /lib/x86_64-linux-gnu/libjson-c.so.3 (0x00007f658001a000)
libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f657fe10000)
libqhull.so.7 => /usr/lib/x86_64-linux-gnu/libqhull.so.7 (0x00007f657fbb3000)
libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f657f984000)
libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f657f723000)
libepsilon.so.1 => /usr/lib/x86_64-linux-gnu/libepsilon.so.1 (0x00007f657f50a000)
libodbc.so.2 => /usr/lib/x86_64-linux-gnu/libodbc.so.2 (0x00007f657f29c000)
libodbcinst.so.2 => /usr/lib/x86_64-linux-gnu/libodbcinst.so.2 (0x00007f657f087000)
libkmlbase.so.1 => /usr/lib/x86_64-linux-gnu/libkmlbase.so.1 (0x00007f657ee6b000)
libkmldom.so.1 => /usr/lib/x86_64-linux-gnu/libkmldom.so.1 (0x00007f657ebab000)
libkmlengine.so.1 => /usr/lib/x86_64-linux-gnu/libkmlengine.so.1 (0x00007f657e971000)
libkmlxsd.so.1 => /usr/lib/x86_64-linux-gnu/libkmlxsd.so.1 (0x00007f657e75c000)
libkmlregionator.so.1 => /usr/lib/x86_64-linux-gnu/libkmlregionator.so.1 (0x00007f657e551000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f657e327000)
libxerces-c-3.1.so => /usr/lib/x86_64-linux-gnu/libxerces-c-3.1.so (0x00007f657dd9c000)
libopenjp2.so.7 => /usr/lib/x86_64-linux-gnu/libopenjp2.so.7 (0x00007f657db61000)
libnetcdf.so.11 => /usr/lib/x86_64-linux-gnu/libnetcdf.so.11 (0x00007f657a7f2000)
libhdf5_serial_hl.so.100 => /usr/lib/x86_64-linux-gnu/libhdf5_serial_hl.so.100 (0x00007f657a5ce000)
libsz.so.2 => /usr/lib/x86_64-linux-gnu/libsz.so.2 (0x00007f657a3cb000)
libhdf5_serial.so.100 => /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.100 (0x00007f6579e63000)
libmfhdfalt.so.0 => /usr/lib/libmfhdfalt.so.0 (0x00007f6579c3b000)
libdfalt.so.0 => /usr/lib/libdfalt.so.0 (0x00007f6579991000)
libogdi.so.3.2 => /usr/lib/libogdi.so.3.2 (0x00007f657976f000)
libgif.so.7 => /usr/lib/x86_64-linux-gnu/libgif.so.7 (0x00007f6579565000)
libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f65792fa000)
libgeotiff.so.2 => /usr/lib/x86_64-linux-gnu/libgeotiff.so.2 (0x00007f65790c3000)
libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f6578e4b000)
libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f6578c18000)
libpq.so.5 => /usr/lib/x86_64-linux-gnu/libpq.so.5 (0x00007f65789ce000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f65787a8000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f65785a0000)
libdapclient.so.6 => /usr/lib/x86_64-linux-gnu/libdapclient.so.6 (0x00007f6578358000)
libdapserver.so.7 => /usr/lib/x86_64-linux-gnu/libdapserver.so.7 (0x00007f6578148000)
libdap.so.23 => /usr/lib/x86_64-linux-gnu/libdap.so.23 (0x00007f6577dab000)
libspatialite.so.7 => /usr/lib/x86_64-linux-gnu/libspatialite.so.7 (0x00007f6577622000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f65773af000)
libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f6577130000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f6576d75000)
libmariadbclient.so.18 => /usr/lib/x86_64-linux-gnu/libmariadbclient.so.18 (0x00007f657676f000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6576555000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6576351000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6582c16000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f65760e4000)
liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007f657583f000)
libarpack.so.2 => /usr/lib/libarpack.so.2 (0x00007f65755f4000)
libsuperlu.so.5 => /usr/lib/x86_64-linux-gnu/libsuperlu.so.5 (0x00007f6575383000)
liblcms2.so.2 => /usr/lib/x86_64-linux-gnu/liblcms2.so.2 (0x00007f6575127000)
libnss3.so => /usr/lib/x86_64-linux-gnu/libnss3.so (0x00007f6574ddd000)
libnssutil3.so => /usr/lib/x86_64-linux-gnu/libnssutil3.so (0x00007f6574baf000)
libsmime3.so => /usr/lib/x86_64-linux-gnu/libsmime3.so (0x00007f6574982000)
libssl3.so => /usr/lib/x86_64-linux-gnu/libssl3.so (0x00007f657472a000)
libplds4.so => /usr/lib/x86_64-linux-gnu/libplds4.so (0x00007f6574526000)
libplc4.so => /usr/lib/x86_64-linux-gnu/libplc4.so (0x00007f6574321000)
libnspr4.so => /usr/lib/x86_64-linux-gnu/libnspr4.so (0x00007f65740e1000)
libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007f6573ea3000)
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f6573bf4000)
libgeos-3.5.1.so => /usr/lib/x86_64-linux-gnu/libgeos-3.5.1.so (0x00007f6573849000)
libpopt.so.0 => /lib/x86_64-linux-gnu/libpopt.so.0 (0x00007f657363c000)
libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f6573432000)
libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f6573226000)
liburiparser.so.1 => /usr/lib/x86_64-linux-gnu/liburiparser.so.1 (0x00007f657300b000)
libkmlconvenience.so.1 => /usr/lib/x86_64-linux-gnu/libkmlconvenience.so.1 (0x00007f6572dec000)
libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f6572bd4000)
libicui18n.so.57 => /usr/lib/x86_64-linux-gnu/libicui18n.so.57 (0x00007f657275a000)
libicuuc.so.57 => /usr/lib/x86_64-linux-gnu/libicuuc.so.57 (0x00007f65723b2000)
libicudata.so.57 => /usr/lib/x86_64-linux-gnu/libicudata.so.57 (0x00007f6570935000)
libaec.so.0 => /usr/lib/x86_64-linux-gnu/libaec.so.0 (0x00007f657072d000)
libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f657051f000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f65702b3000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f656fe1a000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f656fbcf000)
libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f656f97e000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f656f779000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f656f471000)
libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f656f24b000)
libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f656f029000)
librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f656ee0c000)
libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f656ebdf000)
libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f656e9d1000)
libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f656e79a000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f656e401000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f656e127000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f656def4000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f656dcf0000)
liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f656dae1000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f656d7bb000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f656d57c000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f656d370000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f656d16c000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f656cf55000)
libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f656cd3a000)
libunistring.so.0 => /usr/lib/x86_64-linux-gnu/libunistring.so.0 (0x00007f656ca23000)
libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f656c7ee000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f656c56b000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f656c25b000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f656bff6000)
libidn.so.11 => /lib/x86_64-linux-gnu/libidn.so.11 (0x00007f656bdc2000)
libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f656bbaf000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f656b99b000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f656b792000)
If you have specific functionality that is missing, I will be happy to add it. However, I really don't like to go fishing for kitchen sinks - it is a tedious and time consuming task.
@bryteise can we consider making R-extras-dev
perhaps?
I'd rather we not add more *-dev things and instead stick to devpkgs but this might not be too possible in a nice timeline to do otherwise. I'll poke it in.
@znmeb Let me know if this is working on Thursday's release.
OK... by the way, I went through the package building tools last night (https://clearlinux.org/documentation/clear-linux/guides/maintenance/autospec#setup-environment-to-build-source and (https://clearlinux.org/documentation/clear-linux/guides/maintenance/autospec#id12) and was able to reproduce your build process for gdal
. It looks like you set up a mock system, install an entire stack required to build gdal
with dnf
downloading binary RPMs from some repository I haven't located yet.
Then you build a source RPM and all the binary RPMs the spec
file mandates. Once you have that, it looks like you throw away everything except the binary RPMs. If I want to build proj
or geos
or qgis
, I have to go through the whole process again!
The modern Linux GIS stack is an integrated collection of packages, their -dev
or -devel
and debug symbol binary packages. At the low level, there's gdal
, geos
and proj
, there are GUI executables like qgis
and grass
, the PostGIS extension to PostgreSQL and Python and R bindings to most of it.
I would just set up the build process for maker-gis
so it builds the whole stack in one fell swoop and bundles it. Otherwise, people are just going to use Ubuntu or Debian containers and not get the benefits of Clear's optimizations for Intel hardware.
ok so can we do this please
can we describe/get to a list/whatever of what maker-gis should contain. (in terms of files/whatever)
once we have that we can figure out how to get it there; generally that's easy enough :)
we have the current maker-gis content. what would you like to see added/removed from that? (lets try to make a list, we may need to iterate a few times but that's ok)
There's an Ubuntu-based distribution that has all the open source GIS packages out there, even things I rarely / never use: https://live.osgeo.org/en/overview/overview.html
I would say at a minimum PostgreSQL 11 (you have 9.6), PostGIS 2.5, pgRouting 2.6, pgAdmin4, QGIS 3 and all the libraries and main programs required to make those work. I know the package names for Debian / Ubuntu and Arch Linux.
I haven't installed all of this on Fedora in quite a while. If you have everything in https://yum.postgresql.org/news-packagelist.php plus QGIS and all its plugins you should be OK. You only need the latest stable PostgreSQL (11) and the latest stable PostGIS and pgRouting - if someone wants an older one they can use a container.
The PostgreSQL Global Development Group (PGDG) has source RPMs for Fedora 30 at https://yum.postgresql.org/srpms/11/fedora/fedora-30-x86_64/. I don't know your build system well enough yet to figure out how much work it'll be to get all of that built. There are source RPMs for QGIS as well - I'd start with Fedora 30.
@znmeb do you know how to make checkboxes in github?
- [ ] - item 1
turns into:
I'd appreciate it a ton(!) if you could edit your OP and add a list of check box items for this issue, so we can slowly churn through all of them one by one and get this to closure at some point :)
@ahkok I haven't scoped it out into a work breakdown structure - I was planning to make a local bundle with all the stuff I need and post that to a GitHub repo.
To build
rgdal
andrgeos
, you need header files for thegdal
,proj4
andgeos
Linux libraries. You have the binaries inR-extras
, and you have the headers forgdal
. But you don't have the headers forproj4
andgeos
. Symptoms: