izahn
commented
3 years ago Yes, the conda lwgeom package is way out of date and needs help. @conda-forge/r @agcopenhaver maybe you can take a look and see what is holding up the updates?
Closed spittssj closed 3 years ago
izahn
commented
3 years ago Yes, the conda lwgeom package is way out of date and needs help. @conda-forge/r @agcopenhaver maybe you can take a look and see what is holding up the updates?
izahn
commented
3 years ago @spittssj We managed to update lwgeom for Windows and Linux, but we couldn't get the OSX build working. Maybe you can take a look?
spittssj
commented
3 years ago From the logfile here it seems like there is still an issue with linking to proj on the OS X build. https://dev.azure.com/conda-forge/84710dde-1620-425b-80d0-4cf5baca359d/_apis/build/builds/275815/logs/35
2021-02-09T19:10:08.2709670Z checking PROJ: checking whether PROJ and sqlite3 are available for linking:... yes
2021-02-09T19:10:08.4936960Z dyld: Library not loaded: @rpath/libproj.19.dylib
2021-02-09T19:10:08.4944570Z Referenced from: /Users/runner/miniforge3/conda-bld/r-lwgeom_1612897421165/work/./proj_conf_test
2021-02-09T19:10:08.4958840Z Reason: image not found
Later on POSTGIS_PROJ_VERSION is not defined and so you get the preprocessor error
invalid token at start of a preprocessor expression.
I have a deadline for a project tomorrow but I can take a look this weekend.
Did you have to update libproj to get the Windows and Linux versions to work? That is where I would start.
izahn
commented
3 years ago Did you have to update
libprojto get the Windows and Linux versions to work? That is where I would start.
libproj is installed via conda, and should be very recent. I'm a beginner myself at conda packaging, and I haven't had to build software on OS X for years, so I'm not the best person to offer advice. There is documentation at https://conda-forge.org/docs/maintainer/updating_pkgs.html#testing-changes-locally but I'm not sure how (or even if) that works on OS X. https://conda-forge.org/docs/maintainer/knowledge_base.html#testing-the-packages might also be useful.
izahn
commented
3 years ago I've tried again to get this working on OSX, without success. I tried setting environment variables as they do in https://github.com/conda-forge/r-rgdal-feedstock/blob/master/recipe/build.sh, and I tried building the development version of lwgoem. Nothing worked.
Logs from these attempts are available at https://dev.azure.com/izahn/my-conda-tests/_build?view=runs, but probably not really useful since again nothing worked...
@spittssj have you had any luck getting it to build on OSX locally?
spittssj
commented
3 years ago I spent the evening trying to rebuild locally, but I'm stuck in a cycle of dependencies. I still think that there are some problems on the OS X conda-forge side with GIS packages not being completely updated. The dependencies are really subtle! Here is what I have tried:
I followed the instructions here on manually building locally things from CRAN, and I used conda skeleton to rebuild lwgeom from scratch. I got the same link error with libproj that Azure reports about not finding the symbol on libproj.so.19, when not specifying the libproj: the dependency solver picked libproj 7.2.0.
So I edited the recipe manually to force it to use the latest version of libproj, which is 7.2.1 according to conda-forge. The dependency solver came up with a bunch of errors.
Side note: I look forward to the day when conda-build can use mamba's dependency solver. Every step here took 5-10 minutes on my new MacBook Pro, and the dependency conflict messages were hideously complicated.
Here are the dependency errors when I force libproj 7.2.1.
Package zstd conflicts for:
r-sf[version='>=0.9_7'] -> libgdal[version='>=3.1.4,<3.2.0a0'] -> zstd[version='>=1.4.5,<1.5.0a0|>=1.4.8,<1.5.0a0']
proj[version='>=7.2.1'] -> libtiff[version='>=4.2.0,<5.0a0'] -> zstd[version='>=1.3.7,<1.3.8.0a0|>=1.4.4,<1.5.0a0|>=1.4|>=1.4.3,<1.5.0.0a0|>=1.4.4,<1.5.0.0a0|>=1.4.5,<1.5.0a0']
r-base[version='>=4.0'] -> libtiff[version='>=4.1.0,<5.0a0'] -> zstd[version='>=1.3.7,<1.3.8.0a0|>=1.4.4,<1.5.0a0|>=1.4|>=1.4.3,<1.5.0.0a0|>=1.4.4,<1.5.0.0a0|>=1.4.5,<1.5.0a0']
Package libcxx conflicts for:
r-sf[version='>=0.9_7'] -> libgdal[version='>=3.1.4,<3.2.0a0'] -> libcxx[version='>=10.0.0|>=10.0.1|>=9.0.1|>=9.0.0|>=4.0.1']
libcxx[version='>=11.0.1']
proj[version='>=7.2.1'] -> libcxx[version='>=11.0.0|>=11.0.1']
r-rcpp -> libcxx[version='>=11.0.0|>=9.0.1|>=9.0.0|>=4.0.1']
proj[version='>=7.2.1'] -> libtiff[version='>=4.1.0,<5.0a0'] -> libcxx[version='>=10.0.0|>=10.0.1|>=9.0.1|>=9.0.0|>=4.0.1']
r-base[version='>=4.0'] -> krb5[version='>=1.17.1,<1.18.0a0'] -> libcxx[version='>=10.0.1|>=9.0.0|>=4.0.1']
r-sf[version='>=0.9_7'] -> libcxx[version='>=11.0.0|>=11.0.1']
r-rcpp -> r-base[version='>=3.6,<3.7.0a0'] -> libcxx[version='>=10.0.0|>=10.0.1|>=11.0.1']
r-base[version='>=4.0'] -> libcxx[version='>=10.0.0|>=11.0.0|>=11.0.1|>=9.0.1']
Package libnghttp2 conflicts for:
r-base[version='>=4.0'] -> libcurl[version='>=7.71.1,<8.0a0'] -> libnghttp2[version='>=1.41.0,<2.0a0']
proj[version='>=7.2.1'] -> libcurl[version='>=7.71.1,<8.0a0'] -> libnghttp2[version='>=1.41.0,<2.0a0']
Package libopenblas conflicts for:
r-base[version='>=4.0'] -> libblas[version='>=3.8.0,<4.0a0'] -> libopenblas[version='>=0.3.10,<0.3.11.0a0|>=0.3.12,<0.3.13.0a0|>=0.3.12,<1.0a0|>=0.3.10,<1.0a0|>=0.3.9,<0.3.10.0a0|>=0.3.9,<1.0a0|>=0.3.8,<0.3.9.0a0|>=0.3.8,<1.0a0|>=0.3.7,<0.3.8.0a0|>=0.3.7,<1.0a0|>=0.3.6,<0.3.7.0a0|>=0.3.6,<1.0a0']
r-rcpp -> r-base[version='>=3.5,<3.6.0a0'] -> libopenblas[version='>=0.2.20,<0.2.21.0a0']
r-units -> r-base[version='>=3.5,<3.6.0a0'] -> libopenblas[version='>=0.2.20,<0.2.21.0a0']
Package r-base conflicts for:
r-rcpp -> r-base[version='3.2.0.*|3.2.1.*|3.2.2.*|3.3.1.*|3.3.2.*|3.4.1.*|>=3.4.1,<3.4.2.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=4.0,<4.1.0a0|>=3.5.1,<3.5.2.0a0|>=3.5.0,<3.5.1.0a0|>=3.4.3,<3.4.4.0a0|>=3.4.2,<3.4.3a0']
r-base[version='>=4.0']
r-sf[version='>=0.9_7'] -> r-base[version='>=3.6,<3.7.0a0|>=4.0,<4.1.0a0']
r-sf[version='>=0.9_7'] -> r-classint[version='>=0.4_1'] -> r-base[version='3.2.0.*|3.2.1.*|3.2.2.*|3.3.2.*|3.4.1.*|>=3.4.1,<3.4.2.0a0|>=3.5,<3.6.0a0|>=3.5.1,<3.5.2.0a0|>=3.5.0,<3.5.1.0a0|>=3.4.3,<3.4.4.0a0|>=3.4.2,<3.4.3a0|3.3.1.*']
Package r-rcpp conflicts for:
r-rcpp
r-sf[version='>=0.9_7'] -> r-units[version='>=0.6_0'] -> r-rcpp
r-sf[version='>=0.9_7'] -> r-rcpp[version='>=0.12.18']
Package libssh2 conflicts for:
r-units -> r-base[version='>=4.0,<4.1.0a0'] -> libssh2[version='1.*|>=1.8.0,<2.0.0a0|>=1.9.0,<2.0a0|>=1.8.2,<2.0a0|>=1.8.0,<2.0a0']
r-base[version='>=4.0'] -> curl -> libssh2[version='1.*|>=1.8.0,<2.0a0|>=1.8.2,<2.0a0']
r-base[version='>=4.0'] -> libssh2[version='>=1.8.0,<2.0.0a0|>=1.9.0,<2.0a0']
r-rcpp -> r-base[version='>=3.6,<3.7.0a0'] -> libssh2[version='1.*|>=1.8.0,<2.0.0a0|>=1.9.0,<2.0a0|>=1.8.2,<2.0a0|>=1.8.0,<2.0a0']
proj[version='>=7.2.1'] -> libcurl[version='>=7.71.1,<8.0a0'] -> libssh2[version='>=1.9.0,<2.0a0']
r-sf[version='>=0.9_7'] -> r-base[version='>=3.6,<3.7.0a0'] -> libssh2[version='>=1.8.0,<2.0.0a0|>=1.9.0,<2.0a0|>=1.8.2,<2.0a0']
Package libcxxabi conflicts for:
r-rcpp -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|4.0.1|8.0.0|8.0.0|8.0.0|8.0.0|8.0.1',build='hebd6815_0|0|4|3|2|1|hcfea43d_1']
r-units -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|4.0.1|8.0.0|8.0.0|8.0.0|8.0.0|8.0.1',build='hebd6815_0|0|4|3|2|1|hcfea43d_1']
Package proj conflicts for:
proj[version='>=7.2.1']
r-sf[version='>=0.9_7'] -> proj[version='>=7.1.1,<7.1.2.0a0|>=7.2.0,<7.2.1.0a0']
Package r-units conflicts for:
r-sf[version='>=0.9_7'] -> r-units[version='>=0.6_0']
r-units
I thought the next step was to rebuild r-sf from scratch against the new libproj 7.2.1. But then when I tried to force a rebuild of r-sf against the latest libproj I got an even more complicated set of complaints, which led me to GDAL. I can't get the latest GDAL feedstock to build locally. The dependency matrix hell involves 40 packages.
I wish there was a way to tell conda: resolve the dependency problems yourself by rebuilding everything from scratch.
I decided to try something else: create a new environment to examine what is already in conda-forge, to see what packages the Azure CI system is using to build r-lwgeom. I run into similar issues: the core GIS packages (libproj, gdal, and r-sf) don't play nicely.
For example, the latest proj installs in a new environment.
stephenpitts@Ellacuria ~ % mamba create -n r-geospatial2 r-base=4.0
<... output omitted>
stephenpitts@Ellacuria ~ % conda activate r-geospatial2
(r-geospatial2) stephenpitts@Ellacuria ~ % mamba install proj==7.2.1
But then to install the latest gdal forces a downgrade of proj
(r-geospatial2) stephenpitts@Ellacuria ~ % mamba install gdal==3.2.1
Downgrade:
────────────────────────────────────────────────────────────────────────────────
proj 7.2.1 h78d1473_0 installed
proj 7.2.0 h78d1473_2 conda-forge/osx-64 Cached
and then to install r-sf wants to downgrade gdal even more.
(r-geospatial2) stephenpitts@Ellacuria ~ % mamba install r-sf
Downgrade:
──────────────────────────────────────────────────────────────────────────
gdal 3.2.1 py39heec4134_5 installed
gdal 3.1.4 py39he2ad08d_6 conda-forge/osx-64 1 MB
libgdal 3.2.1 h55c8954_5 installed
libgdal 3.1.4 h55c8954_6 conda-forge/osx-64 Cached
From the looks of the condasmith repositories around these packages, updates are happening, and hopefully most of it is automated: to progressively build each package against the latest upstream version of all libraries.
I wonder if MacOS X is behind the curve compared to Linux and Windows, and that is why less binary packages are available that only satisfy certain combinations of dependencies.
At this point I don't know what to do, especially since the dependencies are cyclical: is there a way to tell conda-build to build a set of packages locally, or to use one local package to satisfy another's dependencies? My instinct is to trace the dependencies one by one.
I'm going to leave this command running overnight and see what happens tomorrow morning:
(base) stephenpitts@Ellacuria r-packaging % conda build gdal-feedstock geotiff-feedstock r-lwgeom r-sf-feedstock
Yeah, I feel your pain. FWIW, I see the same libgdal and proj versions on Linux, so I don't think that's the problem.
I've found it useful to use the conda-forge infrastructure as much as possible. You could try this:
mamba create -n buildenv mamba conda-build conda-smithy conda-verify git shyaml
conda activate buildenv
git clone git@github.com:conda-forge/r-lwgeom-feedstock.gitThen edit r-lwgeom-feedstock/recipe/meta.yaml and remove the skip: true # [osx] bit and run
conda smithy rerender
python build-locally.pyThis will presumably fail the same way all the other OSX builds did, but you'll have the build artifacts (and maybe even the build environment, not sure how build-locally.py works on OS X) available locally and can (e.g.) go looking to see where libproj.19.dylib is and try to figure out why its not being found.
Along these same lines, I've figured out how to run on build artifacts in Azure pipelines (following the instructions at https://conda-forge.org/docs/maintainer/conda_forge_yml.html#azure-config). I re-ran the build and the artifacts are available at https://dev.azure.com/izahn/my-conda-tests/_build/results?buildId=14&view=artifacts&pathAsName=false&type=publishedArtifacts . I'm looking at those now to see if I can find any clues.
spittssj
commented
3 years ago Well I feel like I'm learning something about the internals of Conda builds at least. Evidently build-locally.py doesn't work on OS X. I followed those instructions and this is what happens. What's next?
(buildenv) stephenpitts@Ellacuria r-lwgeom-feedstock % python build-locally.py
valid configs are {'win_64_r_base4.0', 'linux_64_r_base4.0', 'osx_64_r_base3.6', 'linux_64_r_base3.6', 'win_64_r_base3.6', 'osx_64_r_base4.0'}
config not selected, please choose from the following:
1. linux_64_r_base3.6
2. linux_64_r_base4.0
3. osx_64_r_base3.6
4. osx_64_r_base4.0
5. win_64_r_base3.6
6. win_64_r_base4.0
> 4
selected osx_64_r_base4.0
Traceback (most recent call last):
File "/Users/stephenpitts/Code/r-packaging/r-lwgeom-feedstock/build-locally.py", line 75, in
main()
File "/Users/stephenpitts/Code/r-packaging/r-lwgeom-feedstock/build-locally.py", line 68, in main
verify_config(ns)
File "/Users/stephenpitts/Code/r-packaging/r-lwgeom-feedstock/build-locally.py", line 50, in verify_config
raise ValueError(
ValueError: only Linux configs currently supported, got osx_64_r_base4.0
Just for the fun of it, I ran conda build .. The recipe won't build the normal way either.
Package libssh2 conflicts for:
r-base=3.5 -> curl -> libssh2[version='1.*|>=1.9.0,<2.0a0']
r-base=3.5 -> libssh2[version='>=1.8.0,<2.0.0a0|>=1.8.2,<2.0a0|>=1.8.0,<2.0a0']
Package libopenblas conflicts for:
r-rcpp -> r-base[version='>=3.5,<3.6.0a0'] -> libopenblas[version='>=0.2.20,<0.2.21.0a0']
r-units -> r-base[version='>=3.5,<3.6.0a0'] -> libopenblas[version='>=0.2.20,<0.2.21.0a0']
r-base=3.5 -> libopenblas[version='>=0.2.20,<0.2.21.0a0']
Package zstd conflicts for:
r-sf[version='>=0.9_3'] -> libgdal[version='>=3.1.4,<3.2.0a0'] -> zstd[version='>=1.4.4,<1.5.0.0a0|>=1.4.5,<1.5.0a0|>=1.4.8,<1.5.0a0']
r-base=3.5 -> libtiff[version='>=4.1.0,<5.0a0'] -> zstd[version='>=1.3.3,<1.3.4.0a0|>=1.3.7,<1.3.8.0a0|>=1.4.4,<1.5.0a0|>=1.4|>=1.4.3,<1.5.0.0a0|>=1.4.4,<1.5.0.0a0|>=1.4.5,<1.5.0a0|>=1.4.0,<1.5.0.0a0']
proj -> libtiff[version='>=4.2.0,<5.0a0'] -> zstd[version='>=1.3.7,<1.3.8.0a0|>=1.4.4,<1.5.0a0|>=1.4|>=1.4.3,<1.5.0.0a0|>=1.4.4,<1.5.0.0a0|>=1.4.5,<1.5.0a0']
Package r-base conflicts for:
r-units -> r-rcpp -> r-base[version='3.2.0.*|3.2.1.*|3.2.2.*|3.3.1.*']
r-rcpp -> r-base[version='3.2.0.*|3.2.1.*|3.2.2.*|3.3.1.*|3.3.2.*|3.4.1.*|>=3.4.1,<3.4.2.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=4.0,<4.1.0a0|>=3.5.1,<3.5.2.0a0|>=3.5.0,<3.5.1.0a0|>=3.4.3,<3.4.4.0a0|>=3.4.2,<3.4.3a0']
r-sf[version='>=0.9_3'] -> r-base[version='>=3.6,<3.7.0a0|>=4.0,<4.1.0a0']
r-sf[version='>=0.9_3'] -> r-classint[version='>=0.4_1'] -> r-base[version='3.2.0.*|3.2.1.*|3.2.2.*|3.3.2.*|3.4.1.*|>=3.4.1,<3.4.2.0a0|>=3.5,<3.6.0a0|>=3.5.1,<3.5.2.0a0|>=3.5.0,<3.5.1.0a0|>=3.4.3,<3.4.4.0a0|>=3.4.2,<3.4.3a0|3.3.1.*']
r-units -> r-base[version='3.3.2.*|3.4.1.*|>=3.4.1,<3.4.2.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=4.0,<4.1.0a0|>=3.5.1,<3.5.2.0a0|>=3.5.0,<3.5.1.0a0|>=3.4.3,<3.4.4.0a0|>=3.4.2,<3.4.3a0']
r-base=3.5
Package libcxx conflicts for:
r-rcpp -> libcxx[version='>=11.0.0|>=9.0.1|>=9.0.0|>=4.0.1']
libcxx[version='>=11.0.1']
r-base=3.5 -> libcxx[version='>=4.0.1|>=9.0.1']
r-rcpp -> r-base[version='>=3.6,<3.7.0a0'] -> libcxx[version='>=10.0.0|>=10.0.1|>=11.0.1']
r-base=3.5 -> clangxx_osx-64=9 -> libcxx[version='>=10.0.0|>=10.0.1|>=11.0.0|>=9.0.0|>=11.0.1|>=8.0.1|>=8.0.0']
proj -> libcxx[version='>=10.0.0|>=10.0.1|>=11.0.0|>=11.0.1|>=9.0.1|>=9.0.0|>=4.0.1']
r-units -> r-base[version='>=4.0,<4.1.0a0'] -> libcxx[version='>=10.0.0|>=11.0.0|>=11.0.1|>=10.0.1']
r-sf[version='>=0.9_3'] -> libcxx[version='>=10.0.0|>=10.0.1|>=11.0.0|>=11.0.1|>=9.0.1']
r-units -> libcxx[version='>=4.0.1|>=9.0.0|>=9.0.1']
r-sf[version='>=0.9_3'] -> r-base[version='>=3.6,<3.7.0a0'] -> libcxx[version='>=4.0.1|>=9.0.0']
Package r-rcpp conflicts for:
r-sf[version='>=0.9_3'] -> r-rcpp[version='>=0.12.18']
r-units -> r-rcpp
r-sf[version='>=0.9_3'] -> r-units[version='>=0.6_0'] -> r-rcpp
r-rcpp
Package krb5 conflicts for:
r-base=3.5 -> curl -> krb5[version='1.14.*|>=1.14.6,<1.15.0a0|>=1.17.1,<1.18.0a0|>=1.18.2,<1.19.0a0']
r-base=3.5 -> krb5[version='>=1.16.1,<1.17.0a0|>=1.16.2,<1.17.0a0|>=1.16.3,<1.17.0a0|>=1.16.4,<1.17.0a0']
Package openssl conflicts for:
r-sf[version='>=0.9_3'] -> libgdal[version='>=3.1.4,<3.2.0a0'] -> openssl[version='>=1.1.1a,<1.1.2a|>=1.1.1d,<1.1.2a|>=1.1.1e,<1.1.2a|>=1.1.1f,<1.1.2a|>=1.1.1g,<1.1.2a|>=1.1.1h,<1.1.2a|>=1.1.1i,<1.1.2a|>=1.1.1j,<1.1.2a']
r-units -> r-base[version='>=3.4.1,<3.4.2.0a0'] -> openssl[version='>=1.0.2o,<1.0.3a']
proj -> libcurl[version='>=7.71.1,<8.0a0'] -> openssl[version='>=1.1.1a,<1.1.2a|>=1.1.1d,<1.1.2a|>=1.1.1f,<1.1.2a|>=1.1.1g,<1.1.2a|>=1.1.1h,<1.1.2a|>=1.1.1c,<1.1.2a|>=1.1.1b,<1.1.2a']
r-base=3.5 -> curl -> openssl[version='1.0.*|>=1.0.2o,<1.0.3a|>=1.0.2p,<1.0.3a|>=1.0.2m,<1.0.3a|>=1.1.1a,<1.1.2a|>=1.1.1d,<1.1.2a|>=1.1.1h,<1.1.2a|>=1.1.1g,<1.1.2a|>=1.1.1f,<1.1.2a|>=1.1.1c,<1.1.2a|>=1.1.1b,<1.1.2a|>=1.0.2n,<1.0.3a']
r-rcpp -> r-base[version='>=3.4.1,<3.4.2.0a0'] -> openssl[version='>=1.0.2o,<1.0.3a']
Package ncurses conflicts for:
r-base=3.5 -> ncurses[version='>=6.1,<7.0a0|>=6.2,<7.0a0']
r-base=3.5 -> readline[version='>=8.0,<9.0a0'] -> ncurses[version='5.9.*|>=6.1,<6.3.0a0|>=6.2,<6.3.0a0|>=6.0,<7.0a0|6.0.*']
Package xz conflicts for:
r-base=3.5 -> libtiff[version='>=4.1.0,<5.0a0'] -> xz[version='>=5.2.3,<5.3.0a0|>=5.2.5,<5.3.0a0|>=5.2.5,<6.0a0|>=5.2.3,<6.0a0']
r-base=3.5 -> xz[version='>=5.2.4,<5.3.0a0|>=5.2.4,<6.0a0']
Package proj conflicts for:
r-sf[version='>=0.9_3'] -> libgdal[version='>=3.0.4,<3.1.0a0'] -> proj[version='>=6.3.0,<6.3.1.0a0']
r-sf[version='>=0.9_3'] -> proj[version='>=6.3.1,<6.3.2.0a0|>=7.0.0,<7.0.1.0a0|>=7.1.0,<7.1.1.0a0|>=7.1.1,<7.1.2.0a0|>=7.2.0,<7.2.1.0a0']
proj
Package jpeg conflicts for:
r-base=3.5 -> libtiff[version='>=4.1.0,<5.0a0'] -> jpeg[version='>=9d,<10a']
r-base=3.5 -> jpeg[version='>=9b,<10a|>=9c,<10a']
Package libedit conflicts for:
r-base=3.5 -> krb5[version='>=1.16.4,<1.17.0a0'] -> libedit[version='>=3.1.20170329,<3.2.0a0|>=3.1.20170329,<4.0a0|>=3.1.20181209,<3.2.0a0|>=3.1.20181209,<4.0a0']
proj -> sqlite[version='>=3.33.0,<4.0a0'] -> libedit[version='>=3.1.20181209,<3.2.0a0|>=3.1.20191231,<3.2.0a0']
Package zlib conflicts for:
r-base=3.5 -> zlib[version='>=1.2.11,<1.3.0a0']
r-base=3.5 -> curl -> zlib[version='1.2.*|1.2.11']
Package libcxxabi conflicts for:
r-rcpp -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|4.0.1|8.0.0|8.0.0|8.0.0|8.0.0|8.0.1',build='hcfea43d_1|1|2|0|4|3|hebd6815_0']
proj -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|4.0.1|8.0.0|8.0.0|8.0.0|8.0.0|8.0.1',build='hcfea43d_1|1|2|0|4|3|hebd6815_0']
r-units -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|4.0.1|8.0.0|8.0.0|8.0.0|8.0.0|8.0.1',build='hcfea43d_1|1|2|0|4|3|hebd6815_0']
r-base=3.5 -> libcxx[version='>=4.0.1'] -> libcxxabi[version='4.0.1|4.0.1|8.0.0|8.0.0|8.0.0|8.0.0|8.0.1',build='hcfea43d_1|1|2|0|4|3|hebd6815_0']
Package r-units conflicts for:
r-sf[version='>=0.9_3'] -> r-units[version='>=0.6_0']
r-units
Package libcurl conflicts for:
r-base=3.5 -> libcurl[version='>=7.60.0,<8.0a0|>=7.61.0,<8.0a0|>=7.62.0,<8.0a0|>=7.64.1,<8.0a0|>=7.67.0,<8.0a0']
r-base=3.5 -> curl -> libcurl[version='7.57.0|7.58.0|7.59.0|7.60.0|7.61.0|7.61.1|7.61.1|7.61.1|7.62.0|7.62.0|7.63.0|7.63.0|7.64.0|7.64.0|7.64.0|7.64.0|7.64.1|7.64.1|7.65.2|7.65.3|7.68.0|7.68.0|7.69.1|7.71.0|7.71.1|7.71.1|7.71.0|7.69.1|7.68.0|7.67.0|7.65.3|7.65.2|7.64.1|7.64.0|7.63.0|7.63.0|7.62.0',build='hf30b1f0_0|hf30b1f0_0|h051b688_1000|h8a08a2b_1|h76de61e_1002|hbdb9355_0|h76de61e_1000|hbdb9355_0|h76de61e_1000|hc0b9707_5|h16faf7d_0|hc0b9707_1|he6690cf_0|he6690cf_0|h9bf37e3_5|h9bf37e3_6|h9bf37e3_8|he6690cf_4|he6690cf_1|hc0b9707_0|h709d2b2_0|h16faf7d_0|h16faf7d_0|hc0b9707_1|h16faf7d_4|he376013_2|h76de61e_0|hbdb9355_2|h8a08a2b_0|h051b688_0|h051b688_0|h051b688_0|h051b688_0|h051b688_0|h051b688_0|h051b688_2|h051b688_0|h051b688_0|hf30b1f0_0|hf30b1f0_0|hf30b1f0_0|hf30b1f0_0']
Package libgcc conflicts for:
r-units -> r-rcpp -> libgcc
r-rcpp -> libgcc
Well I feel like I'm learning something about the internals of Conda builds at least. Evidently
build-locally.pydoesn't work on OS X.
That is a shame
I followed those instructions and this is what happens. What's next?
Maybe reach out over at https://gitter.im/conda-forge/conda-forge.github.io and see if you can get anyone with more experience building conda packages on osx interested in taking a look.
spittssj
commented
3 years ago I tried two more things and I feel like I'm getting somewhere.
1) Looking at the output above from conda build, I edited the recipe to force it to use r-base >=4.0. The dependency solver was wandering around in an r-base==3.5 world. In my opinion, any package that uses a C library should be pinned to the most recent major version of R.
2) I built a local copy of r-units. It turns out it doesn't need r-cpp to run. At least conda build doesn't think so. So that simplifies one branch of dependency hell above.
3) Now using conda build . the dependency solver works, and I can reproduce locally the proj linking error we see on the CI.
(buildenv) stephenpitts@Ellacuria work % pwd
/Users/stephenpitts/miniconda3/conda-bld/r-lwgeom_1613573451614/work
(buildenv) stephenpitts@Ellacuria work % ./proj_conf_test
dyld: Library not loaded: @rpath/libproj.19.dylib
Referenced from: /Users/stephenpitts/miniconda3/conda-bld/r-lwgeom_1613573451614/work/./proj_conf_test
Reason: image not found
zsh: abort ./proj_conf_test
The next step is to figure out why the linker is not finding libproj. It's strange that the library name has code>@rpath</code in it.
I wonder if the part at the end of this document about RPATH is applicable in our case.
izahn
commented
3 years ago I tried two more things and I feel like I'm getting somewhere.
1. Looking at the output above from `conda build`, I edited the recipe to force it to use `r-base >=4.0`. The dependency solver was wandering around in an `r-base==3.5` world. In my opinion, any package that uses a C library should be pinned to the most recent major version of R.
You can specify the R version when you run conda build as documented at https://conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-r-pkgs.html#different-r-version. Also make sure you have conda-forge in your channels, or pass -c conda-forge to conda build as well.
2. I built a local copy of `r-units`. It turns out it doesn't need `r-cpp` to run. At least `conda build` doesn't think so. So that simplifies one branch of dependency hell above.
Try to avoid doing this -- you want to stay in the conda-forge ecosystem as much as possible, otherwise things might only work locally (or only break locally...)
3. Now using `conda build .` the dependency solver works, and I can reproduce locally the `proj` linking error we see on the CI.
Good!
(buildenv) stephenpitts@Ellacuria work % pwd /Users/stephenpitts/miniconda3/conda-bld/r-lwgeom_1613573451614/work (buildenv) stephenpitts@Ellacuria work % ./proj_conf_test dyld: Library not loaded: @rpath/libproj.19.dylib Referenced from: /Users/stephenpitts/miniconda3/conda-bld/r-lwgeom_1613573451614/work/./proj_conf_test Reason: image not found zsh: abort ./proj_conf_testThe next step is to figure out why the linker is not finding
libproj. It's strange that the library name has@rpathin it.I wonder if the part at the end of this document about RPATH is applicable in our case.
Not sure, but note that Anaconda != conda-forge.
spittssj
commented
3 years ago Ok. I keep learning. @edzer perhaps you can help.
1) Wikipedia's article on rpath describes what it is: a way to avoid hardcoding the location of a dynamic library in a linker. So we definitely have a linker issue:
stephenpitts@Ellacuria work % otool -L proj_conf_test
proj_conf_test:
@rpath/libproj.19.dylib (compatibility version 22.0.0, current version 22.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
stephenpitts@Ellacuria work %
2) The library libproj is definitely in the build environment.
stephenpitts@Ellacuria work % ls -l ../_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libproj.19.dylib
-rwxrwxr-x 1 stephenpitts staff 3593696 Feb 17 21:00 ../_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/libproj.19.dylibSide note: if you wonder what is going on in this pathname above https://github.com/conda/conda-build/issues/1482 can help you understand what happens: they want to generate a pathname at least 255 characters long so it can be replaced dynamically.
3) Manually setting the LD_LIBRARY_PATH works fine
stephenpitts@Ellacuria work % LD_LIBRARY_PATH=../_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib ./proj_conf_test
7204) The mystery is why the LD_LIBRARY_PATH is not set correctly in the configure script.
If we look in conda_build.sh, we find a note about a build problem someone had with GDAL.
# This is just to get around a configure failure when trying to link to gdal. ${R} CMD INSTALL --build .
if [[ ${target_platform} == osx-64 ]]; then
export LIBRARY_PATH=${PREFIX}/lib
export LD_LIBRARY_PATH=${PREFIX}/lib
export CXX="${CXX} --std=c++14 -Wl,-rpath,${PREFIX}/lib"
fiI added the LD_LIBRARY_PATH above and no dice. So that LD_LIBRARY_PATH does not make it into the configure environment.
I'm not quite sure how these autogenerated scripts work together (conda build process + autoconf) but I suspect the problem is here, that the LIB_PATH or LD_LIBRARY_PATH is not being set correctly during the configure.
I will check over at the conda-forge Gitter to see if anyone else can give us some advice.
cbrueffer
commented
3 years ago Merging PR https://github.com/conda-forge/r-lwgeom-feedstock/pull/17 hopefully fixed this, please test the new macOS packages.
spittssj
commented
3 years ago This morning I both successfully installed r-lwgeom in a fresh environment and also built it using the feedstock. Both worked perfectly. Thanks for your help @cbrueffer! When I saw your change, I kicked myself -- I forgot CXX means C++ and didn't think to update the C compiler command string as well.
cbrueffer
commented
3 years ago Wasn't my change, the accolades go to @isuruf and @dpryan79! ;-)
Issue:
Using the latest conda-forge packages, there is a link error between lwgeom and libproj. The environment is called
tmap-testbecause I can't get r-tmap to work either and I debugged the problem to here.(tmap-test) stephenpitts@Ellacuria ~ % R R version 4.0.3 (2020-10-10) -- "Bunny-Wunnies Freak Out" Copyright (C) 2020 The R Foundation for Statistical Computing Platform: x86_64-apple-darwin13.4.0 (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. [Previously saved workspace restored] > library("sf") Linking to GEOS 3.8.1, GDAL 3.1.4, PROJ 7.1.1 > library("lwgeom") Error: package or namespace load failed for ‘lwgeom’ in dyn.load(file, DLLpath = DLLpath, ...): unable to load shared object '/Users/stephenpitts/miniconda3/envs/tmap-test/lib/R/library/lwgeom/libs/lwgeom.dylib': dlopen(/Users/stephenpitts/miniconda3/envs/tmap-test/lib/R/library/lwgeom/libs/lwgeom.dylib, 6): Library not loaded: @rpath/libproj.15.dylib Referenced from: /Users/stephenpitts/miniconda3/envs/tmap-test/lib/R/library/lwgeom/libs/lwgeom.dylib Reason: image not foundEnvironment (
conda list):Details about
condaand system (conda info):active environment : tmap-test active env location : /Users/stephenpitts/miniconda3/envs/tmap-test shell level : 1 user config file : /Users/stephenpitts/.condarc populated config files : /Users/stephenpitts/.condarc conda version : 4.9.2 conda-build version : not installed python version : 3.8.5.final.0 virtual packages : __osx=10.16=0 __unix=0=0 __archspec=1=x86_64 base environment : /Users/stephenpitts/miniconda3 (writable) channel URLs : https://conda.anaconda.org/conda-forge/osx-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/osx-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/osx-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /Users/stephenpitts/miniconda3/pkgs /Users/stephenpitts/.conda/pkgs envs directories : /Users/stephenpitts/miniconda3/envs /Users/stephenpitts/.conda/envs platform : osx-64 user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Darwin/20.2.0 OSX/10.16 UID:GID : 501:20 netrc file : None offline mode : False