search

GrossfieldLab / loos

LOOS: a lightweight object-oriented structure analysis library
http://grossfieldlab.github.io/loos/
GNU General Public License v3.0
122 stars 26 forks source link

Installation fails on MacOS Monterey, Apple M1 Pro MBP #88

Closed ayanb73 closed 1 year ago

ayanb73 commented 1 year ago

Describe the bug When I attempt to install loos, the build fails partway through.

To Reproduce Steps to reproduce the behavior:

Expected behavior The command {python -c 'import loos'} should be successful with no errors.

**LOOS version and platform Which version you're running, on what OS, and how it was built (e.g. conda, system libraries, manually installed dependencies) The provided conda_install.sh script was utilized using a miniconda installation, on an Apple M1 Pro MacBook Pro (2021) running MacOS Monterey. In addition, a manual build was attempted (on suggestion from @lgsmith. After re-cloning, the following was ran:

cd loos mkdir build cd build cmake --debug-output --trace-exp .. >& cmake.log The cmake.log file is attached below. cmake.log

Additional context The output from the conda_install.sh run is as follows.

make[2]: *** No rule to make target/Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform /Developer/SDKs/MacOSX11.0.sdk/usr/lib/libm.tbd', needed by src/libloos.dylib'. Stop. make[2]: *** Waiting for unfinished jobs.... [ 14%] Building CXX object src/CMakeFiles/loos.dir/scanner.cc.o /Users/ayan/Documents/loos/src/xtcwriter.cpp:437:29: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion] if (fabs(lf) > INT_MAX-2) ~ ~~~~~~~^~ /Users/ayan/Documents/loos/src/xtcwriter.cpp:451:29: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion] if (fabs(lf) > INT_MAX-2) ~ ~~~~~~~^~ /Users/ayan/Documents/loos/src/xtcwriter.cpp:488:48: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion] (float)maxint[2] - (float)minint[2] >= INT_MAX-2) { ~~ ~~~~~~~^~ /Users/ayan/Documents/loos/src/xtcwriter.cpp:487:48: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion] (float)maxint[1] - (float)minint[1] >= INT_MAX-2 || ~~ ~~~~~~~^~ /Users/ayan/Documents/loos/src/xtcwriter.cpp:486:55: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion] if ((float)maxint[0] - (float)minint[0] >= INT_MAX-2 || ~~ ~~~~~~~^~ 5 warnings generated. make[1]: *** [src/CMakeFiles/loos.dir/all] Error 2 make: *** [all] Error 2

agrossfield commented 1 year ago

Hi,

I'm sorry you're having trouble. The way you ran things, however, isn't the way INSTALL.md indicates, and I'm pretty sure that's why it's not working for you.

I suggest you get rid of the loos conda env, and (with just the base conda active) run the command suggested:

 ./conda_build.sh -e loos -j 8 -i

This will create a new conda env called loos, install the necessary packages, then build and install loos. The fact that it's invoking Xcode stuff makes me think that something didn't get installed correctly -- the whole point of the way we do it is to make sure we use the conda toolchain start to finish.

I confess I would have expected what you did to work as well, but I've seen conda do weird things before.

Again, I'm sorry the process isn't working for you.

Alan

PS Alternatively, I expect the conda-forge recipe to be accepted in the next couple of days. After that, you'll be able to just say conda install -c conda-forge loos to get everything you need to use existing tools and write new python tools.

ayanb73 commented 1 year ago

Hi Alan,

Thanks for your response. I attempted a new installation on a clean environment using the command you provided. I attempted this using both libnetcdf=4.9.0 and libnetcdf=4.8.1, and ran into the same issue both times. The error I receive is the same, with something new added on:

make[2]: *** No rule to make target `/Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk/usr/lib/libm.tbd', needed by `src/libloos.dylib'.  Stop.
make[2]: *** Waiting for unfinished jobs....
[ 14%] Building CXX object src/CMakeFiles/loos.dir/scanner.cc.o
/Users/ayan/Documents/loos/src/xtcwriter.cpp:437:29: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion]
      if (fabs(lf) > INT_MAX-2) 
                   ~ ~~~~~~~^~
/Users/ayan/Documents/loos/src/xtcwriter.cpp:451:29: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion]
      if (fabs(lf) > INT_MAX-2)
                   ~ ~~~~~~~^~
/Users/ayan/Documents/loos/src/xtcwriter.cpp:488:48: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion]
        (float)maxint[2] - (float)minint[2] >= INT_MAX-2) {
                                            ~~ ~~~~~~~^~
/Users/ayan/Documents/loos/src/xtcwriter.cpp:487:48: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion]
        (float)maxint[1] - (float)minint[1] >= INT_MAX-2 ||
                                            ~~ ~~~~~~~^~
/Users/ayan/Documents/loos/src/xtcwriter.cpp:486:55: warning: implicit conversion from 'int' to 'float' changes value from 2147483645 to 2147483648 [-Wimplicit-const-int-float-conversion]
    if ((float)maxint[0] - (float)minint[0] >= INT_MAX-2 ||
                                            ~~ ~~~~~~~^~
5 warnings generated.
make[1]: *** [src/CMakeFiles/loos.dir/all] Error 2
make: *** [all] Error 2
*** Installing LOOS ***
-- Install configuration: ""
ERROR: Invalid requirement: '/Users/ayan/Documents/loos/build/src/pyloos'
Hint: It looks like a path. File '/Users/ayan/Documents/loos/build/src/pyloos' does not exist.
CMake Error at src/cmake_install.cmake:45 (file):
  file INSTALL cannot find "/Users/ayan/Documents/loos/build/src/_pyloos.so":
  No such file or directory.
Call Stack (most recent call first):
  cmake_install.cmake:42 (include)

I provide the above in case it's useful, but I'll look forward to the conda-forge recipe!

All the best, Ayan

tromo commented 1 year ago

Hi Ayan, try downgrading libnetcdf further... Try <4.8, and if you get the same error, then try <4.7.

agrossfield commented 1 year ago

Also, can you redirect conda_build's output to a file and attach it? I suspect whatever is going wrong is happening long before what we're seeing.

Start totally clean -- rm the build directory, get rid of the conda env (preferably a fresh install of conda itself, since I've seen it corrupt itself before).

ayanb73 commented 1 year ago

tried libnetcdf=4.8.0, which did not work. But unfortunately for versions earlier than that I don't think the libnetcdf package is available for an ARM64 architecture on OSX. Attached is an output file of a fresh env, with the build directory removed. conda_build.txt

Edit for detail: I used the -e option to make a fresh env while I've been trying (hence why the env is now on loos14 :)

lgsmith commented 1 year ago

I've looked at whether it's possible to downgrade the netcdf version further, and it appears not to be for osx-arm64. I suspected that this was what Ayan was running into, but I just checked with conda search and it isn't an option. Here's the filtered output from the conda search command I ran to conclude this.

conda search -i libnetcdf[subdir=osx-arm64] -c conda-forge | grep -e version
version     : 4.8.0
version     : 4.8.0
version     : 4.8.0
version     : 4.8.0
version     : 4.8.0
version     : 4.8.0
version     : 4.8.0
version     : 4.8.0
version     : 4.8.0
version     : 4.8.0
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.8.1
version     : 4.9.0
version     : 4.9.0
version     : 4.9.0

If this is at all relevant, the same command without the -c conda-forge optional arg only produces 4.8.0 and 4.8.1.

Are we sure there will be a conda binary for loos on osx-arm64? It seems like it might be hard to make one if the build deps have to come from conda-forge...

agrossfield commented 1 year ago

@tromo : do you think we should report the packaging problem to the netcdf conda-forge team? Are we sure this is their problem?

agrossfield commented 1 year ago

What confuses me is I'm almost positive there are people who have arm Macs who have loos running, or at least I thought there were. I'll need to get ahold of one to try to figure this out (mine is one of the last generation of intel ones).

tromo commented 1 year ago

This wont affect scons. This is a problem with a cmake module file that libnetcdf is bringing in with it. We probably should contact them.

On my mac, I can run mamba create -n foo python=3 libnetcdf. Then if I go and look at /Users/tromo/anaconda3/envs/foo/lib/cmake/netCDF/netCDFTargets.cmake, I find,

set_target_properties(netCDF::netcdf PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "/Users/tromo/anaconda3/envs/foo/lib/libmfhdf.dylib;/Users/tromo/anaconda3/envs/foo/lib/libdf.dylib;/Users/tromo/anaconda3/envs/foo/lib/libhdf5_hl.dylib;/Users/tromo/anaconda3/envs/foo/lib/libhdf5.dylib;m;/Users/tromo/anaconda3/envs/foo/lib/libz.dylib;/Users/tromo/anaconda3/envs/foo/lib/libzip.dylib;/Users/tromo/anaconda3/envs/foo/lib/libzstd.dylib;/Users/tromo/anaconda3/envs/foo/lib/libbz2.dylib;/Users/tromo/anaconda3/envs/foo/lib/libcurl.dylib;/Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libxml2.dylib"
)

Now, I am on MacOS 12.6, and I have XCode 13.4.1 installed, but you see it's got something hard-coded for XCode 13.2.1 and MacOS 10.9.

In contrast, if I look at the same spot in the same file for libnetcdf-4.8.1 from my loos env on my mac, I see,

set_target_properties(netCDF::netcdf PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "/Users/tromo/anaconda3/envs/loos/lib/libmfhdf.dylib;/Users/tromo/anaconda3/envs/loos/lib/libdf.dylib;/Users/tromo/anaconda3/envs/loos/lib/libhdf5_hl.dylib;/Users/tromo/anaconda3/envs/loos/lib/libhdf5.dylib;m;/Users/tromo/anaconda3/envs/loos/lib/libcurl.dylib;/Users/tromo/anaconda3/envs/loos/lib/libzip.dylib"
)

No junk from xcode being brought in!

What I would try to do, just to get it working, is to make a copy of your netCDFTargets.cmake from your loos env, then edit the original file and snip out the parts that reference xcode or macos (this is the set_target_properties() statement at line 61). Then remove the build dir and run the two cmakes.

agrossfield commented 1 year ago

Is that file supplied by cmake or libnetcdf? We need to report the bug to whoever is packaging it

tromo commented 1 year ago

It's supplied by libnetcdf. My "foo" test env doesn't have cmake installed...

agrossfield commented 1 year ago

I just confirmed it, and I’m opening an issue on the feedstock repo right now. Reassuring that it’s not our screwup, but still really quite annoying.

Dr. Alan Grossfield Dept of Biochemistry and Biophysics University of Rochester Medical Center Phone: 585 276 4193 http://membrane.urmc.rochester.edu https://orcid.org/0000-0002-5877-2789 Pronouns: He/his

From: Tod Romo @.> Reply-To: GrossfieldLab/loos @.> Date: Tuesday, October 25, 2022 at 4:17 PM To: GrossfieldLab/loos @.> Cc: "Grossfield, Alan" @.>, Comment @.***> Subject: [EXT] Re: [GrossfieldLab/loos] Installation fails on MacOS Monterey, Apple M1 Pro MBP (Issue #88)

It's supplied by libnetcdf. My "foo" test env doesn't have cmake installed...

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_GrossfieldLab_loos_issues_88-23issuecomment-2D1291087426&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=lBSRLC2965e_GWcXcZsFQDTMKibXnVC-xRU3c4CX_vQ&s=SfxwKEWrCkdgTiIHsPMbbJHcu72aRTLWZ0jvg2vr6YI&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADH754WMDRZN7WQEXAW4ZKTWFA54FANCNFSM6AAAAAAROCXFHY&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=lBSRLC2965e_GWcXcZsFQDTMKibXnVC-xRU3c4CX_vQ&s=kJxiNYWAnsYL0h0hrA9_6uTxXZQMNTOuLUZB01A-d6g&e=. You are receiving this because you commented.Message ID: @.***>

lgsmith commented 1 year ago

On the other hand we did a good deed and can reduce restrictions on our install, which seems like it'll be helpful.

I'm wondering why 4.8.1 didn't fix the issue, if it doesn't have that cmake weirdness in it. Maybe the osx-arm64 feedstock got that particular bug earlier in the version tree?

On Tue, Oct 25, 2022 at 4:39 PM Grossfield Lab @.***> wrote:

I just confirmed it, and I’m opening an issue on the feedstock repo right now. Reassuring that it’s not our screwup, but still really quite annoying.

Dr. Alan Grossfield Dept of Biochemistry and Biophysics University of Rochester Medical Center Phone: 585 276 4193 http://membrane.urmc.rochester.edu https://orcid.org/0000-0002-5877-2789 Pronouns: He/his

From: Tod Romo @.> Reply-To: GrossfieldLab/loos @.> Date: Tuesday, October 25, 2022 at 4:17 PM To: GrossfieldLab/loos @.> Cc: "Grossfield, Alan" @.>, Comment @.***> Subject: [EXT] Re: [GrossfieldLab/loos] Installation fails on MacOS Monterey, Apple M1 Pro MBP (Issue #88)

It's supplied by libnetcdf. My "foo" test env doesn't have cmake installed...

— Reply to this email directly, view it on GitHub< https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_GrossfieldLab_loos_issues_88-23issuecomment-2D1291087426&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=lBSRLC2965e_GWcXcZsFQDTMKibXnVC-xRU3c4CX_vQ&s=SfxwKEWrCkdgTiIHsPMbbJHcu72aRTLWZ0jvg2vr6YI&e=>, or unsubscribe< https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADH754WMDRZN7WQEXAW4ZKTWFA54FANCNFSM6AAAAAAROCXFHY&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhA&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=lBSRLC2965e_GWcXcZsFQDTMKibXnVC-xRU3c4CX_vQ&s=kJxiNYWAnsYL0h0hrA9_6uTxXZQMNTOuLUZB01A-d6g&e=>.

You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/GrossfieldLab/loos/issues/88#issuecomment-1291116299, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAZVYPIE5CEL5EI52IPTTDWFBARTANCNFSM6AAAAAAROCXFHY . You are receiving this because you were mentioned.Message ID: @.***>

agrossfield commented 1 year ago

Opened https://github.com/conda-forge/libnetcdf-feedstock/issues/155 to notify the libnetcdf folks about the problem

agrossfield commented 1 year ago

Per @tromo's workaround above: you'll need to install libxml2 from conda-forge too -- the bug is in part caused because they now have a dependence on libxml2 but forgot to update their conda recipe. I strongly suspect that if you do that, you can just delete the two fields that have the Xcode crap in them and it will work.

In the longer term, the libnetcdf-feedstock maintainers are on the case, and I strongly suspect they'll release new packages soon, at which point we'll update our recipes as well.

agrossfield commented 1 year ago

@lgsmith : I suspect the reason it's hitting different versions of libnetcdf on different architectures is the screwup is in the build.sh script in the feedstock: any package built after they introduced the bug will have this problem, independent of the version of libnetcdf itself.

agrossfield commented 1 year ago

The libnetcdf folks pushed a "fix" that broke both linux and Mac -- they fixed half the problem by adding libxml2 to their dependencies, but that left the libm problem. I just opened a pull request (https://github.com/conda-forge/libnetcdf-feedstock/pull/158) that should fix the problem.

agrossfield commented 1 year ago

My PR was accepted, so hopefully it will percolate through and this will all be resolved shortly

lgsmith commented 1 year ago

When you made the fixes did it build for you? I'm assuming yes...

On Mon, Oct 31, 2022 at 8:43 AM Grossfield Lab @.***> wrote:

My PR was accepted, so hopefully it will percolate through and this will all be resolved shortly

— Reply to this email directly, view it on GitHub https://github.com/GrossfieldLab/loos/issues/88#issuecomment-1297030429, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAZVYLGSPRDS6OXEMZURNDWF65IXANCNFSM6AAAAAAROCXFHY . You are receiving this because you were mentioned.Message ID: @.***>

agrossfield commented 1 year ago

I didn't really check -- I was editing their build.sh script, specifically modifying a sed command in there that edited some of their cmake files. I restored the sed command that previously fixed the linux build (assuming nothing had changed) and created an equivalent that fixed the OSX problem. All I did was verify that I got a netCDFTargets.cmake file that looked like the ones that worked. The problem is I don't really know what that file is supposed to do.

agrossfield commented 1 year ago

I just verified that loos builds correctly from conda_build.sh on linux and Mac. You'll need to re-pull, because I removed the lock of libnetcdf=4.8.1 and at this moment only version 4.9.0 works (the back-port to 4.8.1 has been committed but hasn't propagated to the servers yet).

agrossfield commented 1 year ago

Current status: there are conda packages for all Macs and linux. However, if you don't have Xcode installed, they might not work -- for some reason, the build on the conda forge server pulls in the Xcode blas and lapack libraries even though when we build locally it correctly grabs the ones from within conda.

I just want people to know we're aware of the problem and working on it (by we, I mean @tromo is working in it).

lgsmith commented 1 year ago

Does 4.0.3 fix this?

agrossfield commented 1 year ago

Yes, it will. I just didn’t want it closed until the 4.0.3 package was available, which should be later this week.

Dr. Alan Grossfield Department of Biochemistry and Biophysics University of Rochester Medical Center

On Nov 14, 2022, at 4:35 PM, lgsmith @.***> wrote:



Does 4.0.3 fix this?

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_GrossfieldLab_loos_issues_88-23issuecomment-2D1314428117&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhCwrmQ9ZDKmVaAI6sBPw4Jo&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=OVyOzvqSX31hbqR6dboz4HBWCpDBf_hqlomeMUR5lynvn70l5ellwBL-QWwp458q&s=2__il24T858reTkFXdWFAqwtwUvWkVLded838BOGmYI&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADH754TBCM4S7L33AI4EG3TWIKV75ANCNFSM6AAAAAAROCXFHY&d=DwMCaQ&c=4sF48jRmVAe_CH-k9mXYXEGfSnM3bY53YSKuLUQRxhCwrmQ9ZDKmVaAI6sBPw4Jo&r=49qnaP-kgQR_zujl5kbj_PmvQeXyz1NAoiLoIzsc27zuRX32UDM2oX8NQCaAsZzH&m=OVyOzvqSX31hbqR6dboz4HBWCpDBf_hqlomeMUR5lynvn70l5ellwBL-QWwp458q&s=33oGIXt7LtVrzOXLx8ccWo6rozYQ6A9YWBAl27T0ijY&e=. You are receiving this because you commented.Message ID: @.***>

agrossfield commented 1 year ago

4.0.3 is out in both GitHub and conda-forge, so the fix to this problem is available to everyone.