search

kconnour / pyRT_DISORT

A Python package for helping to compute input arrays to DISORT.
https://kconnour.github.io/pyRT_DISORT/
BSD 3-Clause "New" or "Revised" License
14 stars 5 forks source link

Ubuntu 20.04 installation failed #14

Open AStcherbinine opened 2 years ago

AStcherbinine commented 2 years ago

I guess similar issue to #2 but even the trick provided there did not work... I tried on two different machines running under Ubuntu desktop 20.04 and Ubuntu server 20.04, with Python 3.9 and gfortran.

As noted in #2, pip install . did not compile the Fortran code. python setup.py returns the following error:

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: no commands supplied

So as suggested I tried to copy/paste the Fortran link command (last statement line)

INFO: /usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpqohzdv0p/tmp/tmpqohzdv0p/src.linux-x86_64-3.9/disortmodule.o /tmp/tmpqohzdv0p/tmp/tmpqohzdv0p/src.linux-x86_64-3.9/fortranobject.o /tmp/tmpqohzdv0p/tmp/tmp2yanqjl_.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/BDREF.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/DISOBRDF.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/ERRPACK.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/LAPACK.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/LINPAK.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/RDI1MACH.o /tmp/tmpqohzdv0p/tmp/tmpqohzdv0p/src.linux-x86_64-3.9/disort-f2pywrappers.o -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -o ./disort.cpython-39-x86_64-linux-gnu.so

Yet, that did not work either as I still get an error

(EXI) astcherbinine@tiu:~/python_modules/pyRT_DISORT$ /usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmpqohzdv0p/tmp/tmpqohzdv0p/src.linux-x86_64-3.9/disortmodule.o /tmp/tmpqohzdv0p/tmp/tmpqohzdv0p/src.linux-x86_64-3.9/fortranobject.o /tmp/tmpqohzdv0p/tmp/tmp2yanqjl_.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/BDREF.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/DISOBRDF.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/ERRPACK.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/LAPACK.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/LINPAK.o /tmp/tmpqohzdv0p/home/astcherbinine/python_modules/pyRT_DISORT/disort4.0.99/RDI1MACH.o /tmp/tmpqohzdv0p/tmp/tmpqohzdv0p/src.linux-x86_64-3.9/disort-f2pywrappers.o -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -o ./disort.cpython-39-x86_64-linux-gnu.so
gfortran: error: /tmp/tmpqohzdv0p/tmp/tmpqohzdv0p/src.linux-x86_64-3.9/disortmodule.o: No such file or directory
gfortran: error: /tmp/tmpqohzdv0p/tmp/tmpqohzdv0p/src.linux-x86_64-3.9/fortranobject.o: No such file or directory
kconnour commented 2 years ago

When you say pip install . didn't compile the Fortran code, did it seemingly do nothing or did it provide you some error? This may be the most diagnostic message before I can provide further help. If it did nothing, did you install gfortran with apt? My only immediate guess on why it wouldn't compile is because the gfortran isn't linked properly (which seems to be the root cause of #2)

Running python setup.py should give that message because setup.py is a specially named file. If you aren't already doing so, I'd recommend making a virtual environment and running <path to venv interpreter> -m pip install ..

P.S. I'm happy to get a new user! Beware that I'm currently working on a more intuitive user interface so fairly soon most of the interface will change (so it may be worth your time to temporarily hold off figuring out how to use everything described in the documentation). However, compiling and installing the disort binary file will need to be fixed either way.

AStcherbinine commented 2 years ago

Thanks for your reply!

Actually, I do not get any error from the pip install ., and that's why I initially thought it worked. But it seems that the pyrt module is installed, but not the disort one Capture d’écran de 2022-03-20 21-24-00

And by looking at pip freeze, I can find pyRT-DISORT==0.0.3 so looks like the module is properly installed.

mjwolff commented 2 years ago

I have verified this on a machine at NAU.

The problem seems to be related to python not gfortran:

(vpyrt) @.***:~/test/pyRT_DISORT$ /usr/bin/gfortran -Wall -g -Wall -g -shared /tmp/tmp82w5u0p0/tmp/tmp82w5u0p0/src.linux-x86_64-3.9/disortmodule.o /tmp/tmp82w5u0p0/tmp/tmp82w5u0p0/src.linux-x86_64-3.9/fortranobject.o /tmp/tmp82w5u0p0/tmp/tmp_9pt9ixi.o /tmp/tmp82w5u0p0/home/mwolff/test/pyRT_DISORT/disort4.0.99/BDREF.o /tmp/tmp82w5u0p0/home/mwolff/test/pyRT_DISORT/disort4.0.99/DISOBRDF.o /tmp/tmp82w5u0p0/home/mwolff/test/pyRT_DISORT/disort4.0.99/ERRPACK.o /tmp/tmp82w5u0p0/home/mwolff/test/pyRT_DISORT/disort4.0.99/LAPACK.o /tmp/tmp82w5u0p0/home/mwolff/test/pyRT_DISORT/disort4.0.99/LINPAK.o /tmp/tmp82w5u0p0/home/mwolff/test/pyRT_DISORT/disort4.0.99/RDI1MACH.o /tmp/tmp82w5u0p0/tmp/tmp82w5u0p0/src.linux-x86_64-3.9/disort-f2pywrappers.o -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -o ./disort.cpython-39-x86_64-linux-gnu.so gfortran: error: /tmp/tmp82w5u0p0/tmp/tmp82w5u0p0/src.linux-x86_64-3.9/disortmodule.o: No such file or directory gfortran: error: /tmp/tmp82w5u0p0/tmp/tmp82w5u0p0/src.linux-x86_64-3.9/fortranobject.o: No such file or directory

The libraries just aren’t there. I suppose it could be the version of python or gfortran. The f2py stuff (i.e, it is the missing libraries in the link, not the compiled fortran objects.

(vpyrt) @.***:~/test/pyRT_DISORT$ ls /tmp/tmp82w5u0p0/tmp/tmp82w5u0p0/src.linux-x86_64-3.9/ disort-f2pywrappers.o

(vpyrt) @.***:~/test/pyRT_DISORT$ ls /tmp/tmp82w5u0p0/home/mwolff/test/pyRT_DISORT/disort4.0.99 BDREF.o DISOBRDF.o ERRPACK.o LAPACK.o LINPAK.o RDI1MACH.o

On Mar 20, 2022, at 11:26 PM, Aurélien Stcherbinine @.**@.>> wrote:

Thanks for your reply!

Actually, I do not get any error from the pip install ., and that's why I initially thought it worked. But it seems that the pyrt module is installed, but not the disort one [Capture d’écran de 2022-03-20 21-24-00]https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F31817302%2F159205269-74045e4c-b983-4d63-871a-5e7db06a8ab5.png&data=04%7C01%7Cmjwolff%40spacescience.org%7C958591181eee4003213008da0af2f028%7C3461589e91624c2e8e5b764489129af6%7C1%7C0%7C637834335799251673%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=k0Yck8gzQgKfidfyOz5GrnZYOX5AArnu69qYxp0ni9s%3D&reserved=0

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkconnour%2FpyRT_DISORT%2Fissues%2F14%23issuecomment-1073474011&data=04%7C01%7Cmjwolff%40spacescience.org%7C958591181eee4003213008da0af2f028%7C3461589e91624c2e8e5b764489129af6%7C1%7C0%7C637834335799251673%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=FmjS%2BSzkYIGB9FZZHR4as%2FYviVeJgaYLM58cT5xO5zs%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAKXHPEFEZPT5O4H7RBQW5TVA726NANCNFSM5RGK234A&data=04%7C01%7Cmjwolff%40spacescience.org%7C958591181eee4003213008da0af2f028%7C3461589e91624c2e8e5b764489129af6%7C1%7C0%7C637834335799251673%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=LwLTlCibtFZCNdRiFATvR1JBIrStE%2F%2BTEYXnZNq%2BQpM%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cmjwolff%40spacescience.org%7C958591181eee4003213008da0af2f028%7C3461589e91624c2e8e5b764489129af6%7C1%7C0%7C637834335799251673%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=qwQ1MYFV94jX9sqJBVZjW46iKorfUFYnTC%2B%2Fs0%2B9cZc%3D&reserved=0 or Androidhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cmjwolff%40spacescience.org%7C958591181eee4003213008da0af2f028%7C3461589e91624c2e8e5b764489129af6%7C1%7C0%7C637834335799251673%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=F%2Bw6uyyA8Z66evVsuo1PnvGbtbOIXgSaCRK%2F99nsL7o%3D&reserved=0. You are receiving this because you are subscribed to this thread.Message ID: @.***>

kconnour commented 2 years ago

Hmm I see. This issue will be difficult for me to fully diagnose because it works on my Ubuntu computer and it's the only one I have access to.

In the meantime, this is the code that compiles the .so object:

        path_to_pyrt_DISORT = 'changeme'
        folder_name = 'disort4.0.99'
        module_name = 'disort'

        disort_source_dir = os.path.join(path_to_pyrt, folder_name)
        mods = ['BDREF.f', 'DISOBRDF.f', 'ERRPACK.f', 'LAPACK.f',
                'LINPAK.f', 'RDI1MACH.f']
        paths = [os.path.join(disort_source_dir, m) for m in mods]
        with open(os.path.join(disort_source_dir, 'DISORT.f')) as mod:
            f2py.compile(mod.read(), modulename=module_name, extra_args=paths)

You'll just need to change the first line. This should put the .so file into the disort4.0.99 directory. Move this file into site-packages and your code should now work. Let me know how this goes.

Did pip install . take 1+ minutes? If so, that would suggest it compiled the object but it was placed in an unexpected directory and thus wasn't installed in the library.

AStcherbinine commented 2 years ago

I've updated the setup.py file as suggested (see below) but I cannot find any .so file The pip install . takes approximately 1min so not sure it will help you, but I attached the output of the command

import glob
import os
import setuptools
from numpy import f2py

# TODO: see if there's a way to check if the .so file is up to date. If so,
#  don't bother reinstalling it. Otherwise, do install it.
class SetupDISORT:
    def __init__(self) -> None:
        self.__project_path = self.__get_project_path()
        self.__install_disort()
        # TODO: I want to move the so file into the package, but if I do I
        #  can't figure out how to import disort. If I do this, I need to
        #  remove the ../ part in setup.cfg
        #self.__move_so_file_into_package()
        setuptools.setup()

    @staticmethod
    def __get_project_path() -> str:
        return os.path.dirname(os.path.realpath(__file__))

    def __install_disort(self) -> None:
        path_to_pyrt = '/home/astcherbinine/python_modules/pyRT_DISORT/'
        folder_name = 'disort4.0.99'
        module_name = 'disort'

        # disort_source_dir = os.path.join(self.__project_path, folder_name)
        disort_source_dir = os.path.join(path_to_pyrt, folder_name)
        mods = ['BDREF.f', 'DISOBRDF.f', 'ERRPACK.f', 'LAPACK.f',
                'LINPAK.f', 'RDI1MACH.f']
        paths = [os.path.join(disort_source_dir, m) for m in mods]
        # I'm disgusted to say I'm adding a comment. I want to compile DISORT.f
        #  as a module, and I can do that by adding the other modules it needs
        #  in extra_args (this wasn't clear in f2py documentation).
        with open(os.path.join(disort_source_dir, 'DISORT.f')) as mod:
            f2py.compile(mod.read(), modulename=module_name, extra_args=paths)

    # def __move_so_file_into_package(self) -> None:
        # libname = glob.glob(os.path.join(self.__project_path, '*.so'))[0]
        # os.rename(libname, os.path.join(self.__project_path, 'pyRT_DISORT',
                                        # os.path.basename(libname)))

SetupDISORT()

log_pip.log

kconnour commented 2 years ago

Oops, I wasn't clear in my last message. I would suggest trying to run that code in any .py file not named setup.py, and then running python on that file. When running setup.py, pip suppresses all possibly helpful messages and occasionally some things created during setup.py vanish after the installation is done.

This would isolate the problem to those few lines of code and remove pip from the equation.

AStcherbinine commented 2 years ago

Oh yes sorry. Ok so here is the log of this code, but it still did not see the .so file... Except from the folder name, the output is the same as the one Mike gets in https://github.com/kconnour/pyRT_DISORT/issues/2#issuecomment-1073806359

test_install.log

kconnour commented 2 years ago

I made a file named foobar.py containing the 9 lines of code I wrote in a previous message. I confirmed those lines by themselves allow me to compile the .so file; attached is the log. The only real difference I notice is the last line where f2py tells me it removed the build directory.

I'm running out of ideas because I'm not a Fortran expert. I'm using numpy version 1.22.2 and gcc/gfortran version 9.4.0 that I installed with apt. If the install is taking 1 minute-ish I'm guessing it's compiling but for whatever reason the binary file is getting stuck in /tmp/.

installpyrt.log

AStcherbinine commented 2 years ago

I think I've figured it out. By running f2py as a bash script and not a Python one, I can get a more detailed log. Actually, the compilation wasn't complete: image

So it can be fixed by installing also the python3.9-dev package (and not only the python3.9 one). Now the installation through pip install . seems to work!

kconnour commented 2 years ago

Great idea! I didn't consider that I had already installed those on my computers. Thank you for taking the time to track down this issue. I'll put info on this in the next round of documentation.

@mjwolff Whenever you get to it, can you see if this solves the Mac installation issue too?

mjwolff commented 2 years ago

Good catch, @aurelien!

I create a virtual environment when I use pyRT_DISORT, but adding the following line during creation solved it (I could do the “import disort" test.

pip install python-dev-tools

On Mar 28, 2022, at 3:41 PM, kconnour @.**@.>> wrote:

Great idea! I didn't consider that I had already installed those on my computers. Thank you for taking the time to track down this issue. I'll put info on this in the next round of documentation.

@mjwolffhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmjwolff&data=04%7C01%7Cmjwolff%40spacescience.org%7Cfc5b0e8617c64fcdcb6f08da10fb66b6%7C3461589e91624c2e8e5b764489129af6%7C1%7C0%7C637840969201986444%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=t66ni3BahlXTJEcOdMLrfKWXq8fxMAzpyY96IA5A68Q%3D&reserved=0 Whenever you get to it, can you see if this solves the Mac installation issue too?

— Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkconnour%2FpyRT_DISORT%2Fissues%2F14%23issuecomment-1081121246&data=04%7C01%7Cmjwolff%40spacescience.org%7Cfc5b0e8617c64fcdcb6f08da10fb66b6%7C3461589e91624c2e8e5b764489129af6%7C1%7C0%7C637840969201986444%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yqG72OiaO1fLRNtRAGw7wfYHygOXeZrgHLmLCRNItZI%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAKXHPEAAHK6ODPYLNIMOZDVCIKRJANCNFSM5RGK234A&data=04%7C01%7Cmjwolff%40spacescience.org%7Cfc5b0e8617c64fcdcb6f08da10fb66b6%7C3461589e91624c2e8e5b764489129af6%7C1%7C0%7C637840969201986444%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=m375yN8D7Nt1LOjWHwRmVxMaLee95gnuPty5HuOnwzY%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>