Can pip install telfit produce a functioning installation on a fresh machine?

[shihyuntang]

Hi @gully

Thank you for setting up the pip installation. However, I find something weird.

When using pip install telfit, stuff that used to be inside the .Telfit dir are all mess up. For example in an env called igrins_rv, the runlblrtm_v3.sh that used to be under an rundirXX is placed directly under ~/anaconda3/envs/igrins_rv (this will mess up with the multiprocess, right?). Oh, and now I cannot find any of the rundirXX elsewhere.

MakeModel.py, MakeTape5.py, and TelluricFitter.py are placed directly under ~/anaconda3/envs/igrins_rv/lib/python3.7/site-packages.

And the .Telfit file is now at a weird place: ~/anaconda3/envs/igrins_rv/lib/python3.7/site-packages/Users/username with only the example folder under it.

Finally, there are only the __init__.py and __pycache__ under ~/anaconda3/envs/igrins_rv/lib/python3.7/site-packages/telfit.

Maybe the pip setup file have some bug in it??

[gully]

As I was packaging the code I wondered how pip install could work at all, given that there are some seemingly platform-specific steps in the setup, such as the ones you describe. For example, how could someone maintain two different TelFit installs on the same computer, if they are both overriding the contents in .telfit directory? But pip install used to work, so surely the python packaging steps found a way to bottle up all of those customizations in the past, right? So I'm not sure this is a "bug" in the sense that a pip install version should have a different installation location than a python setup.py develop approach. It'd be useful to attempt pip install telfit on a machine that has not previously had a telfit install of any type (i.e. continuous integration testing). Under that blank slate approach, does pip install telfit produce a functioning install?

[gully]

related ideas that may be relevant here: python setup.py build_ext and other extension building may be the path forward.

Whatever we find out here we should create a new documentation page called "Developer Guide" or "Developer Instructions" that narrates how to build and deploy a custom build to PyPI.

[shihyuntang]

First time install with pip install telfit do work and able to run the example .py file (at least for single thread. Not sure how things will be if run it simultaneously in two terminals...). However, as @gully mentioned:

how could someone maintain two different TelFit installs on the same computer

If install telfit in one env via pip install telfit (which works), and you want to pip install telfit in another env,. Although the installation process won't give you any error, you will see this error while importing Telfit:

In [1]: import telfit
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-80758dd9b7f2> in <module>
----> 1 import telfit

~/anaconda3/lib/python3.7/site-packages/telfit/__init__.py in <module>
----> 1 from TelluricFitter import TelluricFitter
      2 from MakeModel import Modeler
      3 import DataStructures
      4 import FittingUtilities

~/anaconda3/lib/python3.7/site-packages/TelluricFitter.py in <module>
     51 import os
     52 from functools import partial
---> 53 import FittingUtilities
     54 import warnings
     55 import logging

src/FittingUtilities.pyx in init FittingUtilities()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Also, you will see this if you pip install telfit and uninstall it than what to reinstall it via pip install telfit again...

Moreover, if one use Telfit quite often, situation like all rundirXX are been locked will happen from time to time. It used to be easy to resolve by simply rm those .lock dirs. But withpip install`, stuff are been ??hidden?? during the installation process.

Before this issue been resolved, I think it is safer (easy to debug) for user to install telfit via source, which is still an easy way now... you just need to clone/download telfit and python setup.py build/install.

[gully]

Yes, exactly, telfit remains experiment research code, and so at the moment pip installation should be considered experimental. Since your system has already "seen" telfit you cannot easily test the premise I am curious about: a new, inexperienced user attempting to just run the code on a fresh installation, not a power user such as yourself, maintaining a developer version and a sandboxed version.

In practice I think all users of telfit should be considered co-developers or at least beta testers at this point, and so yes, I recommend the git clone and setup.py build pathway for essentially everyone.

[gully]

By the way, we could actually test the premise of this Issue with a docker container, so if you are keen to resolve this issue, I'd recommend experimenting with either Docker locally or for "extra credit" set up a GitHub actions continuous integration test.

[reillymilburn]

Hi @gully

New, inexperienced user here trying to do some telluric line removal for research! As far as I can tell, my machine is fresh. When I run pip install telfit I get to the part about building a wheel for telfit (setup.py) and I get the error "Suitable Compiler Not Found."

I am using a Mac running Big Sur 11.2.3 if that helps. I think this may be a gfortran issue, whereas new mac systems use clang. Perhaps I could bypass with homebrew?? If you have any wisdom it would be greatly appreciated! Also, apologies if this is the wrong thread for this!!

Thanks, Reilly

[shihyuntang]

Hi @reillymilburn , Yes, I think using brew install gfortran should fix the issue. (running ok on my mac with Big Sur) Note that the LBLRTM code is looking for gfortran; thus, if gfortran --version work in your terminal, things should be ok!

[reillymilburn]

Hi @gully,

Thank you! After doing brew install gfortran TelFit installed successfully and I was able to run the first fit example!

The full echelle example has a syntax error and was not able to run. I got this error message back:

File "Fit.py", line 189 print "\n***\nFitting order %i: " % (i) ^ SyntaxError: invalid syntax

Aside from that, pip install seems to be working for fresh machines!

Thanks, Reilly

[gully]

Thanks @shihyuntang for sharing guidance on gfortran for Mac. That Fit.py Syntax error is a hold-over from Python 2 days. @reillymilburn are you willing to modify that single line of code to be python 3 compatible and submit a pull request?

I would say the premise of this Issue has been resolved--- in at lease one case we can do a fresh install with pip that works. I will open a new issue regarding "Remove lingering Python 2 print statements".

[irebollido]

Hi! I am unable to install Delft using pip, I get a bunch of errors, the first one when trying to build the wheel. I can't install via brew, as I do not have sudo permissions in my computer (work-related). Is there a workaround for this?

[kgullikson88]

@irebollido I think you should also be able to install gfortran in your conda environment (which will be in your user directory and therefore you have permissions) with conda install gfortran

There may also be a way to have brew install things for the user, but I am not familiar with that