Installation woes due to not originally having Python 3

[svw26]

Hiya Matt!

So it's crunch time for me in trying this out this week... :see_no_evil:

I have a new-ish laptop (Mac OS Mojave), but it still only came with Python 2.7 (located at /Users/sarahwhite/anaconda2/bin/python). So I also installed Python 3.5 via another Anaconda (now at /Users/sarahwhite/anaconda3/bin/python). In order to avoid conflicts, I thought I'd best create a virtual environment, specifically for running your pipeline. As Python 2.7 remains the default, I did this using virtualenv as follows:

virtualenv -p /Users/sarahwhite/anaconda3/bin/python rsspipeline_env
source rsspipeline_env/bin/activate

I then pip install all the various packages, and have a go at running the pipeline. As I don't have any masks, I thought I'd try 'all' and see what happens:

rss_mos_reducer product_xmmlss0577714 reduced_xmmlss0577714 all

Unfortunately this gives the following Traceback:

Traceback (most recent call last):
  File "/Users/sarahwhite/Documents/SALT/RQAGN/reduction/try_MattHiltons_pipeline/rsspipeline_env/bin/rss_mos_reducer", line 34, in <module>
    from RSSMOSPipeline import RSSMOSTools
  File "/Users/sarahwhite/Documents/SALT/RQAGN/reduction/try_MattHiltons_pipeline/rsspipeline_env/lib/python3.5/site-packages/RSSMOSPipeline/__init__.py", line 23, in <module>
    from . import RSSMOSTools
  File "/Users/sarahwhite/Documents/SALT/RQAGN/reduction/try_MattHiltons_pipeline/rsspipeline_env/lib/python3.5/site-packages/RSSMOSPipeline/RSSMOSTools.py", line 27, in <module>
    import matplotlib.pyplot as plt
  File "/Users/sarahwhite/Documents/SALT/RQAGN/reduction/try_MattHiltons_pipeline/rsspipeline_env/lib/python3.5/site-packages/matplotlib/pyplot.py", line 2372, in <module>
    switch_backend(rcParams["backend"])
  File "/Users/sarahwhite/Documents/SALT/RQAGN/reduction/try_MattHiltons_pipeline/rsspipeline_env/lib/python3.5/site-packages/matplotlib/pyplot.py", line 207, in switch_backend
    backend_mod = importlib.import_module(backend_name)
  File "/Users/sarahwhite/anaconda3/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/sarahwhite/Documents/SALT/RQAGN/reduction/try_MattHiltons_pipeline/rsspipeline_env/lib/python3.5/site-packages/matplotlib/backends/backend_macosx.py", line 14, in <module>
    from matplotlib.backends import _macosx
ImportError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information.

I tried installing the python.app as they suggest, conda install -c anaconda python.app, but this just gives me Python-2.7 builds again, and I can't seem to find a Python-3 equivalent...

And I get the same Traceback when I instead try using conda to create the virtual environment, using the commands:

conda create --name rsspipeline_conda python=3.5
conda activate rsspipeline_conda
conda install numpy scipy astropy matplotlib IPython 
pip install RSSMOSPipeline
pip install cython
rss_mos_reducer product_xmmlss0577714 reduced_xmmlss0577714 all

So maybe I just need to fix a path somewhere? My head's in a bit of a spin now...

Thanks, Sarah x

[mattyowl]

Sorry, I don't do Macs so I'm not sure how much help I can be. This isn't an issue with installing RSSMOSPipeline, this is a general problem with your python installation (so far as I can see).

I'm not aware that you'd need a virtual environment when using anaconda - if you have anaconda installed and set-up (so that 'python' is launching the anaconda-installed python3), then you should just be able to install RSSMOSPipeline using the setup.py script in the usual way. Similarly, if you have pip installed in your anaconda distribution, that should also work. I'm happily using the pipeline through an anaconda3 installation on one of the machines here (running Linux though, not MacOS). I've never bothered with any virtualenv installs.

Maybe check that you don't have a copy of the script rss_mos_reducer somewhere else in your path (whatever the equivalent of /usr/local/bin is on MacOS)? Perhaps when you run rss_mos_reducer, it's finding one that's using your python2.7 installation? That's just a guess. You might be able to use 'which rss_mos_reducer' to check where the script that you are running has been installed, and then check the first line ('#!/usr/bin env python' would have been re-written with the path to the interpreter that was used to run the setup.py script).

[tushar-nitave]

Refer this https://stackoverflow.com/questions/21784641/installation-issue-with-matplotlib-python

[svw26]

Hey both. Thanks for your quick responses, but unfortunately the Python installations on my Mac got messed up, and I've (still) not had much of a gap to sort things out... Will report back when I do. In the meantime, I pointed my student to your pipeline, @mattyowl . He may open/raise a different issue soon. x

[svw26]

Hiya again,

So I've fixed the installation now (so that it runs within a virtual environment). Not sure which of the hidden files (.bash_profile or .bash_profile-anaconda.bak) made the difference, but I edited the pathnames in both for good measure (changing 'anaconda3' to just 'anaconda'). That at least got me back to reproducing the above Traceback.

Then, following the solution linked by @tushar-nitave (thank you!), I

• checked the contents of ~/.matplotlib,
• created a new file called 'matplotlibrc'
• wrote this single line in the file: backend: TkAgg

[Writing this out in case it helps others.]

The ImportError: Python is not installed as a framework message went away and was replaced by ImportError: No module named 'IPython'. So just do pip install ipython.

Running rss_mos_reducer product reduced all then got me:

2020-10-15 21:57:56,589 - RSSMOSPipeline - INFO - started: 2020-10-15T21:57:56.589505
2020-10-15 21:57:56,589 - RSSMOSPipeline - INFO - parameters: Namespace(excludeMasks='', extensionsList='all', iterativeMethod=False, longslitThreshold=2.0, maskName='all', rawDir='product', reducedDir='reduced', skipDone=False, subFrac=0.8, threshold=0.1)
2020-10-15 21:57:56,590 - RSSMOSPipeline - INFO - Reading image headers (cache file location: product/imageInfo.pkl)
2020-10-15 21:57:56,748 - RSSMOSPipeline - WARNING - Skipping mask 'G4Jy 1657_PL0200N001' as no wavelength calibration reference model currently exists.
2020-10-15 21:57:56,748 - RSSMOSPipeline - INFO -     Use rss_mos_create_arc_model to make a reference model and then re-run.
2020-10-15 21:57:56,748 - RSSMOSPipeline - INFO -     arc files list: ['product/mbxgpP202010130062.fits']

so I can close this issue now.

Thanks, Sarah x

[EDIT: adding the "(changing 'anaconda3' to just 'anaconda')" bit.]

[svw26]

[Message for myself and other silly Mac users...]

For some other code on my laptop I followed the prompt to update my version of pip, to 21.1.1. This appears to cause all sorts of problems (including recursive ones?) but I have found this: https://stackoverflow.com/questions/65896334/python-pip-broken-with-sys-stderr-writeferror-exc

The following command: sudo easy_install pip==20.3.4 appears to put things back to some sort of normal (EDIT: confirmed). x