Updating Docker to run with Fermitools 2.0.0

[jleagle94]

Hi all,

I use the docker container (currently up to date at version 2.3.0.5) to pull the fssc/fermibottle and then do my analysis that way. The fermibottle container is behind (conda 4.6.14 with fermitools 1.0.5) so I tried to update conda in order to update fermitools to 2.0.0 and here is the prescription I followed:

docker start 5b
 docker attach 5b
 (base) [fermi@5b0b1bac1d2e ~]$ conda info

     active environment : base
    active env location : /opt/anaconda
            shell level : 1
       user config file : /home/fermi/.condarc
 populated config files : /home/fermi/.condarc
          conda version : 4.6.14
    conda-build version : not installed
         python version : 2.7.15.final.0
       base environment : /opt/anaconda  (writable)
           channel URLs : https://conda.anaconda.org/fermi/linux-64
                          https://conda.anaconda.org/fermi/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/anaconda/pkgs
                          /home/fermi/.conda/pkgs
       envs directories : /opt/anaconda/envs
                          /home/fermi/.conda/envs
               platform : linux-64
             user-agent : conda/4.6.14 requests/2.21.0 CPython/2.7.15 Linux/4.19.76-linuxkit centos/6.10 glibc/2.12
                UID:GID : 501:501
             netrc file : None
           offline mode : False

To update conda to 4.8.* I run the following

(base) [fermi@5b0b1bac1d2e ~]$ sudo chown -R fermi /opt/anaconda/
(base) [fermi@5b0b1bac1d2e ~]$ sudo chmod -R +x /opt/anaconda/
(base) [fermi@5b0b1bac1d2e ~]$ conda update conda

This allows conda to update to 4.8.3.

Then I created my python3 environment to prepare for fermitools 2.0* (base) [fermi@5b0b1bac1d2e ~]$ conda create --named python3 python=3.7

I activated my python3 environment and then downloaded the fermitools 2.0.0 (python3) [fermi@5b0b1bac1d2e ~]$ conda create --name fermi -c conda-forge -c fermi fermitools (which removes my old fermi (1.0.5) environment) I like to use fermipy so I download this using "conda install fermipy" within my fermi environment. I can see everything is up to date now

(fermi) [fermi@5b0b1bac1d2e snr_g310.6-1.6]$ conda list fermi
 #packages in environment at /opt/anaconda/envs/fermi:
#
 Name                    Version                   Build  Channel
fermipy                   0.20.0           py37h7b7c402_0    conda-forge
fermitools                2.0.0                hf484d3e_0    fermi
fermitools-data           0.18                          0    fermi
fermitools-dependencies   2.0.0                         0    fermi

so when I open python and import fermipy, things run as usual until I try to read in my configuration file.

(fermi) [fermi@5b0b1bac1d2e snr_g310.6-1.6]$ python
Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:25:08) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fermipy.gtanalysis import GTAnalysis
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-0umhz0e6 because the default path (/home/fermi/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
WARNING: AstropyDeprecationWarning: astropy.extern.six will be removed in 4.0, use the six module directly if it is still needed [astropy.extern.six]
>>> gta = GTAnalysis('/data/snr_g310.6-1.6/config_all_events_20deg_roi.yaml',logging={'verbosity' : 3})
/opt/anaconda/envs/fermi/lib/python3.7/site-packages/fermipy/config.py:213: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config_dict = yaml.load(open(config))
Traceback (most recent call last):
  File "/opt/anaconda/envs/fermi/lib/python3.7/site-packages/fermipy/__init__.py", line 60, in get_ft_conda_version
    lines = subprocess.check_output(['conda', 'list', '-f',  'fermitools']).decode().split('\n')
  File "/opt/anaconda/envs/fermi/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/opt/anaconda/envs/fermi/lib/python3.7/subprocess.py", line 488, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/opt/anaconda/envs/fermi/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/opt/anaconda/envs/fermi/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'conda': 'conda'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda/envs/fermi/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 317, in __init__
    'ScienceTools version %s', fermipy.get_st_version())
  File "/opt/anaconda/envs/fermi/lib/python3.7/site-packages/fermipy/__init__.py", line 37, in get_st_version
    vv = get_ft_conda_version()
  File "/opt/anaconda/envs/fermi/lib/python3.7/site-packages/fermipy/__init__.py", line 62, in get_ft_conda_version
    lines = subprocess.check_output(['conda', 'list', '-f',  'fermitools']).split('\n')
  File "/opt/anaconda/envs/fermi/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/opt/anaconda/envs/fermi/lib/python3.7/subprocess.py", line 488, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/opt/anaconda/envs/fermi/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/opt/anaconda/envs/fermi/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'conda': 'conda'

There is nowhere in my config file that says 'conda'. I have checked with older config files that I know work and I still get the same error. What is going on?

Thanks

[Areustle]

Thank you for the detailed report. I will generate a new version of the Fermibottle docker container with updated versions of conda, the Fermitools, and other software.

It appears that your final issue is being caused by a bug in the way fermipy is interacting with conda, rather than the Fermitools themselves. I will try to reproduce and fix that issue within Fermibottle

[Areustle]

This issue will now be resolved in future versions of the FermiBottle. The problem @jleagle94 experienced came from minor differences in the python2 and python3 versions of the subprocess library. These differences should pose no problem outside of the FermiBottle docker container.

Explicitly when the fermi conda environment is activated the "conda" executable is no longer in the $PATH. It remains callable however as conda init sets it as a bash shell alias. The python2 version of subprocess respects the shell alias, while the python3 version does not!

@jleagle94 you should be able to solve this in your local docker container by running this command from within the container: sudo ln -s /opt/anaconda/bin/conda /home/fermi/astrosoft/bin/conda

Then you can check afterwards by running this command python -c "import fermipy; print(fermipy.get_ft_conda_version())" which should return "2.0.0"

[Areustle]

Dev image now available on dockerhub: https://hub.docker.com/layers/fssc/fermibottle/dev/images/sha256-edc69b9c4bdece730632c9b9209caaa2d7ebeae94f37346bd9b3737b4e26c43a?context=repo

[jleagle94]

This indeed fixes the issue (thank you! so quick too :D) but unfortunately another one arises that occurs on both Docker and my new python3 Fermitools 2.0 environment on my local machine (MacOS Catalina 10.15.6). It is still when I read in the config file. It now reads as a python runtime error:

>>> gta = GTAnalysis('test.yaml',logging={'verbosity' : 3})
/Users/jordaneagle/anaconda2/envs/fermi/lib/python3.7/site-packages/fermipy/config.py:213: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config_dict = yaml.load(open(config))
2020-09-29 18:20:08 INFO    GTAnalysis.__init__(): 
--------------------------------------------------------------------------------
fermipy version 0.20.0 
ScienceTools version 2.0.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jordaneagle/anaconda2/envs/fermi/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 356, in __init__
    comp = self._create_component(cfg, loglevel=self.loglevel)
  File "/Users/jordaneagle/anaconda2/envs/fermi/lib/python3.7/site-packages/fermipy/gtanalysis.py", line 942, in _create_component
    for k in cfg.keys():
RuntimeError: dictionary changed size during iteration

I thought it was a conflicting problem with Catalina but since it now occurs on the Docker set up (after running the sudo ln -s /opt/anaconda/bin/conda /home/fermi/astrosoft/bin/conda) I'm not so sure.

[Areustle]

This looks more like an issue with fermipy than with the fermitools or the docker container. Perhaps they failed to properly update to the new python3 way of iterating over a dictionary. I recommended taking it up with them in either their GitHub repo or the slack channel.

[jleagle94]

ok great, will contact them then, thanks!