conda installation issue [BUG]

[oscarhdz7]

I originally had the same installation issue as described in Installation issue #248 when I tried to install on my MacOS 10.14.6 with conda install -c conda-forge 21cmFAST.

Since I saw that this was version 3.03 I removed it and I followed the suggestion in #248 and installed version 3.1.1 as follows conda install -c conda-forge 21cmFAST=3.1.1

But when I ran "import py21cmFAST" I continued to get the following:

Python 3.7.3 (default, Mar 27 2019, 16:54:48) [Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin Type "help", "copyright", "credits" or "license" for more information.

import py21cmfast as p21c /Users/oscar/anaconda3/lib/python3.7/site-packages/py21cmfast/_cfg.py:50: UserWarning: Your configuration file is out of date. Updating... "Your configuration file is out of date. Updating..." /Users/oscar/anaconda3/lib/python3.7/site-packages/py21cmfast/_cfg.py:33: UserWarning: Your configuration file is out of date. Updating... warnings.warn("Your configuration file is out of date. Updating...") Traceback (most recent call last): File "", line 1, in File "/Users/oscar/anaconda3/lib/python3.7/site-packages/py21cmfast/init.py", line 17, in from . import cache_tools, inputs, outputs, plotting, wrapper File "/Users/oscar/anaconda3/lib/python3.7/site-packages/py21cmfast/cache_tools.py", line 9, in from . import outputs, wrapper File "/Users/oscar/anaconda3/lib/python3.7/site-packages/py21cmfast/outputs.py", line 26, in from . import _utils as _ut File "/Users/oscar/anaconda3/lib/python3.7/site-packages/py21cmfast/_utils.py", line 18, in from .c_21cmfast import lib ImportError: dlopen(/Users/oscar/anaconda3/lib/python3.7/site-packages/py21cmfast/c_21cmfast.abi3.so, 2): Library not loaded: @rpath/libfftw3f_omp.3.dylib Referenced from: /Users/oscar/anaconda3/lib/python3.7/site-packages/py21cmfast/c_21cmfast.abi3.so Reason: image not found

[steven-murray]

Hi @oscarhdz7, sorry to hear you're having trouble. Can you list the output of conda list?

[oscarhdz7]

The conda list output is below. I should add that since my last post I removed Anaconda and did a new installation of Anaconda and I created a new environment with conda into which I installed with conda install -c conda-forge 21cmFAST=3.1.1

Last login: Mon Aug 30 11:47:05 on ttys000 (base) Mac13o:~ oscar$ conda activate 21cmfast (21cmfast) Mac13o:~ oscar$ conda list

packages in environment at /Users/oscar/opt/anaconda3/envs/21cmfast:

#

Name Version Build Channel

21cmfast 3.1.1 py39hffa0f39_0 conda-forge astropy 4.3.1 py39hc89836e_0 conda-forge bidict 0.21.2 py_0 conda-forge c-ares 1.17.2 h0d85af4_0 conda-forge ca-certificates 2021.5.30 h033912b_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge certifi 2021.5.30 py39h6e9494a_0 conda-forge cffi 1.14.6 py39hb71fe58_0 conda-forge click 8.0.1 py39h6e9494a_0 conda-forge cycler 0.10.0 py_2 conda-forge fftw 3.3.9 nompi_h02cd531_101 conda-forge freetype 2.10.4 h4cff582_1 conda-forge gsl 2.6 h71c5fe9_2 conda-forge h5py 3.4.0 nompi_py39hbc6cb89_100 conda-forge hdf5 1.12.1 nompi_h2f0ef1a_100 conda-forge importlib-metadata 4.8.1 py39h6e9494a_0 conda-forge jbig 2.1 h0d85af4_2003 conda-forge jpeg 9d hbcb3906_0 conda-forge kiwisolver 1.3.2 py39hf018cea_0 conda-forge krb5 1.19.2 hcfbf3a7_0 conda-forge lcms2 2.12 h577c468_0 conda-forge lerc 2.2.1 h046ec9c_0 conda-forge libblas 3.9.0 11_osx64_openblas conda-forge libcblas 3.9.0 11_osx64_openblas conda-forge libcurl 7.78.0 hf45b732_0 conda-forge libcxx 12.0.1 habf9029_0 conda-forge libdeflate 1.7 h35c211d_5 conda-forge libedit 3.1.20191231 h0678c8f_2 conda-forge libev 4.33 haf1e3a3_1 conda-forge libffi 3.3 h046ec9c_2 conda-forge libgfortran 5.0.0 9_3_0_h6c81a4c_23 conda-forge libgfortran5 9.3.0 h6c81a4c_23 conda-forge liblapack 3.9.0 11_osx64_openblas conda-forge libnghttp2 1.43.0 h07e645a_0 conda-forge libopenblas 0.3.17 openmp_h3351f45_1 conda-forge libpng 1.6.37 h7cec526_2 conda-forge libssh2 1.9.0 h52ee1ee_6 conda-forge libtiff 4.3.0 h1167814_1 conda-forge libwebp-base 1.2.1 h0d85af4_0 conda-forge llvm-openmp 12.0.1 hda6cdc1_1 conda-forge lz4-c 1.9.3 he49afe7_1 conda-forge matplotlib-base 3.4.3 py39hb07454d_0 conda-forge ncurses 6.2 h2e338ed_4 conda-forge nomkl 1.0 h5ca1d4c_0 conda-forge numpy 1.21.2 py39h7eed0ac_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjpeg 2.4.0 h6e7aa92_1 conda-forge openssl 1.1.1k h0d85af4_1 conda-forge pillow 8.3.1 py39he9bb72f_0 conda-forge pip 21.2.4 pyhd8ed1ab_0 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pyerfa 2.0.0 py39hc89836e_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge python 3.9.6 hd187cdc_1_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.9 2_cp39 conda-forge pyyaml 5.4.1 py39h89e85a6_1 conda-forge readline 8.1 h05e3726_0 conda-forge scipy 1.7.1 py39h056f1c0_0 conda-forge setuptools 57.4.0 py39h6e9494a_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sqlite 3.36.0 h23a322b_0 conda-forge tk 8.6.11 hd798d34_0 conda-forge tornado 6.1 py39h89e85a6_1 conda-forge tzdata 2021a he74cb21_1 conda-forge wheel 0.37.0 pyhd8ed1ab_1 conda-forge xz 5.2.5 haf1e3a3_1 conda-forge yaml 0.2.5 haf1e3a3_0 conda-forge zipp 3.5.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h7795811_1010 conda-forge zstd 1.5.0 h582d3a0_0 conda-forge (21cmfast) Mac13o:~ oscar$

[oscarhdz7]

And even with the reinstall, the same problems remain. Here is what I got when I tried to import py21cmfast

(21cmfast) Mac13o:~ oscar$ python Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:36:15) [Clang 11.1.0 ] on darwin Type "help", "copyright", "credits" or "license" for more information.

import py21cmfast as p21c /Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_cfg.py:49: UserWarning: Your configuration file is out of date. Updating... warnings.warn( /Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_cfg.py:33: UserWarning: Your configuration file is out of date. Updating... warnings.warn("Your configuration file is out of date. Updating...")

[steven-murray]

Hmmm, I think that warning is OK (it just means you previously installed 21cmFAST when it had an older configuration file). But it doesn't seem to give the rest of the error any more? Or did you just omit that in the copy in your last comment? Does running basic commands like p21c.initial_conditions() work OK?

[oscarhdz7]

Below is what I get when I run p21c.initial_conditions() . How should I resolve the configuration file warnings?

p21c.initial_conditions() InitialConditions(UserParams(BOX_LEN:300.0, DIM:600, FAST_FCOLL_TABLES:False, HII_DIM:200, HMF:1, MINIMIZE_MEMORY:False, NO_RNG:False, N_THREADS:1, PERTURB_ON_HIGH_RES:False, POWER_SPECTRUM:0, USE_2LPT:True, USE_FFTW_WISDOM:False, USE_INTERPOLATION_TABLES:False, USE_RELATIVE_VELOCITIES:False); CosmoParams(OMb:0.04897468161869667, OMm:0.30964144154550644, POWER_INDEX:0.9665, SIGMA_8:0.8102, hlittle:0.6766); GlobalParams(CRIT_DENS_TRANSITION=1.5;DELTA_CRIT_MODE=1;DELTA_R_FACTOR=1.100000023841858;FILTER=0;HALO_FILTER=0;MIN_DENSITY_LOW_LIMIT=9.000000034120603e-08;M_MAX_INTEGRAL=1.0000000272564224e+16;M_MIN_INTEGRAL=100000.0;M_WDM=2.0;NBINS_LF=100;OMk=0.0;OMn=0.0;OMr=8.600000001024455e-05;OMtot=1.0;OPTIMIZE=0;OPTIMIZE_MIN_MASS=99999997952.0;P_CUTOFF=0;PhotonConsAsymptoteTo=0.009999999776482582;PhotonConsEnd=0.30000001192092896;PhotonConsEndCalibz=5.0;PhotonConsStart=0.9950000047683716;R_BUBBLE_MIN=0.6203504800796509;R_OVERLAP_FACTOR=1.0;RecombPhotonCons=0;SHETH_b=0.15000000596046448;SHETH_c=0.05000000074505806;T_RE=20000.0;USE_FAST_ATOMIC=False;VAVG=25.860000610351562;Y_He=0.24500000476837158;Zreion_HeII=3.0;g_x=1.5;wl=-1.0); v0.0)_random_seed=585522457268

[oscarhdz7]

Again referring to my config file warnings, I saw that v3.03 has a user_data directory but I could not find such a thing in v3.1.1.

[steven-murray]

OK, looks like things are working fine for you now.

As for the config warnings, it would be a good idea for us to fix that up automatically, but I think you should be able to get rid of them by doing the following:

import py21cmfast as p21c
p21c.config.write()

This will write the auto-updated configuration to the file ~/.21cmfast/config.yml, and then every time you start 21cmFAST it should have an up-to-date configuration :-)

As for your last question about the user_data directory: you are right that it no longer exists -- or at least, it has been moved to within the package itself, which makes it easier for the code to locate at run-time.

[oscarhdz7]

The p21c.config.write() did not seem to work (see below). Should I just copy the user_data directory to ~/.21cmfast ?

p21c.config.write() Traceback (most recent call last): File "", line 1, in File "/Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_cfg.py", line 78, in write fname = Path(fname or self.file_name) File "/Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/pathlib.py", line 1072, in new self = cls._from_parts(args, init=False) File "/Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/pathlib.py", line 697, in _from_parts drv, root, parts = self._parse_args(args) File "/Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/pathlib.py", line 681, in _parse_args a = os.fspath(a) TypeError: expected str, bytes or os.PathLike object, not NoneType

[steven-murray]

Huh. OK try doing p21c.config.write('/YOUR/HOME/.21cmfast/config.yml')? I have made an issue for us to do this automatically (#255 ) so this shouldn't be a problem in the future.

However, I should not that it is NOT an actual problem -- the config being used is correct (it's just warning you that the one on file needs to be updated). Also, you do not need to copy any user_data or anything!

[oscarhdz7]

Thanks, that did get rid of the first set of warnings when I run import py21cmfast as p21c. However I still get all sorts of warnings when I the next line of the basic interactive example from the main documentation page. How can I address these or are they to be ignored?

import py21cmfast as p21c coeval = p21c.run_coeval( ... redshift=8.0, ... user_params={'HII_DIM': 50, "USE_INTERPOLATION_TABLES": False} ... ) /Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_utils.py:801: UserWarning: Trying to remove array that isn't yet created: hires_vx warnings.warn(f"Trying to remove array that isn't yet created: {k}") /Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_utils.py:801: UserWarning: Trying to remove array that isn't yet created: hires_vy warnings.warn(f"Trying to remove array that isn't yet created: {k}") /Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_utils.py:801: UserWarning: Trying to remove array that isn't yet created: hires_vz warnings.warn(f"Trying to remove array that isn't yet created: {k}") /Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_utils.py:801: UserWarning: Trying to remove array that isn't yet created: hires_vx_2LPT warnings.warn(f"Trying to remove array that isn't yet created: {k}") /Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_utils.py:801: UserWarning: Trying to remove array that isn't yet created: hires_vy_2LPT warnings.warn(f"Trying to remove array that isn't yet created: {k}") /Users/oscar/opt/anaconda3/envs/21cmfast/lib/python3.9/site-packages/py21cmfast/_utils.py:801: UserWarning: Trying to remove array that isn't yet created: hires_vz_2LPT warnings.warn(f"Trying to remove array that isn't yet created: {k}")

[steven-murray]

You can also ignore those warnings. There's also an issue open to remove them.

[ChenxiSSS]

I use conda install with a virtualenv set-up, also got the same Trying to remove array that isn't yet created warning, but the results seem fine.

I also got another minor issue, not worth to start a new issue: When I print the 21cmFAST version, print(f"Using 21cmFAST version {p21c.__version__}") and get Using 21cmFAST version 0.0.0. I tried other conda virtual env with different python versions, got the same 0.0.0 version. Feels kinda of scary when you first use 21cmFAST, but the results seem fine (just the in the mindset suspecting something is wrong).

[steven-murray]

Thanks @ChenxiSSS, I have also seen this behaviour (of printing 0.0.0 for the version). I think it's only when it's installed with conda. I'll try to have a look into it when I can/

Then excess warning has an issue (#239 ) and hopefully will be addressed soon!

[steven-murray]

Closing this as resolved, but feel free to reopen if anything comes up.