search

galsci / mapsims

Map based simulations software for CMB Experiments
BSD 2-Clause "Simplified" License
3 stars 1 forks source link

Very slow installation #79

Closed tskisner closed 2 years ago

tskisner commented 3 years ago

While attempting to install this package with pip at NERSC, it seems like pip / poetry is downloading many different versions of dependency wheels, amounting to hundreds of MB. Is this the expected behavior? Output of 

python3 -m pip install --prefix <foo> .

was the following:

Processing /global/cscratch1/sd/sobs/build/mapsims-2.4.0
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Requirement already satisfied: healpy<2.0.0,>=1.14.0 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from mapsims==2.4.0) (1.14.0)
Requirement already satisfied: so_noise_models==3.1.1 in /global/common/software/sobs/cori/gcc_20210629/lib/python3.7/site-packages (from mapsims==2.4.0) (3.1.1)
Requirement already satisfied: numpy<2.0.0,>=1.18.0 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from mapsims==2.4.0) (1.19.4)
Requirement already satisfied: pysm3<4.0.0,>=3.3.0 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from mapsims==2.4.0) (3.3.0)
Requirement already satisfied: astropy<5,>=4 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from mapsims==2.4.0) (4.2.1)
Requirement already satisfied: toml<0.11.0,>=0.10.1 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from mapsims==2.4.0) (0.10.2)
Requirement already satisfied: pyyaml<6,>=5 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from mapsims==2.4.0) (5.3.1)
Requirement already satisfied: so-pysm-models<3.0.0,>=2.4.0 in /global/common/software/sobs/cori/gcc_20210629/lib/python3.7/site-packages (from mapsims==2.4.0) (2.4.0)
Collecting pixell<0.11.0,>=0.10.3
  Downloading pixell-0.10.3-cp37-cp37m-manylinux2010_x86_64.whl (9.8 MB)
Collecting importlib-metadata<4.0.0,>=3.4.0
  Downloading importlib_metadata-3.10.1-py3-none-any.whl (14 kB)
Requirement already satisfied: pyerfa in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from astropy<5,>=4->mapsims==2.4.0) (2.0.0)
Requirement already satisfied: scipy in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from healpy<2.0.0,>=1.14.0->mapsims==2.4.0) (1.5.3)
Requirement already satisfied: six in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from healpy<2.0.0,>=1.14.0->mapsims==2.4.0) (1.16.0)
Requirement already satisfied: matplotlib in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from healpy<2.0.0,>=1.14.0->mapsims==2.4.0) (3.2.2)
Requirement already satisfied: zipp>=0.5 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from importlib-metadata<4.0.0,>=3.4.0->mapsims==2.4.0) (3.4.1)
Requirement already satisfied: typing-extensions>=3.6.4 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from importlib-metadata<4.0.0,>=3.4.0->mapsims==2.4.0) (3.7.4.3)
Requirement already satisfied: setuptools>=39 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (49.6.0.post20210108)
Requirement already satisfied: cython>=0.28 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (0.29.23)
Requirement already satisfied: pytest>=4.6 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (6.2.4)
Requirement already satisfied: h5py>=2.7<=2.10 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (3.2.1)
Requirement already satisfied: python-dateutil>=2.7 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (2.8.1)
Requirement already satisfied: pytest-cov>=2.6 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (2.12.0)
Requirement already satisfied: Pillow>=5.3.0 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (7.2.0)
Requirement already satisfied: coveralls>=1.5 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (3.1.0)
Requirement already satisfied: requests>=1.0.0 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from coveralls>=1.5->pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (2.24.0)
Requirement already satisfied: coverage<6.0,>=4.1 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from coveralls>=1.5->pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (5.5)
Requirement already satisfied: docopt>=0.6.1 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from coveralls>=1.5->pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (0.6.2)
Requirement already satisfied: cached-property in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_aux/lib/python3.7/site-packages (from h5py>=2.7<=2.10->pixell<0.11.0,>=0.10.3->mapsims==2.4.0) (1.5.2)
Requirement already satisfied: cycler>=0.10 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from matplotlib->healpy<2.0.0,>=1.14.0->mapsims==2.4.0) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from matplotlib->healpy<2.0.0,>=1.14.0->mapsims==2.4.0) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from matplotlib->healpy<2.0.0,>=1.14.0->mapsims==2.4.0) (1.3.1)
Requirement already satisfied: numba>0.49.1 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from pysm3<4.0.0,>=3.3.0->mapsims==2.4.0) (0.53.1)
INFO: pip is looking at multiple versions of pyparsing to determine which version is compatible with other requirements. This could take a while.
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
  Downloading pyparsing-2.4.6-py2.py3-none-any.whl (67 kB)
  Downloading pyparsing-2.4.5-py2.py3-none-any.whl (67 kB)
  Downloading pyparsing-2.4.4-py2.py3-none-any.whl (67 kB)
  Downloading pyparsing-2.4.3-py2.py3-none-any.whl (67 kB)
  Downloading pyparsing-2.4.2-py2.py3-none-any.whl (65 kB)
  Downloading pyparsing-2.4.1.1-py2.py3-none-any.whl (64 kB)
INFO: pip is looking at multiple versions of pyparsing to determine which version is compatible with other requirements. This could take a while.
  Downloading pyparsing-2.4.0-py2.py3-none-any.whl (62 kB)
  Downloading pyparsing-2.3.1-py2.py3-none-any.whl (61 kB)
  Downloading pyparsing-2.3.0-py2.py3-none-any.whl (59 kB)
  Downloading pyparsing-2.2.2-py2.py3-none-any.whl (57 kB)
  Downloading pyparsing-2.2.1-py2.py3-none-any.whl (57 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56 kB)
  Downloading pyparsing-2.1.10-py2.py3-none-any.whl (56 kB)
  Downloading pyparsing-2.1.9-py2.py3-none-any.whl (55 kB)
  Downloading pyparsing-2.1.8-py2.py3-none-any.whl (54 kB)
  Downloading pyparsing-2.1.7-py2.py3-none-any.whl (53 kB)
  Downloading pyparsing-2.1.5-py2.py3-none-any.whl (42 kB)
  Downloading pyparsing-2.1.4-py2.py3-none-any.whl (40 kB)
  Downloading pyparsing-2.1.3-py2.py3-none-any.whl (39 kB)
  Downloading pyparsing-2.1.1-py2.py3-none-any.whl (38 kB)
  Downloading pyparsing-2.1.0-py2.py3-none-any.whl (38 kB)
  Downloading pyparsing-2.0.7-py2.py3-none-any.whl (38 kB)
  Downloading pyparsing-2.0.6-py2.py3-none-any.whl (38 kB)
  Downloading pyparsing-2.0.5-py2.py3-none-any.whl (38 kB)
  Downloading pyparsing-2.0.3-py2.py3-none-any.whl (37 kB)
  Downloading pyparsing-2.0.2.zip (1.3 MB)
  Downloading pyparsing-2.0.1.zip (1.3 MB)
INFO: pip is looking at multiple versions of pillow to determine which version is compatible with other requirements. This could take a while.
Collecting Pillow>=5.3.0
  Downloading Pillow-8.2.0-cp37-cp37m-manylinux1_x86_64.whl (3.0 MB)
  Downloading Pillow-8.1.2-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
  Downloading Pillow-8.1.1-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
  Downloading Pillow-8.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
  Downloading Pillow-8.0.1-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
  Downloading Pillow-8.0.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
  Downloading Pillow-7.2.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
INFO: pip is looking at multiple versions of pillow to determine which version is compatible with other requirements. This could take a while.
  Using cached Pillow-7.1.2-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
  Downloading Pillow-7.1.1-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
  Downloading Pillow-7.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
  Downloading Pillow-7.0.0-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
  Downloading Pillow-6.2.2-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading Pillow-6.2.1-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
  Downloading Pillow-6.2.0-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
  Downloading Pillow-6.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
  Downloading Pillow-6.0.0-cp37-cp37m-manylinux1_x86_64.whl (2.0 MB)
  Downloading Pillow-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (2.0 MB)
  Downloading Pillow-5.4.0-cp37-cp37m-manylinux1_x86_64.whl (2.0 MB)
  Downloading Pillow-5.3.0-cp37-cp37m-manylinux1_x86_64.whl (2.0 MB)
INFO: pip is looking at multiple versions of kiwisolver to determine which version is compatible with other requirements. This could take a while.
Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.3.1-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB)
  Downloading kiwisolver-1.2.0-cp37-cp37m-manylinux1_x86_64.whl (88 kB)
  Downloading kiwisolver-1.1.0-cp37-cp37m-manylinux1_x86_64.whl (90 kB)
  Downloading kiwisolver-1.0.1-cp37-cp37m-manylinux1_x86_64.whl (89 kB)
INFO: pip is looking at multiple versions of cycler to determine which version is compatible with other requirements. This could take a while.
Collecting cycler>=0.10
  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
INFO: pip is looking at multiple versions of kiwisolver to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of matplotlib to determine which version is compatible with other requirements. This could take a while.
Collecting matplotlib
  Downloading matplotlib-3.4.2-cp37-cp37m-manylinux1_x86_64.whl (10.3 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading matplotlib-3.4.1-cp37-cp37m-manylinux1_x86_64.whl (10.3 MB)
  Downloading matplotlib-3.4.0-cp37-cp37m-manylinux1_x86_64.whl (10.3 MB)
INFO: pip is looking at multiple versions of cycler to determine which version is compatible with other requirements. This could take a while.
  Downloading matplotlib-3.3.4-cp37-cp37m-manylinux1_x86_64.whl (11.5 MB)
  Downloading matplotlib-3.3.3-cp37-cp37m-manylinux1_x86_64.whl (11.6 MB)
  Downloading matplotlib-3.3.2-cp37-cp37m-manylinux1_x86_64.whl (11.6 MB)
Requirement already satisfied: certifi>=2020.06.20 in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from matplotlib->healpy<2.0.0,>=1.14.0->mapsims==2.4.0) (2020.12.5)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of certifi to determine which version is compatible with other requirements. This could take a while.
Collecting certifi>=2020.06.20
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
  Downloading certifi-2020.11.8-py2.py3-none-any.whl (155 kB)
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting matplotlib
  Downloading matplotlib-3.3.1-cp37-cp37m-manylinux1_x86_64.whl (11.6 MB)
INFO: pip is looking at multiple versions of certifi to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of matplotlib to determine which version is compatible with other requirements. This could take a while.
  Downloading matplotlib-3.3.0-1-cp37-cp37m-manylinux1_x86_64.whl (11.5 MB)
  Downloading matplotlib-3.2.2-cp37-cp37m-manylinux1_x86_64.whl (12.4 MB)
  Downloading matplotlib-3.2.1-cp37-cp37m-manylinux1_x86_64.whl (12.4 MB)
  Downloading matplotlib-3.2.0-cp37-cp37m-manylinux1_x86_64.whl (12.4 MB)
  Downloading matplotlib-3.1.3-cp37-cp37m-manylinux1_x86_64.whl (13.1 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Downloading matplotlib-3.1.2-cp37-cp37m-manylinux1_x86_64.whl (13.1 MB)
  Downloading matplotlib-3.1.1-cp37-cp37m-manylinux1_x86_64.whl (13.1 MB)
  Downloading matplotlib-3.1.0-cp37-cp37m-manylinux1_x86_64.whl (13.1 MB)
  Downloading matplotlib-3.0.3-cp37-cp37m-manylinux1_x86_64.whl (13.0 MB)
  Downloading matplotlib-3.0.2-cp37-cp37m-manylinux1_x86_64.whl (12.9 MB)
  Downloading matplotlib-3.0.1-cp37-cp37m-manylinux1_x86_64.whl (12.9 MB)
  Downloading matplotlib-3.0.0-cp37-cp37m-manylinux1_x86_64.whl (12.8 MB)
  Downloading matplotlib-2.2.5-cp37-cp37m-manylinux1_x86_64.whl (12.8 MB)
Requirement already satisfied: pytz in /global/common/software/cmb/cori/cmbenv-gcc_20210527/cmbenv_python/lib/python3.7/site-packages (from matplotlib->healpy<2.0.0,>=1.14.0->mapsims==2.4.0) (2021.1)
  Downloading matplotlib-2.2.4-cp37-cp37m-manylinux1_x86_64.whl (12.8 MB)
ERROR: Operation cancelled by user

As you can see I finally killed it after 20 minutes. I'll try again and let it run for longer, but it seems kind of crazy...

zonca commented 3 years ago

I'll probably remove poetry and use something more reasonable sometime in September or October.

tskisner commented 3 years ago

The installation eventually died with an error trying to install pyparsing. I then generated a setup.py from the pyproject.toml using a slightly modifed version of this script. The resulting setup.py had the same problem. I think the issue is some kind of mutually exclusive version requirements of the dependencies. Can you tell me what python version(s) you have used successfully with this?

zonca commented 3 years ago

In Github action it was working with 3.6 7 8

zonca commented 2 years ago

switched from poetry to flit in https://github.com/galsci/mapsims/commit/1942d3193bcab083f943fdd18f149fc25c5ea7fa this should hopefully be fixed.