numpy is not installing

yarikoptic commented 1 year ago

started to happen a few days back.

$ flock -E 0 -e -n /home/dandi/.run/backup2datalad-cron.lock bash -c '/mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron --mode verify 000351'
...
Collecting numpy<1.24,>=1.16 (from pynwb!=1.1.0,>=1.0.3->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/42/38/775b43da55fa7473015eddc9a819571517d9a271a9f8134f68fb9be2f212/numpy-1.23.5.tar.gz (10.7MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting h5py<4,>=2.10 (from pynwb!=1.1.0,>=1.0.3->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/4b/ad/67136179798bef7f49eebac34024f9dc3052f4c4a5d1372fcfd5dd9aca5f/h5py-3.7.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Requirement already satisfied: setuptools in /mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages (from pynwb!=1.1.0,>=1.0.3->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7)) (41.2.0)
Collecting numcodecs>=0.10.0 (from zarr~=2.10->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/91/71/3942cb3cc8de0d702a896435c1d905a000facd55ba58afe33bd19b93f232/numcodecs-0.10.2.tar.gz
Collecting asciitree (from zarr~=2.10->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/2d/6a/885bc91484e1aa8f618f6f0228d76d0e67000b0fdd6090673b777e311913/asciitree-0.3.3.tar.gz
Requirement already satisfied: six>=1.5 in /mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages (from python-dateutil->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7)) (1.16.0)
Collecting ruamel.yaml.clib>=0.2.6; platform_python_implementation == "CPython" and python_version < "3.11" (from ruamel.yaml<1,>=0.15->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/d5/31/a3e6411947eb7a4f1c669f887e9e47d61a68f9d117f10c3c620296694a0b/ruamel.yaml.clib-0.2.7.tar.gz
Collecting blessings; sys_platform != "win32" (from pyout!=0.6.0,>=0.5->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/03/74/489f85a78247609c6b4f13733cbf3ba0d864b11aa565617b645d6fdf2a4a/blessings-1.7-py3-none-any.whl
Collecting natsort (from nwbinspector>=0.4.12->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/3e/58/61c4b4fd9e597affdcd3347d5991fa5be404af26f19932d3116b67e133da/natsort-8.2.0-py3-none-any.whl
Requirement already satisfied: PyYAML in /mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages (from nwbinspector>=0.4.12->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7)) (5.4.1)
Collecting ci-info>=0.2 (from etelemetry>=0.2.2->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/13/c3/8ac768b389d5b6dda1c3ce7992b3acd2b46401f9b71439123858b17b1a2c/ci_info-0.3.0-py3-none-any.whl
Collecting importlib-resources; python_version < "3.9" (from bidsschematools->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/c8/47/6bfe2147eae436391916b3741a2cd9a76763e9671703a0d1d8e83142816e/importlib_resources-5.10.0-py3-none-any.whl
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/54/d3/986fcfeaf62047840c571a857fb8f3ad1e9622081d5e7d0ee5e3451ca2e7/pyrsistent-0.19.2-py3-none-any.whl
Collecting pkgutil-resolve-name>=1.3.10; python_version < "3.9" (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/c9/5c/3d4882ba113fd55bdba9326c1e4c62a15e674a2501de4869e6bd6301f87e/pkgutil_resolve_name-1.3.10-py3-none-any.whl
Collecting rfc3339-validator; extra == "format" (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/7b/44/4e421b96b67b2daff264473f7465db72fbdf36a07e05494f50300cc7b0c6/rfc3339_validator-0.1.4-py2.py3-none-any.whl
Collecting webcolors>=1.11; extra == "format" (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/af/a9/71beaebcacd7dd848af470740dbe7f39aa6bd23a60b8162f8c050090eed4/webcolors-1.12-py3-none-any.whl
Collecting isoduration; extra == "format" (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/7b/55/e5326141505c5d5e34c5e0935d2908a74e4561eca44108fbfb9c13d2911a/isoduration-20.11.0-py3-none-any.whl
Collecting rfc3987; extra == "format" (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/65/d4/f7407c3d15d5ac779c3dd34fbbc6ea2090f77bd7dd12f207ccf881551208/rfc3987-1.3.8-py2.py3-none-any.whl
Collecting fqdn; extra == "format" (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/cf/58/8acf1b3e91c58313ce5cb67df61001fc9dcd21be4fadb76c1a2d540e09ed/fqdn-1.5.1-py3-none-any.whl
Collecting jsonpointer>1.13; extra == "format" (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/a3/be/8dc9d31b50e38172c8020c40f497ce8debdb721545ddb9fcb7cca89ea9e6/jsonpointer-2.3-py2.py3-none-any.whl
Collecting uri-template; extra == "format" (from jsonschema[format]->dandischema->-r tools/backups2datalad.req.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/c0/db/d4f9c75b43541f7235daf4d13eb43f4491f9d5f5df45ce41daeed3a903f6/uri_template-1.2.0-py3-none-any.whl
Requirement already satisfied: requests-toolbelt>=0.9.1 in /mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages (from python-gitlab->datalad>=0.17.0->-r tools/backups2datalad.req.txt (line 9)) (0.9.1)
Requirement already satisfied: zipp>=0.5 in /mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages (from importlib-metadata>=3.6; python_version < "3.10"->datalad>=0.17.0->-r tools/backups2datalad.req.txt (line 9)) (3.7.0)
Collecting h11<0.13,>=0.11 (from httpcore<0.15.0,>=0.14.5->httpx~=0.22.0->-r tools/backups2datalad.req.txt (line 11))
  Using cached https://files.pythonhosted.org/packages/60/0f/7a0eeea938eaf61074f29fed9717f2010e8d0e0905d36b38d3275a1e4622/h11-0.12.0-py3-none-any.whl
Requirement already satisfied: cryptography>=2.0 in /mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages (from SecretStorage>=3.2; sys_platform == "linux"->keyring!=23.9.0->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7)) (36.0.2)
Collecting pytz>=2020.1 (from pandas<2,>=1.1.5->pynwb!=1.1.0,>=1.0.3->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/85/ac/92f998fc52a70afd7f6b788142632afb27cd60c8c782d1452b7466603332/pytz-2022.6-py2.py3-none-any.whl
Collecting scipy<2,>=1.1 (from hdmf<4,>=3.4.2->pynwb!=1.1.0,>=1.0.3->dandi>=0.46.5->-r tools/backups2datalad.req.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/0a/2e/44795c6398e24e45fa0bb61c3e98de1cfea567b1b51efd3751e2f7ff9720/scipy-1.9.3.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /mnt/backup/dandi/tinuous-logs/venv/bin/python3.8 /mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp8i959pa6
       cwd: /tmp/pip-install-yyrxkzr3/scipy
  Complete output (68 lines):
  The Meson build system
  Version: 0.64.0
  Source dir: /tmp/pip-install-yyrxkzr3/scipy
  Build dir: /tmp/pip-install-yyrxkzr3/scipy/.mesonpy-hatjnbn1/build
  Build type: native build
  Project name: SciPy
  Project version: 1.9.3
  C compiler for the host machine: cc (gcc 10.2.1 "cc (Debian 10.2.1-6) 10.2.1 20210110")
  C linker for the host machine: cc ld.bfd 2.35.2
  C++ compiler for the host machine: c++ (gcc 10.2.1 "c++ (Debian 10.2.1-6) 10.2.1 20210110")
  C++ linker for the host machine: c++ ld.bfd 2.35.2
  Host machine cpu family: x86_64
  Host machine cpu: x86_64
  Compiler for C supports arguments -Wno-unused-but-set-variable: YES
  Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
  Compiler for C supports arguments -Wno-unused-function: YES
  Compiler for C supports arguments -Wno-conversion: YES
  Compiler for C supports arguments -Wno-misleading-indentation: YES
  Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
  Library m found: YES

  ../../meson.build:57:0: ERROR: Unknown compiler(s): [['gfortran'], ['flang'], ['nvfortran'], ['pgfortran'], ['ifort'], ['ifx'], ['g95']]
  The following exception(s) were encountered:
  Running `gfortran --version` gave "[Errno 2] No such file or directory: 'gfortran'"
  Running `gfortran -V` gave "[Errno 2] No such file or directory: 'gfortran'"
  Running `flang --version` gave "[Errno 2] No such file or directory: 'flang'"
  Running `flang -V` gave "[Errno 2] No such file or directory: 'flang'"
  Running `nvfortran --version` gave "[Errno 2] No such file or directory: 'nvfortran'"
  Running `nvfortran -V` gave "[Errno 2] No such file or directory: 'nvfortran'"
  Running `pgfortran --version` gave "[Errno 2] No such file or directory: 'pgfortran'"
  Running `pgfortran -V` gave "[Errno 2] No such file or directory: 'pgfortran'"
  Running `ifort --version` gave "[Errno 2] No such file or directory: 'ifort'"
  Running `ifort -V` gave "[Errno 2] No such file or directory: 'ifort'"
  Running `ifx --version` gave "[Errno 2] No such file or directory: 'ifx'"
  Running `ifx -V` gave "[Errno 2] No such file or directory: 'ifx'"
  Running `g95 --version` gave "[Errno 2] No such file or directory: 'g95'"
  Running `g95 -V` gave "[Errno 2] No such file or directory: 'g95'"

  A full log can be found at /tmp/pip-install-yyrxkzr3/scipy/.mesonpy-hatjnbn1/build/meson-logs/meson-log.txt
  + meson setup --native-file=/tmp/pip-install-yyrxkzr3/scipy/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=/home/dandi/miniconda3 /tmp/pip-install-yyrxkzr3/scipy /tmp/pip-install-yyrxkzr3/scipy/.mesonpy-hatjnbn1/build
  Traceback (most recent call last):
    File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-er5vfait/overlay/lib/python3.8/site-packages/mesonpy/__init__.py", line 969, in get_requires_for_build_wheel
      with _project(config_settings) as project:
    File "/home/dandi/miniconda3/lib/python3.8/contextlib.py", line 113, in __enter__
      return next(self.gen)
    File "/tmp/pip-build-env-er5vfait/overlay/lib/python3.8/site-packages/mesonpy/__init__.py", line 948, in _project
      with Project.with_temp_working_dir(
    File "/home/dandi/miniconda3/lib/python3.8/contextlib.py", line 113, in __enter__
      return next(self.gen)
    File "/tmp/pip-build-env-er5vfait/overlay/lib/python3.8/site-packages/mesonpy/__init__.py", line 777, in with_temp_working_dir
      yield cls(source_dir, tmpdir, build_dir)
    File "/tmp/pip-build-env-er5vfait/overlay/lib/python3.8/site-packages/mesonpy/__init__.py", line 682, in __init__
      self._configure(reconfigure=bool(build_dir) and not native_file_mismatch)
    File "/tmp/pip-build-env-er5vfait/overlay/lib/python3.8/site-packages/mesonpy/__init__.py", line 713, in _configure
      self._meson(
    File "/tmp/pip-build-env-er5vfait/overlay/lib/python3.8/site-packages/mesonpy/__init__.py", line 696, in _meson
      return self._proc('meson', *args)
    File "/tmp/pip-build-env-er5vfait/overlay/lib/python3.8/site-packages/mesonpy/__init__.py", line 691, in _proc
      subprocess.check_call(list(args))
    File "/home/dandi/miniconda3/lib/python3.8/subprocess.py", line 364, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['meson', 'setup', '--native-file=/tmp/pip-install-yyrxkzr3/scipy/.mesonpy-native-file.ini', '-Ddebug=false', '-Doptimization=2', '--prefix=/home/dandi/miniconda3', '/tmp/pip-install-yyrxkzr3/scipy', '/tmp/pip-install-yyrxkzr3/scipy/.mesonpy-hatjnbn1/build']' returned non-zero exit status 1.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /mnt/backup/dandi/tinuous-logs/venv/bin/python3.8 /mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp8i959pa6 Check the logs for full command output.
WARNING: You are using pip version 19.2.3, however version 22.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

@jwoder -- please research and mitigate

jwodder commented 1 year ago

@yarikoptic Try upgrading pip. All the Linux wheels for numpy 1.23.5 seem to be manylinux 2, and it's possible pip 19 doesn't support that.

yarikoptic commented 1 year ago

it is all in the /mnt/backup/dandi/dandisets/tools/backups2datalad-update-cron -- please fix it the way you see it might work.

jwodder commented 1 year ago

@yarikoptic Did you upgrade pip? Because the pip in the dandisets environment is now the latest version, and running pip install -r backups2datalad.req.txt there successfully installs numpy.

yarikoptic commented 1 year ago

nope. I also got confused because I did run

dandisets) dandi@drogon:/mnt/backup/dandi/dandisets$ pip install --upgrade numpy
Requirement already satisfied: numpy in /home/dandi/miniconda3/envs/dandisets/lib/python3.8/site-packages (1.22.4)
Collecting numpy
  Downloading numpy-1.23.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.1/17.1 MB 10.7 MB/s eta 0:00:00
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.22.4
    Uninstalling numpy-1.22.4:
      Successfully uninstalled numpy-1.22.4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
nwbinspector 0.4.17 requires numpy<1.23.0,>=1.22.0; python_version >= "3.8", but you have numpy 1.23.5 which is incompatible.
Successfully installed numpy-1.23.5

so that one worked. But I didn't upgrade pip. So may be issue was on numpy end that they didn't have appropriate .whl or smth like that. ok, let's hope it doesn't come back

dandi / dandisets

numpy is not installing #304