Unable to install with `conda env create -f dicodile_env.yml`

[chmendoza]

Hi!

Thanks for sharing your code! I am trying to install your package in an isolated conda environment, using

conda env create -f dicodile_env.yml

However, I am getting this error

Ran pip subprocess with arguments:
['/home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/cmendoza/dicodile/condaenv.daadydz8.requirements.txt']
Pip subprocess output:
Collecting git+https://github.com/dask/dask-jobqueue.git (from -r /home/cmendoza/dicodile/condaenv.daadydz8.requirements.txt (line 1))
  Cloning https://github.com/dask/dask-jobqueue.git to /tmp/pip-req-build-e5f3dopd
  Resolved https://github.com/dask/dask-jobqueue.git to commit bad0e7f6ce578397e06a2fcfe5fb7f2e405fb358
Obtaining file:///home/cmendoza/dicodile (from -r /home/cmendoza/dicodile/condaenv.daadydz8.requirements.txt (line 3))
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'

Pip subprocess error:
  Running command git clone -q https://github.com/dask/dask-jobqueue.git /tmp/pip-req-build-e5f3dopd
  ERROR: Command errored out with exit status 1:
   command: /home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/bin/python /home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmphof_g2w8
       cwd: /home/cmendoza/dicodile
  Complete output (18 lines): 
  Traceback (most recent call last):
    File "/home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-mzif3rqf/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 155, in get_requires_for_build_wheel
      config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-mzif3rqf/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-mzif3rqf/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 6, in <module>
      import toml  # noqa: F401
  ModuleNotFoundError: No module named 'toml'
  ----------------------------------------
WARNING: Discarding file:///home/cmendoza/dicodile. Command errored out with exit status 1: /home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/bin/python /home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmphof_g2w8 Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/bin/python /home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmphof_g2w8 Check the logs for full command output.

CondaEnvException: Pip failed

If I print the system path variable just before importing setuptools in setup.py, I get

 ['/home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6/site-packages/pip/_vendor/pep517/in_process', '/tmp/pip-build-env-mzif3rqf/site', '/home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python36.zip', '/home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6', '/home/cmendoza/local/bin/anaconda/anaconda3/envs/dicodile/lib/python3.6/lib-dynload', '/tmp/pip-build-env-mzif3rqf/overlay/lib/python3.6/site-packages', '/tmp/pip-build-env-mzif3rqf/normal/lib/python3.6/site-packages']

So, conda creates the environment and install most of the packages, but the installation of the dicodile package using pip is failing. I tried to look for some hints on the Internet, but couldn't find anything. Could you please provide some hints to solve this issue?

My system: Ubuntu 18.04. conda 4.8.2

[tomMoral]

Hello, thanks for the report. This error seems strange as it seems that what is failing when trying to install dask-jobqueue. However, dask-jobqueue is not used in the project, so we shoudl simply remove it from the env. (I think this is a leftover thing from when I tried to make a non-MPI backend). Could you try removing dask-jobqueue from the dependencies and retry building the env?

[tomMoral]

It seems that there is something more profound than that as our CI is broken now... https://github.com/tomMoral/dicodile/pull/44/checks?check_run_id=3828456175

I will try to check why the install is not working. Most probably, we need to remove the toml import which does not seem to be a deps of setuptools_scm anymore?

[chmendoza]

Hi Thomas,

Thanks for getting back to me. I removed the line

- git+https://github.com/dask/dask-jobqueue.git

from dicodile_env.yml, but it keeps throwing an error related to the import toml line in setup.py:

Installing pip dependencies: - Ran pip subprocess with arguments:
['/home/cmendoza/anaconda3/envs/dicodile/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/cmendoza/dicodile/condaenv.va6vdibt.requirements.txt']
Pip subprocess output:
Obtaining file:///home/cmendoza/dicodile (from -r /home/cmendoza/dicodile/condaenv.va6vdibt.requirements.txt (line 2))
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'

Pip subprocess error:
  ERROR: Command errored out with exit status 1:
   command: /home/cmendoza/anaconda3/envs/dicodile/bin/python /home/cmendoza/anaconda3/envs/dicodile/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpcjatgml0
       cwd: /home/cmendoza/dicodile
  Complete output (17 lines):
  Traceback (most recent call last):
    File "/home/cmendoza/anaconda3/envs/dicodile/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/home/cmendoza/anaconda3/envs/dicodile/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/cmendoza/anaconda3/envs/dicodile/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-sqtpssok/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 154, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-sqtpssok/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-sqtpssok/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 5, in <module>
      import toml  # noqa: F401
  ModuleNotFoundError: No module named 'toml'
  ----------------------------------------
WARNING: Discarding file:///home/cmendoza/dicodile. Command errored out with exit status 1: /home/cmendoza/anaconda3/envs/dicodile/bin/python /home/cmendoza/anaconda3/envs/dicodile/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpcjatgml0 Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/cmendoza/anaconda3/envs/dicodile/bin/python /home/cmendoza/anaconda3/envs/dicodile/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpcjatgml0 Check the logs for full command output.

failed

CondaEnvException: Pip failed

I added a line in setup.py to print the system path variable:

['/home/cmendoza/anaconda3/envs/dicodile/lib/python3.8/site-packages/pip/_vendor/pep517/in_process', '/tmp/pip-build-env-sqtpssok/site', '/home/cmendoza/anaconda3/envs/dicodile/lib/python38.zip', '/home/cmendoza/anaconda3/envs/dicodile/lib/python3.8', '/home/cmendoza/anaconda3/envs/dicodile/lib/python3.8/lib-dynload', '/tmp/pip-build-env-sqtpssok/overlay/lib/python3.8/site-packages', '/tmp/pip-build-env-sqtpssok/normal/lib/python3.8/site-packages']

I don't have experience packaging on Python, but you have this line in pyproject.toml:

requires = ["setuptools>=42", "wheel", "setuptools_scm[toml]>=3.4"]

so, maybe toml is being used by setuptools_scm...

[chmendoza]

After a "setuptools_scm toml import error" search, I found this: https://github.com/pypa/setuptools_scm/issues/608

I replaced toml by tomli in the two instances where it appears, and voilá, I was able to build dicodile! No errors, no warnings.

More specifically, I changed this line pyproject.toml

requires = ["setuptools>=42", "wheel", "setuptools_scm[toml]>=3.4"]

by this line

requires = ["setuptools>=42", "wheel", "setuptools_scm[tomli]>=3.4"]

and the impor toml line in setup.py by import tomli.

My system: Ubuntu 18.04 conda 4.9.2

After skimming the docs of setuptools_scm, I am guessing that you might not need to specify tomli as requirement to setuptools_scm, and that as you said import tomli is not even necessary. I could test that later. Do you want me to send a PR with those changes?

[chmendoza]

Ok, I just double checked, you can just get rid of those two occurrences of toml in your package. It will build without them.

[tomMoral]

Thanks a lot for getting back to us! Apparently, setuptools_scm changed a bit and we did not detect it as the CI is not running dayly. :) We removed the import on toml altogether as it is not needed anymore.

I just merged #45 that should resolve this on master.

[tomMoral]

I closed the issue, let us know if you have further problems!