search

snorkel-team / snorkel

A system for quickly generating training data with weak supervision
https://snorkel.org
Apache License 2.0
5.81k stars 857 forks source link

Snorkel fails to install for Python 3.8 due to scikit-learn version #1627

Closed rjurney closed 3 years ago

rjurney commented 3 years ago

Issue description

When you install Snorkel, it tries to install scikit-learn - which does not build. Thus Snorkel does not build. Can we please update the version of scikit-learn so people can build and use Snorkel?

Code example/repro steps

pip install snorkel

It tries to install scikit-learn<0.22.0,>=0.20.2 and fails :(

Building wheels for collected packages: scikit-learn
  Building wheel for scikit-learn (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/rjurney/opt/anaconda3/envs/deep_discovery/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py'"'"'; __file__='"'"'/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-wheel-1u1ts_5o
       cwd: /private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/
  Complete output (28 lines):
  Partial import of sklearn during the build process.
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py", line 290, in <module>
      setup_package()
    File "/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py", line 286, in setup_package
      setup(**metadata)
    File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/core.py", line 135, in setup
      config = configuration()
    File "/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py", line 174, in configuration
      config.add_subpackage('sklearn')
    File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
      config_list = self.get_subpackage(subpackage_name, subpackage_path,
    File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
      config = self._get_configuration_from_setup_py(
    File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
      config = setup_module.configuration(*args)
    File "sklearn/setup.py", line 62, in configuration
      config.add_subpackage('utils')
    File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
      config_list = self.get_subpackage(subpackage_name, subpackage_path,
    File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
      config = self._get_configuration_from_setup_py(
    File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
      config = setup_module.configuration(*args)
    File "sklearn/utils/setup.py", line 8, in configuration
      from Cython import Tempita
  ModuleNotFoundError: No module named 'Cython'
  ----------------------------------------
  ERROR: Failed building wheel for scikit-learn
  Running setup.py clean for scikit-learn
Failed to build scikit-learn
Installing collected packages: grpcio, absl-py, tensorboard, scikit-learn, networkx, munkres, snorkel
  Attempting uninstall: scikit-learn
    Found existing installation: scikit-learn 0.24.1
    Uninstalling scikit-learn-0.24.1:
      Successfully uninstalled scikit-learn-0.24.1
    Running setup.py install for scikit-learn ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/rjurney/opt/anaconda3/envs/deep_discovery/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py'"'"'; __file__='"'"'/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-record-kias8bzm/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjurney/opt/anaconda3/envs/deep_discovery/include/python3.8/scikit-learn
         cwd: /private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/
    Complete output (28 lines):
    Partial import of sklearn during the build process.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py", line 290, in <module>
        setup_package()
      File "/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py", line 286, in setup_package
        setup(**metadata)
      File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/core.py", line 135, in setup
        config = configuration()
      File "/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py", line 174, in configuration
        config.add_subpackage('sklearn')
      File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
        config_list = self.get_subpackage(subpackage_name, subpackage_path,
      File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
        config = self._get_configuration_from_setup_py(
      File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "sklearn/setup.py", line 62, in configuration
        config.add_subpackage('utils')
      File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
        config_list = self.get_subpackage(subpackage_name, subpackage_path,
      File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
        config = self._get_configuration_from_setup_py(
      File "/Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "sklearn/utils/setup.py", line 8, in configuration
        from Cython import Tempita
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------
  Rolling back uninstall of scikit-learn
  Moving to /Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/scikit_learn-0.24.1.dist-info/
   from /Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/~cikit_learn-0.24.1.dist-info
  Moving to /Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/sklearn/
   from /Users/rjurney/opt/anaconda3/envs/deep_discovery/lib/python3.8/site-packages/~klearn
ERROR: Command errored out with exit status 1: /Users/rjurney/opt/anaconda3/envs/deep_discovery/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py'"'"'; __file__='"'"'/private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-install-oo_m8xg5/scikit-learn_d710130497834af2bd9b9f9514867b07/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/3h/vnn5rkyx3nx7yyrfyynrhrlw0000gn/T/pip-record-kias8bzm/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rjurney/opt/anaconda3/envs/deep_discovery/include/python3.8/scikit-learn Check the logs for full command output.

Expected Behavior

I expect the install to succeed. It fails.

System Info

Additional context

Add any other context about the problem here.

bhancock8 commented 3 years ago

Hi @rjurney, we're taking a look. I haven't been able to repro your issue yet. As a minimal example, on Mac OS X 11.1 with Python 3.7.3, the following commands succeed for me:

python -m venv env
source env/bin/activate
pip install snorkel
python [enter python shell]
import snorkel
bhancock8 commented 3 years ago

It looks like Python 3.7 works fine, Python 3.8 hits the Cython import issue.

ScottIsWorking commented 3 years ago

scikit-learn versioning is definitely hampering my ability to use snorkel either locally with pipenv and either python 3.7 or 3.8, or remotely google colab. I'll explain:

Locally

mkdir snorkel_test; cd snorkel_test
pyenv local 3.7.7
pipenv install snorkel

output:

Creating a Pipfile for this project…
Error:  An error occurred while installing snorkel!
Error text: Collecting snorkel
  Using cached snorkel-0.9.6-py3-none-any.whl (144 kB)
Requirement already satisfied, skipping upgrade: pandas<2.0.0,>=0.25.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.2.2)
Requirement already satisfied, skipping upgrade: munkres>=1.0.6 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.1.4)
Requirement already satisfied, skipping upgrade: networkx<2.4,>=2.2 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (2.3)
Requirement already satisfied, skipping upgrade: tqdm<5.0.0,>=4.33.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (4.56.2)
Collecting scikit-learn<0.22.0,>=0.20.2
  Using cached scikit-learn-0.21.3.tar.gz (12.2 MB)
Requirement already satisfied, skipping upgrade: torch<2.0.0,>=1.2.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.7.1)
Requirement already satisfied, skipping upgrade: numpy<2.0.0,>=1.16.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.20.1)
Requirement already satisfied, skipping upgrade: scipy<2.0.0,>=1.2.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.6.0)
Requirement already satisfied, skipping upgrade: tensorboard<2.0.0,>=1.14.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.15.0)
Requirement already satisfied, skipping upgrade: pytz>=2017.3 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from pandas<2.0.0,>=0.25.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (2021.1)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.7.3 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from pandas<2.0.0,>=0.25.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (2.8.1)
Requirement already satisfied, skipping upgrade: decorator>=4.3.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from networkx<2.4,>=2.2->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (4.4.2)
Requirement already satisfied, skipping upgrade: joblib>=0.11 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from scikit-learn<0.22.0,>=0.20.2->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.0.1)
Requirement already satisfied, skipping upgrade: typing-extensions in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from torch<2.0.0,>=1.2.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (3.7.4.3)
Requirement already satisfied, skipping upgrade: setuptools>=41.0.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (51.1.2)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.15.0)
Requirement already satisfied, skipping upgrade: wheel>=0.26; python_version >= "3" in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (0.36.2)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (3.14.0)
Requirement already satisfied, skipping upgrade: werkzeug>=0.11.15 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.0.1)
Requirement already satisfied, skipping upgrade: absl-py>=0.4 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (0.11.0)
Requirement already satisfied, skipping upgrade: markdown>=2.6.8 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (3.3.3)
Requirement already satisfied, skipping upgrade: grpcio>=1.6.3 in /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages (from tensorboard<2.0.0,>=1.14.0->snorkel->-r /var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pipenv-mn50gqr6-requirements/pipenv-f03uyyos-requirement.txt (line 1)) (1.35.0)
Building wheels for collected packages: scikit-learn
  Building wheel for scikit-learn (setup.py): started
  Building wheel for scikit-learn (setup.py): finished with status 'error'
  Running setup.py clean for scikit-learn
Failed to build scikit-learn
Installing collected packages: scikit-learn, snorkel
    Running setup.py install for scikit-learn: started
    Running setup.py install for scikit-learn: finished with status 'error'

  ERROR: Command errored out with exit status 1:
   command: /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py'"'"'; __file__='"'"'/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-wheel-hhb3lksb
       cwd: /private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/
  Complete output (28 lines):
  Partial import of sklearn during the build process.
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py", line 290, in <module>
      setup_package()
    File "/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py", line 286, in setup_package
      setup(**metadata)
    File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/core.py", line 135, in setup
      config = configuration()
    File "/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py", line 174, in configuration
      config.add_subpackage('sklearn')
    File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
      config_list = self.get_subpackage(subpackage_name, subpackage_path,
    File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
      config = self._get_configuration_from_setup_py(
    File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
      config = setup_module.configuration(*args)
    File "sklearn/setup.py", line 62, in configuration
      config.add_subpackage('utils')
    File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
      config_list = self.get_subpackage(subpackage_name, subpackage_path,
    File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
      config = self._get_configuration_from_setup_py(
    File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
      config = setup_module.configuration(*args)
    File "sklearn/utils/setup.py", line 8, in configuration
      from Cython import Tempita
  ModuleNotFoundError: No module named 'Cython'
  ----------------------------------------
  ERROR: Failed building wheel for scikit-learn
    ERROR: Command errored out with exit status 1:
     command: /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py'"'"'; __file__='"'"'/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-record-vxkh22ce/install-record.txt --single-version-externally-managed --compile --install-headers /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/include/site/python3.8/scikit-learn
         cwd: /private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/
    Complete output (28 lines):
    Partial import of sklearn during the build process.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py", line 290, in <module>
        setup_package()
      File "/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py", line 286, in setup_package
        setup(**metadata)
      File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/core.py", line 135, in setup
        config = configuration()
      File "/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py", line 174, in configuration
        config.add_subpackage('sklearn')
      File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
        config_list = self.get_subpackage(subpackage_name, subpackage_path,
      File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
        config = self._get_configuration_from_setup_py(
      File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "sklearn/setup.py", line 62, in configuration
        config.add_subpackage('utils')
      File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 1019, in add_subpackage
        config_list = self.get_subpackage(subpackage_name, subpackage_path,
      File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 985, in get_subpackage
        config = self._get_configuration_from_setup_py(
      File "/Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/lib/python3.8/site-packages/numpy/distutils/misc_util.py", line 927, in _get_configuration_from_setup_py
        config = setup_module.configuration(*args)
      File "sklearn/utils/setup.py", line 8, in configuration
        from Cython import Tempita
    ModuleNotFoundError: No module named 'Cython'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py'"'"'; __file__='"'"'/private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-install-9qhcv9yj/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/g6/z_v9cfkj2w74ns3cfz8dn2b40000gn/T/pip-record-vxkh22ce/install-record.txt --single-version-externally-managed --compile --install-headers /Users/cambo/.local/share/virtualenvs/snorkel_test-mdj0W--R/include/site/python3.8/scikit-learn Check the logs for full command output.

I can confirm that the virtualenv approach laid out by @bhancock8 above works for 3.7 and not 3.8, but it seems using pipenv does not work in either case.

Google Colab

Google Colab seems to run Python 3.6.9. Here the issue seems to be that snorkel wants to install UMAP which requires scikit-learn>=0.22 while colab installs 0.21.3 by default. At the same time, snorkel 0.9.6 requires scikit-learn<0.22.0.

Here's a Colab notebook that runs through this problem

rth commented 3 years ago

Yes, this is due to the fact that the scikit-learn dependency is outdated and does not have pre-built wheels for Python 3.8. The solution is to relax the dependency to 0.24. Related https://github.com/snorkel-team/snorkel/issues/1613

Should be resolved in https://github.com/snorkel-team/snorkel/pull/1628

yongzx commented 3 years ago

When I use pip3 install snorkel I still run into the same error "ModuleNotFoundError: No module named 'Cython'"

I have to use the command pip3 install git+https://github.com/snorkel-team/snorkel.git#egg=snorkel to install snorkel.

rjurney commented 3 years ago

@yongzx this is fixed by bumping the version of scikit-learn to scikit-learn>=0.24.0 which you can do in a fork. There is a PR right now for this that was just merged into master branch, so try using master or at least monkey patching that commit into v0.9.6: https://github.com/snorkel-team/snorkel/pull/1628

You may need to create a fresh conda, pipenv or virtualenv and then do the install as old components screw things up. I had your issue, bumped the version myself, still had it, nuked my conda env, rebuilt it from scratch with the new snorkel and it works.