twitter / communitynotes

Documentation and source code powering Twitter's Community Notes
https://twitter.github.io/communitynotes
Apache License 2.0
1.42k stars 196 forks source link

Installing via pip requirements hits a Cython error #184

Closed jbaxter closed 9 months ago

jbaxter commented 9 months ago
$ python -m venv communitynotes_env
$ source communitynotes_env/bin/activate
(communitynotes_env) $ pip install -r requirements.txt
Collecting numpy==1.19.2 (from -r requirements.txt (line 1))
  Downloading numpy-1.19.2.zip (7.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/7.3 MB 11.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [57 lines of output]
      Running from numpy source directory.
      setup.py:470: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
        run_build = parse_setuppy_commands()

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
                  self.rng_state.ctr.v[i] = counter[i]

              self._reset_state_variables()

              self._bitgen.state = <void *>&self.rng_state
              self._bitgen.next_uint64 = &philox_uint64
                                         ^
      ------------------------------------------------------------

      _philox.pyx:195:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to type 'uint64_t (void *) except? -1 nogil'.
      Processing numpy/random/_bounded_integers.pxd.in
      Processing numpy/random/_philox.pyx
      Traceback (most recent call last):
        File "/private/var/folders/nz/x3d3wwcx1tl_g6zcxvd2f58c0000gn/T/pip-install-vasbel4p/numpy_2967227e8dcb4dd88990a0cd56277393/tools/cythonize.py", line 235, in <module>
          main()
        File "/private/var/folders/nz/x3d3wwcx1tl_g6zcxvd2f58c0000gn/T/pip-install-vasbel4p/numpy_2967227e8dcb4dd88990a0cd56277393/tools/cythonize.py", line 231, in main
          find_process_files(root_dir)
        File "/private/var/folders/nz/x3d3wwcx1tl_g6zcxvd2f58c0000gn/T/pip-install-vasbel4p/numpy_2967227e8dcb4dd88990a0cd56277393/tools/cythonize.py", line 222, in find_process_files
          process(root_dir, fromfile, tofile, function, hash_db)
        File "/private/var/folders/nz/x3d3wwcx1tl_g6zcxvd2f58c0000gn/T/pip-install-vasbel4p/numpy_2967227e8dcb4dd88990a0cd56277393/tools/cythonize.py", line 188, in process
          processor_function(fromfile, tofile)
        File "/private/var/folders/nz/x3d3wwcx1tl_g6zcxvd2f58c0000gn/T/pip-install-vasbel4p/numpy_2967227e8dcb4dd88990a0cd56277393/tools/cythonize.py", line 77, in process_pyx
          subprocess.check_call(
        File "/Users/jbaxter/anaconda3/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/jbaxter/workspace/communitynotes/communitynotes_env/bin/python', '-m', 'cython', '-3', '--fast-fail', '-o', '_philox.c', '_philox.pyx']' returned non-zero exit status 1.
      Cythonizing sources
      Traceback (most recent call last):
        File "/Users/jbaxter/workspace/communitynotes/communitynotes_env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/jbaxter/workspace/communitynotes/communitynotes_env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/jbaxter/workspace/communitynotes/communitynotes_env/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/nz/x3d3wwcx1tl_g6zcxvd2f58c0000gn/T/pip-build-env-lfzl2oxr/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/private/var/folders/nz/x3d3wwcx1tl_g6zcxvd2f58c0000gn/T/pip-build-env-lfzl2oxr/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 249, in run_setup
          self).run_setup(setup_script=setup_script)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/nz/x3d3wwcx1tl_g6zcxvd2f58c0000gn/T/pip-build-env-lfzl2oxr/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 142, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 499, in <module>
          setup_package()
        File "setup.py", line 479, in setup_package
          generate_cython()
        File "setup.py", line 274, in generate_cython
          raise RuntimeError("Running cythonize failed!")
      RuntimeError: Running cythonize failed!
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[notice] A new release of pip is available: 23.1.2 -> 23.3.2
[notice] To update, run: pip install --upgrade pip
jbaxter commented 9 months ago

Should be fixed now! We have updated requirements.txt to use newer versions of packages that are compatible with Cython 3, and fixed areas in the rest of our code that were using deprecated functionality for packages we needed to bump.

cc @colinmegill