search

ijl / orjson

Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy
Apache License 2.0
6.3k stars 218 forks source link

Orjson 3.10.7 degradation, while CI/CD #508

Closed D1-3105 closed 3 months ago

D1-3105 commented 3 months ago

I've just faced an error, while installing orjson with poetry. Here is the log:

 #8 24.48   RuntimeError
#8 24.48 
#8 24.48   Unable to find installation candidates for orjson (3.10.7)
#8 24.48 
#8 24.48   at /usr/local/lib/python3.12/site-packages/poetry/installation/chooser.py:74 in choose_for
#8 24.49        70│ 
#8 24.49        71│             links.append(link)
#8 24.49        72│ 
#8 24.49        73│         if not links:
#8 24.49     →  74│             raise RuntimeError(f"Unable to find installation candidates for {package}")
#8 24.49        75│ 
#8 24.49        76│         # Get the best link
#8 24.49        77│         chosen = max(links, key=lambda link: self._sort_key(package, link))
#8 24.49        78│ 
#8 24.49 
#8 24.49 Cannot install orjson.

It happened, while your "Actions" were in progress.

UPD: just discovered another error:

$ pip install orjson==3.10.7

  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [24 lines of output]
      Checking for Rust toolchain....
      Traceback (most recent call last):
        File "/home/oleg/PycharmProjects/vli_autoscaler/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/oleg/PycharmProjects/vli_autoscaler/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/oleg/PycharmProjects/vli_autoscaler/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-56_ghpl0/overlay/lib/python3.12/site-packages/maturin/__init__.py", line 171, in prepare_metadata_for_build_wheel
          output = subprocess.check_output(["cargo", "--version"]).decode("utf-8", "ignore")
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.12/subprocess.py", line 466, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.12/subprocess.py", line 548, in run
          with Popen(*popenargs, **kwargs) as process:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.12/subprocess.py", line 1026, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/usr/lib/python3.12/subprocess.py", line 1955, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      NotADirectoryError: [Errno 20] Not a directory: 'cargo'
      [end of output]
jmoralez commented 3 months ago

Seems like there are a couple of wheels missing on the latest release (3.10.7), in particular linux x86 cpython >= 3.9, I can only see cp38 (files).

stas00 commented 3 months ago

yes, I was just about to report the same! Our CI is on fire

we need:

Thank you!

D1-3105 commented 3 months ago

Oh, that's a nice point. We need:

smallsam commented 3 months ago

Yes missing wheel for us too. Just to close the loop, we're after 3.11!

Our symptom is

#9 13.65   Installing build dependencies: started
#9 15.41   Installing build dependencies: finished with status 'done'
#9 15.41   Getting requirements to build wheel: started
#9 15.58   Getting requirements to build wheel: finished with status 'done'
#9 15.58   Preparing metadata (pyproject.toml): started
#9 15.74   Preparing metadata (pyproject.toml): finished with status 'error'
#9 15.74   error: subprocess-exited-with-error
#9 15.74   
#9 15.74   × Preparing metadata (pyproject.toml) did not run successfully.
#9 15.74   │ exit code: 1
#9 15.74   ╰─> [6 lines of output]
#9 15.74       Checking for Rust toolchain....
#9 15.74       
#9 15.74       Cargo, the Rust package manager, is not installed or is not on PATH.
#9 15.74       This package requires Rust and Cargo to compile extensions. Install it through
#9 15.74       the system's package manager or via https://rustup.rs/
#9 15.74       
#9 15.74       [end of output]
#9 15.74   
#9 15.74   note: This error originates from a subprocess, and is likely not a problem with pip.
#9 15.74 error: metadata-generation-failed
#9 15.74 
#9 15.74 × Encountered error while generating package metadata.
#9 15.74 ╰─> See above for output.

We're going to pin back to 3.10.6 for now to get back up and running.

stas00 commented 3 months ago

tagging @ijl - this is quite urgent unless we drop this module.

stas00 commented 3 months ago

oh, wait, where did you find old wheels, @smallsam - I see only the latest ones on pypi https://pypi.org/project/orjson/#files

unless it doesn't show older versions? let me try the old one then

stas00 commented 3 months ago

You're correct the old wheels are there - pypi just doesn't show them - I was able to recover with orjson==3.10.6 - I appreciate your suggestion, @smallsam!

philipperemy commented 3 months ago

Same here. You need rust installed from 3.10.7....

smallsam commented 3 months ago

oh, wait, where did you find old wheels, @smallsam - I see only the latest ones on pypi https://pypi.org/project/orjson/#files

unless it doesn't show older versions? let me try the old one then

https://pypi.org/project/orjson/3.10.6/#files

If you need them directly.

stas00 commented 3 months ago

Same here. You need rust installed from 3.10.7....

it's because wheels are missing so it's trying to build from source which requires rust.

use https://github.com/ijl/orjson/issues/508#issuecomment-2276979404 solution

philipperemy commented 3 months ago

@stas00 I don't think wheels are missing for the latest release, they are there: https://pypi.org/project/orjson/3.10.7/#files But seems like Rust is now required when pip installing. Probably to have a faster "custom" build.

stas00 commented 3 months ago

Nope, the wheel we need is missing, we need:

It's there in 3.10.6, but not 3.10.7

orjson-3.10.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141.1 kB view hashes)

Uploaded Jul 2, 2024 CPython 3.10 manylinux: glibc 2.17+ x86-64

stas00 commented 3 months ago

And I found how to navigate to the older version on pypi - via "Release history" - I think I mixed it up with anaconda where it's all together in one dump.

stas00 commented 3 months ago

musllinux != manylinux https://peps.python.org/pep-0656/

hughcontek commented 3 months ago

Same here. CI is on fire

Kamillaova commented 3 months ago

Same here. CI is on fire

lock orjson to 3.10.6

like this: "orjson==3.10.6", # orjson==3.10.7 has no amd64 wheels???

philipperemy commented 3 months ago

Ok my bad I didn't spend much time looking at the exact wheels @stas00

stas00 commented 3 months ago

no damage done, Philippe! Thank you for checking.

ijl commented 3 months ago

amd64 wheels for glibc and 3.9, 3.10, 3.11, 3.12, and 3.13 were published.