search

Apicurio / apicurio-registry

An API/Schema registry - stores APIs and Schemas.
https://www.apicur.io/registry/
Apache License 2.0
609 stars 269 forks source link

Published Python SDK doesn't include the generated code #3596

Closed forsberg closed 1 year ago

forsberg commented 1 year ago

Description

Registry Version: 2.4.4

Environment

Linux, Python 3.10

Steps to Reproduce

  1. Create a new virtualenv
  2. Try to pip install apicurioregistrysdk`
  3. Fail.
mkvirtualenv apicurio-registry-sdk
created virtual environment CPython3.10.12.final.0-64 in 148ms
  creator CPython3Posix(dest=/home/forsberg/.virtualenvs/apicurio-registry-sdk, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/forsberg/.local/share/virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.0.0, wheel==0.41.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
virtualenvwrapper.user_scripts creating /home/forsberg/.virtualenvs/apicurio-registry-sdk/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/forsberg/.virtualenvs/apicurio-registry-sdk/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/forsberg/.virtualenvs/apicurio-registry-sdk/bin/preactivate
virtualenvwrapper.user_scripts creating /home/forsberg/.virtualenvs/apicurio-registry-sdk/bin/postactivate
virtualenvwrapper.user_scripts creating /home/forsberg/.virtualenvs/apicurio-registry-sdk/bin/get_env_details
(apicurio-registry-sdk) $ pip install apicurioregistrysdk
Looking in indexes: https://pypi.org/simple
Collecting apicurioregistrysdk
  Using cached apicurioregistrysdk-2.4.4.tar.gz (2.4 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting microsoft-kiota-abstractions<0.7.0,>=0.6.0 (from apicurioregistrysdk)
  Obtaining dependency information for microsoft-kiota-abstractions<0.7.0,>=0.6.0 from https://files.pythonhosted.org/packages/d6/5a/8d810173b81051e978f82a2fb0d6821d0743088ce85f6371b0df0b396f38/microsoft_kiota_abstractions-0.6.0-py2.py3-none-any.whl.metadata
  Downloading microsoft_kiota_abstractions-0.6.0-py2.py3-none-any.whl.metadata (3.4 kB)
Collecting microsoft-kiota-http<0.5.0,>=0.4.4 (from apicurioregistrysdk)
  Obtaining dependency information for microsoft-kiota-http<0.5.0,>=0.4.4 from https://files.pythonhosted.org/packages/ee/6c/f7c2d04dde314da76addf71079d0da942a91dadce416a342508286e9523f/microsoft_kiota_http-0.4.4-py2.py3-none-any.whl.metadata
  Downloading microsoft_kiota_http-0.4.4-py2.py3-none-any.whl.metadata (3.3 kB)
Collecting microsoft-kiota-serialization-json<0.4.0,>=0.3.7 (from apicurioregistrysdk)
  Obtaining dependency information for microsoft-kiota-serialization-json<0.4.0,>=0.3.7 from https://files.pythonhosted.org/packages/c3/79/0d2a19e4936b7719d9cf5cc97e22d119fb515f2eb346c905696337a4f030/microsoft_kiota_serialization_json-0.3.7-py2.py3-none-any.whl.metadata
  Downloading microsoft_kiota_serialization_json-0.3.7-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting microsoft-kiota-serialization-text<0.3.0,>=0.2.1 (from apicurioregistrysdk)
  Obtaining dependency information for microsoft-kiota-serialization-text<0.3.0,>=0.2.1 from https://files.pythonhosted.org/packages/b4/41/ffa16e9a9b61182f1badc7fee861fbe2ddeb02b05743e6db377b88d4fd40/microsoft_kiota_serialization_text-0.2.1-py2.py3-none-any.whl.metadata
  Downloading microsoft_kiota_serialization_text-0.2.1-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting uritemplate>=4.1.1 (from microsoft-kiota-abstractions<0.7.0,>=0.6.0->apicurioregistrysdk)
  Using cached uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
Collecting httpx[http2]>=0.23.0 (from microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Obtaining dependency information for httpx[http2]>=0.23.0 from https://files.pythonhosted.org/packages/ec/91/e41f64f03d2a13aee7e8c819d82ee3aa7cdc484d18c0ae859742597d5aa0/httpx-0.24.1-py3-none-any.whl.metadata
  Using cached httpx-0.24.1-py3-none-any.whl.metadata (7.4 kB)
Collecting python-dateutil>=2.8.2 (from microsoft-kiota-serialization-json<0.4.0,>=0.3.7->apicurioregistrysdk)
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting certifi (from httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Obtaining dependency information for certifi from https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl.metadata
  Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore<0.18.0,>=0.15.0 (from httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Obtaining dependency information for httpcore<0.18.0,>=0.15.0 from https://files.pythonhosted.org/packages/94/2c/2bde7ff8dd2064395555220cbf7cba79991172bf5315a07eb3ac7688d9f1/httpcore-0.17.3-py3-none-any.whl.metadata
  Using cached httpcore-0.17.3-py3-none-any.whl.metadata (18 kB)
Collecting idna (from httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting sniffio (from httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting h2<5,>=3 (from httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Using cached h2-4.1.0-py3-none-any.whl (57 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->microsoft-kiota-serialization-json<0.4.0,>=0.3.7->apicurioregistrysdk)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting hyperframe<7,>=6.0 (from h2<5,>=3->httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Using cached hyperframe-6.0.1-py3-none-any.whl (12 kB)
Collecting hpack<5,>=4.0 (from h2<5,>=3->httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Using cached hpack-4.0.0-py3-none-any.whl (32 kB)
Collecting h11<0.15,>=0.13 (from httpcore<0.18.0,>=0.15.0->httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting anyio<5.0,>=3.0 (from httpcore<0.18.0,>=0.15.0->httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Obtaining dependency information for anyio<5.0,>=3.0 from https://files.pythonhosted.org/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl.metadata
  Using cached anyio-3.7.1-py3-none-any.whl.metadata (4.7 kB)
Collecting exceptiongroup (from anyio<5.0,>=3.0->httpcore<0.18.0,>=0.15.0->httpx[http2]>=0.23.0->microsoft-kiota-http<0.5.0,>=0.4.4->apicurioregistrysdk)
  Obtaining dependency information for exceptiongroup from https://files.pythonhosted.org/packages/ad/83/b71e58666f156a39fb29417e4c8ca4bc7400c0dd4ed9e8842ab54dc8c344/exceptiongroup-1.1.3-py3-none-any.whl.metadata
  Downloading exceptiongroup-1.1.3-py3-none-any.whl.metadata (6.1 kB)
Using cached microsoft_kiota_abstractions-0.6.0-py2.py3-none-any.whl (33 kB)
Using cached microsoft_kiota_http-0.4.4-py2.py3-none-any.whl (21 kB)
Using cached microsoft_kiota_serialization_json-0.3.7-py2.py3-none-any.whl (10 kB)
Using cached microsoft_kiota_serialization_text-0.2.1-py2.py3-none-any.whl (8.5 kB)
Using cached httpcore-0.17.3-py3-none-any.whl (74 kB)
Using cached certifi-2023.7.22-py3-none-any.whl (158 kB)
Using cached httpx-0.24.1-py3-none-any.whl (75 kB)
Using cached anyio-3.7.1-py3-none-any.whl (80 kB)
Downloading exceptiongroup-1.1.3-py3-none-any.whl (14 kB)
Building wheels for collected packages: apicurioregistrysdk
  Building wheel for apicurioregistrysdk (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for apicurioregistrysdk (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [29 lines of output]
      Downloading Kiota from URL: https://github.com/microsoft/kiota/releases/download/v1.4.0/linux-x64.zip
      Traceback (most recent call last):
        File "/tmp/pip-install-7ssbyb1x/apicurioregistrysdk_a213301ae1d94386bc795a1ddac03389/kiota-gen.py", line 81, in <module>
          generate_kiota_client_files({})
        File "/tmp/pip-install-7ssbyb1x/apicurioregistrysdk_a213301ae1d94386bc795a1ddac03389/kiota-gen.py", line 53, in generate_kiota_client_files
          shutil.copyfile(
        File "/usr/lib/python3.10/shutil.py", line 254, in copyfile
          with open(src, 'rb') as fsrc:
      FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-7ssbyb1x/apicurioregistrysdk_a213301ae1d94386bc795a1ddac03389/../common/src/main/resources/META-INF/openapi.json'
      Traceback (most recent call last):
        File "/home/forsberg/.virtualenvs/apicurio-registry-sdk/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/forsberg/.virtualenvs/apicurio-registry-sdk/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/forsberg/.virtualenvs/apicurio-registry-sdk/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-kd0b0ndw/overlay/lib/python3.10/site-packages/poetry/core/masonry/api.py", line 57, in build_wheel
          return WheelBuilder.make_in(
        File "/tmp/pip-build-env-kd0b0ndw/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 87, in make_in
          wb.build(target_dir=directory)
        File "/tmp/pip-build-env-kd0b0ndw/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 122, in build
          self._build(zip_file)
        File "/tmp/pip-build-env-kd0b0ndw/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 171, in _build
          self._run_build_script(self._package.build_script)
        File "/tmp/pip-build-env-kd0b0ndw/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 231, in _run_build_script
          subprocess.check_call([self.executable.as_posix(), build_script])
        File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/home/forsberg/.virtualenvs/apicurio-registry-sdk/bin/python', 'kiota-gen.py']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for apicurioregistrysdk
Failed to build apicurioregistrysdk
ERROR: Could not build wheels for apicurioregistrysdk, which is required to install pyproject.toml-based projects

Inspecting the .tar.gz downloaded by pip:

$ tar -zxvf apicurioregistrysdk-2.4.4.tar.gz 
apicurioregistrysdk-2.4.4/README.md
apicurioregistrysdk-2.4.4/apicurioregistrysdk/__init__.py
apicurioregistrysdk-2.4.4/kiota-gen.py
apicurioregistrysdk-2.4.4/pyproject.toml
apicurioregistrysdk-2.4.4/PKG-INFO

..the likely cause is that there simply is no openapi.json included.

Suggested solutions:

  1. Ensure openapi.json is included in source distribution.
  2. Build a pure python wheel instead of a binary one. This seems not entirely easy in poetry according to https://github.com/python-poetry/poetry/issues/3594

@andreaTP

apicurio-bot[bot] commented 1 year ago

Thank you for reporting an issue!

Pinging @jsenko, @EricWittmann to respond or triage.

andreaTP commented 1 year ago

Hi @forsberg , thanks for this report!

I'm on PTO till Mon, but I'm a bit surprised by the error you see, as the kiota-gen script should copy locally the OpenApi.json before packaging the distribution.

Anyhow, either I'll have a look on Mon or, if you already see where the error is, I'll review your PR. The logic should be to copy the OpenApi.json locally if not already present and include it in the distribution.(maybe an inclusion/exclusion issue with poetry?)

forsberg commented 1 year ago

Agree it's a bit mysterious, as when I try to repeat it locally, the .tar.gz built has all the generated code + openapi.json.

Yet, if you download the .tar.gz listed under https://pypi.org/project/apicurioregistrysdk/#files, it's very small and only contains the 5 files listed above.

I would suspect some kind of difference in how a local checkout looks vs. how it looks when run in the CI/CD system.

Note: I get a proper .tar.gz when I run poetry build, but CI seems to run poetry install. Not sure it matters.

andreaTP commented 1 year ago

I just checked, and quite a few things are off in the package automatically published 😢 . Thanks for reporting, I'm on it.

andreaTP commented 1 year ago

@forsberg this published release is good to go: https://pypi.org/project/apicurioregistrysdk/2.4.5/#files

would you mind confirming?

forsberg commented 1 year ago

Tried installing it in a python 3.10 virtualenv, works as a charm.

andreaTP commented 1 year ago

Thanks for the feedback as always @forsberg 👍