pyocd / cmsis-pack-manager

A Rust and Python module for handling CMSIS Pack files
Apache License 2.0
25 stars 33 forks source link

Error Building Wheel with Python 3.11 #194

Closed banduccm closed 1 year ago

banduccm commented 1 year ago

I was attempting to update my Zephyr workspace after updating to Fedora 37 and it seems as though the wheel for cmsis-pack-manager no longer builds:

(.venv) [mbanducci zephyr]$ pip install cmsis-pack-manager
Collecting cmsis-pack-manager
  Using cached cmsis-pack-manager-0.4.0.tar.gz (43 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting appdirs<2.0,>=1.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting milksnake<1.0,>=0.1.5
  Using cached milksnake-0.1.5-py2.py3-none-any.whl (9.6 kB)
Requirement already satisfied: pyyaml<7.0,>=6.0 in /home/mbanducci/zephyrproject/.venv/lib64/python3.11/site-packages (from cmsis-pack-manager) (6.0)
Collecting cffi>=1.6.0
  Using cached cffi-1.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (462 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: cmsis-pack-manager
  Building wheel for cmsis-pack-manager (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for cmsis-pack-manager (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [285 lines of output]
      /tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
        warnings.warn(msg, warning_class)
      running bdist_wheel
      running build
      running build_py
      creating build/lib
      creating build/lib/cmsis_pack_manager
      copying cmsis_pack_manager/pack_manager.py -> build/lib/cmsis_pack_manager
      copying cmsis_pack_manager/_version.py -> build/lib/cmsis_pack_manager
      copying cmsis_pack_manager/_native__ffi.py -> build/lib/cmsis_pack_manager
      copying cmsis_pack_manager/_native.py -> build/lib/cmsis_pack_manager
      copying cmsis_pack_manager/__init__.py -> build/lib/cmsis_pack_manager
          Updating crates.io index
         Compiling libc v0.2.137
         Compiling autocfg v1.1.0
         Compiling cfg-if v1.0.0
         Compiling cfg-if v0.1.10
         Compiling futures v0.1.31
         Compiling log v0.4.17
         Compiling byteorder v1.4.3
         Compiling semver-parser v0.7.0
         Compiling either v1.8.0
         Compiling maybe-uninit v2.0.0
         Compiling lazy_static v1.4.0
         Compiling cc v1.0.76
         Compiling scopeguard v1.1.0
         Compiling proc-macro2 v1.0.47
         Compiling unicode-ident v1.0.5
         Compiling quote v1.0.21
         Compiling fnv v1.0.7
         Compiling tinyvec_macros v0.1.0
         Compiling syn v1.0.103
         Compiling rand_core v0.4.2
         Compiling unicode-bidi v0.3.8
         Compiling once_cell v1.16.0
         Compiling version_check v0.9.4
         Compiling spin v0.5.2
         Compiling untrusted v0.7.1
         Compiling tinyvec v1.6.0
         Compiling memchr v2.5.0
         Compiling lock_api v0.3.4
         Compiling matches v0.1.9
         Compiling serde_derive v1.0.147
         Compiling itoa v0.4.8
         Compiling serde v1.0.147
         Compiling semver v0.9.0
         Compiling autocfg v0.1.8
         Compiling adler v1.0.2
         Compiling rand_core v0.3.1
         Compiling httparse v1.8.0
         Compiling failure_derive v0.1.8
         Compiling hashbrown v0.12.3
         Compiling percent-encoding v1.0.1
         Compiling miniz_oxide v0.5.4
         Compiling gimli v0.26.2
         Compiling percent-encoding v2.2.0
         Compiling unicode-xid v0.2.4
         Compiling rustc_version v0.2.3
         Compiling crossbeam-utils v0.7.2
         Compiling slab v0.4.7
         Compiling memoffset v0.5.6
         Compiling crossbeam-epoch v0.8.2
         Compiling unicase v2.6.0
         Compiling smallvec v0.6.14
         Compiling indexmap v1.9.1
         Compiling rand_pcg v0.1.2
         Compiling rand_chacha v0.1.1
         Compiling form_urlencoded v1.1.0
         Compiling rand v0.6.5
         Compiling num-traits v0.2.15
         Compiling base64 v0.10.1
         Compiling rustc-demangle v0.1.21
         Compiling parking_lot_core v0.6.2
         Compiling parking_lot v0.9.0
         Compiling hyper v0.12.36
         Compiling try-lock v0.2.3
         Compiling serde_json v1.0.87
         Compiling crc32fast v1.3.2
         Compiling num-integer v0.1.45
         Compiling object v0.29.0
         Compiling unicode-normalization v0.1.22
         Compiling rand_xorshift v0.1.1
         Compiling rand_hc v0.1.0
         Compiling rand_isaac v0.1.1
         Compiling rand_jitter v0.1.4
         Compiling ryu v1.0.11
         Compiling encoding_rs v0.8.31
         Compiling iovec v0.1.4
         Compiling num_cpus v1.14.0
         Compiling net2 v0.2.38
         Compiling ring v0.16.20
         Compiling tokio-sync v0.1.8
         Compiling bytes v0.4.12
         Compiling time v0.1.44
         Compiling backtrace v0.3.66
         Compiling want v0.2.0
         Compiling mime_guess v2.0.4
         Compiling rand_os v0.1.3
         Compiling itoa v1.0.4
         Compiling try_from v0.3.2
         Compiling tokio-executor v0.1.10
         Compiling mio v0.6.23
         Compiling crossbeam-queue v0.2.3
         Compiling futures-cpupool v0.1.8
         Compiling mime v0.3.16
         Compiling dtoa v0.4.8
         Compiling crossbeam-utils v0.8.12
         Compiling idna v0.1.5
         Compiling tokio-current-thread v0.1.7
         Compiling tokio-timer v0.2.13
         Compiling tokio-io v0.1.13
         Compiling http v0.1.21
         Compiling string v0.2.1
         Compiling crossbeam-deque v0.7.4
         Compiling tokio-buf v0.1.1
         Compiling idna v0.3.0
         Compiling idna v0.2.3
         Compiling flate2 v1.0.24
         Compiling dirs-sys v0.3.7
         Compiling quick-xml v0.17.2
         Compiling scoped-tls v0.1.2
         Compiling iana-time-zone v0.1.53
         Compiling tokio-threadpool v0.1.18
         Compiling unicode-width v0.1.10
         Compiling dirs v4.0.0
         Compiling termcolor v1.1.3
         Compiling tokio-codec v0.1.2
         Compiling xdg v2.4.1
         Compiling textwrap v0.11.0
         Compiling atty v0.2.14
         Compiling strsim v0.8.0
         Compiling url v1.7.2
         Compiling tokio-reactor v0.1.12
         Compiling mio-uds v0.6.8
         Compiling url v2.3.1
         Compiling uuid v0.7.4
         Compiling tokio-fs v0.1.7
         Compiling minidom v0.12.0
         Compiling crossbeam-channel v0.5.6
         Compiling bitflags v1.3.2
         Compiling ansi_term v0.12.1
         Compiling vec_map v0.8.2
         Compiling tokio-tcp v0.1.4
         Compiling tokio-uds v0.2.7
         Compiling tokio-udp v0.1.6
         Compiling app_dirs v1.2.1
         Compiling clap v2.34.0
         Compiling chrono v0.4.23
         Compiling cookie v0.12.0
         Compiling tokio v0.1.22
         Compiling publicsuffix v1.5.6
         Compiling pbr v1.0.4
         Compiling addr2line v0.17.0
         Compiling tokio-core v0.1.18
         Compiling sct v0.6.1
         Compiling webpki v0.21.4
         Compiling http-body v0.1.0
         Compiling h2 v0.1.26
         Compiling ct-logs v0.6.0
         Compiling rustls v0.16.0
         Compiling webpki-roots v0.17.0
         Compiling simplelog v0.8.0
         Compiling synstructure v0.12.6
         Compiling tokio-rustls v0.10.3
         Compiling ctor v0.1.26
         Compiling failure v0.1.8
         Compiling hyper-rustls v0.17.1
         Compiling serde_urlencoded v0.5.5
         Compiling cookie_store v0.7.0
         Compiling reqwest v0.9.24
         Compiling cmsis-pack v0.3.0 (/tmp/pip-install-t1h86i5d/cmsis-pack-manager_0df7306226554d2ab758737493fb3cfd/rust/cmsis-pack)
      warning: fields `name`, `description` and `doc` are never read
         --> cmsis-pack/src/pdsc/component.rs:160:5
          |
      159 | pub struct Bundle {
          |            ------ fields in this struct
      160 |     name: String,
          |     ^^^^
      ...
      164 |     description: String,
          |     ^^^^^^^^^^^
      165 |     doc: String,
          |     ^^^
          |
          = note: `#[warn(dead_code)]` on by default
          = note: `Bundle` has a derived impl for the trait `Debug`, but this is intentionally ignored during dead code analysis

         Compiling cmsis-cli v0.3.0 (/tmp/pip-install-t1h86i5d/cmsis-pack-manager_0df7306226554d2ab758737493fb3cfd/rust/cmsis-cli)
      warning: `cmsis-pack` (lib) generated 1 warning
         Compiling cmsis-cffi v0.3.0 (/tmp/pip-install-t1h86i5d/cmsis-pack-manager_0df7306226554d2ab758737493fb3cfd/rust/cmsis-cffi)
      warning: unused return value of `std::mem::replace` that must be used
         --> cmsis-cffi/src/pack_index.rs:145:13
          |
      145 |             mem::replace(boxed.borrow_mut(), next_state);
          |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          |
          = note: `#[warn(unused_must_use)]` on by default
          = note: if you don't need the old value, you can just assign the new value directly

      warning: unused return value of `std::mem::replace` that must be used
         --> cmsis-cffi/src/pack_index.rs:191:13
          |
      191 |             mem::replace(boxed.borrow_mut(), next_state);
          |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          |
          = note: if you don't need the old value, you can just assign the new value directly

      warning: `cmsis-cffi` (lib) generated 2 warnings
          Finished release [optimized] target(s) in 28.11s
      copying dylib libcmsis_cffi.so
      Traceback (most recent call last):
        File "/home/mbanducci/zephyrproject/.venv/lib64/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/home/mbanducci/zephyrproject/.venv/lib64/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/mbanducci/zephyrproject/.venv/lib64/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 412, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 397, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 66, in <module>
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/milksnake/setuptools_ext.py", line 88, in run
          func(base_path=base_path, inplace=False)
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/milksnake/setuptools_ext.py", line 276, in build_cffi
          ffi = make_ffi()
                ^^^^^^^^^^
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/milksnake/setuptools_ext.py", line 262, in make_ffi
          from milksnake.ffi import make_ffi
        File "/tmp/pip-build-env-a67mmg56/overlay/lib/python3.11/site-packages/milksnake/ffi.py", line 8, in <module>
          _directive_re = re.compile(r'^\s*#.*?$(?m)')
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.11/re/__init__.py", line 227, in compile
          return _compile(pattern, flags)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.11/re/__init__.py", line 294, in _compile
          p = _compiler.compile(pattern, flags)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.11/re/_compiler.py", line 743, in compile
          p = _parser.parse(p, flags)
              ^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.11/re/_parser.py", line 980, in parse
          p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.11/re/_parser.py", line 455, in _parse_sub
          itemsappend(_parse(source, state, verbose, nested + 1,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.11/re/_parser.py", line 841, in _parse
          raise source.error('global flags not at the start '
      re.error: global flags not at the start of the expression at position 9
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cmsis-pack-manager
Failed to build cmsis-pack-manager
ERROR: Could not build wheels for cmsis-pack-manager, which is required to install pyproject.toml-based projects

Other notes:

bp-amitchone commented 1 year ago

I'm also experiencing this exact same issue when attempting to setup a fresh Zephyr environment on a new machine running Win 11 Pro 21H2

mbrossard commented 1 year ago

Could you try to build the maturin branch? I am not sure I have properly connected all the pieces. I need to update GitHub Actions configuration, and setuptools_scm needs to be replaced.

banduccm commented 1 year ago

@mbrossard I'll give it a shot. Initial attempts were unsuccessful, as Fedora has deprecated python2 pip, and I don't have a good way to get a python2-compatible version of setuptools on this machine. I'll try to make some time to get a VM going to try the build.

Just for the sake of completeness, I can confirm that running setup.py with python3 doesn't work :)

mbrossard commented 1 year ago

Initial attempts were unsuccessful, as Fedora has deprecated python2 pip, and I don't have a good way to get a python2-compatible version of setuptools on this machine. I'll try to make some time to get a VM going to try the build.

Just for the sake of completeness, I can confirm that running setup.py with python3 doesn't work :)

cmsis-pack-manager does not support Python 2.x anymore and you probably need Python 3.7 with maturin (Python 3.6 is EOL).

To test the maturin you need the rust compiler installed (https://rustup.rs/) and Python with pip, then it should be as simple as:

pip install .
banduccm commented 1 year ago

Oh! I'm sorry, I misunderstood and went with the readme.

I tried again with maturin, and the build appears to have worked:

$ pip install .
Defaulting to user installation because normal site-packages is not writeable
Processing /home/mbanducci/repos/cmsis-pack-manager
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting appdirs<2.0,>=1.4
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Requirement already satisfied: cffi in /usr/lib64/python3.11/site-packages (from cmsis-pack-manager==0.5.0) (1.15.1)
Requirement already satisfied: pyyaml<7.0,>=6.0 in /usr/lib64/python3.11/site-packages (from cmsis-pack-manager==0.5.0) (6.0)
Requirement already satisfied: pycparser in /usr/lib/python3.11/site-packages (from cffi->cmsis-pack-manager==0.5.0) (2.20)
Requirement already satisfied: ply==3.11 in /usr/lib/python3.11/site-packages (from pycparser->cffi->cmsis-pack-manager==0.5.0) (3.11)
Building wheels for collected packages: cmsis-pack-manager
  Building wheel for cmsis-pack-manager (pyproject.toml) ... done
  Created wheel for cmsis-pack-manager: filename=cmsis_pack_manager-0.5.0-py3-none-linux_x86_64.whl size=3739428 sha256=774bf0d470b7f25b2ffe540a093ee8e0f74df30fbaa08d1716f45129fded66e4
  Stored in directory: /home/mbanducci/.cache/pip/wheels/36/fc/b2/1799e56fe913dedc539a78b08828ead292fbb3ea77e774a4f5
Successfully built cmsis-pack-manager
Installing collected packages: appdirs, cmsis-pack-manager
Successfully installed appdirs-1.4.4 cmsis-pack-manager-0.5.0

After testing the build, I tried making a new venv for Zephyr, installing cmsis-pack-manager manually from the checked-out repo, then following the rest of the typical setup steps for west. Everything installed as expected and I'm able to build Zephyr projects again.