Setup instructions should specify nrfutil version

micolous commented 1 year ago

Expected Behavior

Following setup instructions installs an appropriate version of nrfutil for OpenSK.

Actual Behavior

Running pip install nrfutil on a system with Python 3.11 results in ./deploy.py saying:

You need to install nrfutil python3 package v6.0 or above. Found: v5.2.0.

Installing nrfutil==6.1.7 shows the cause: the package refuses to run on Python > 3.10, but nrfutil==5.2.0 is the newest version which doesn't block Python versions.

While this is an error in nrfutil's manifest, OpenSK's setup instructions should select a compatible version of nrfutil using appropriate version selectors.

Steps to Reproduce the Problem

Install Python 3.11
Check out OpenSK
./setup.sh
pip install nrfutil
./deploy.py --board nrf52840_dongle_dfu --opensk --programmer nordicdfu

Specifications

Version:
Platform: debian bookworm x86_64

kaczmarczyck commented 1 year ago

The pip version of nrfutil is archived on GitHub and replaced by a new tool that Nordic distributes over their website. The setup.py there lists Python version requirements as:

python_requires='>=3.7, <3.11',

So even if we fix it for 3.11, the tool will stop working soon. Instead, we will have to send users here to download the new tool. I'd say we require Python 3.10 for DFU for now, and wait for the new nrfutil to be available on pip.

I'll keep this open to track the problem.

micolous commented 1 year ago

The version of nrfutil on Nordic's website is an entirely different tool, marked as version 7 as a single static binary, rather than a Python library.

However, in addition to shelling out to nrfutil, deploy.py uses a Python module that is part of nrfutil v6.x, and would break on v7:

https://github.com/google/OpenSK/blob/f2496a8e6d71a4e838884996a1c9b62121f87df2/deploy.py#L694-L697

That bit of code looks like it's just doing a USB device listing, which could be done with libusb and looking for a device with DFU descriptors and the appropriate USB IDs.

That should unblock supporting newer versions of Python, modulo nrfutil v7's limited architecture support (only really a problem on Linux with older CPUs; current Windows ARM boxes run basic amd64 binaries "ok")

kaczmarczyck commented 1 year ago

I agree the new tool is different. This is what we needed the old tool for:

        dfu_pkg_file = f"{CARGO_TARGET_DIR}/{self.args.board}_dfu.zip"
        self.checked_command([ 
            "nrfutil", "pkg", "generate", "--hw-version=52", "--sd-req=0",
            "--application-version=1", f"--application={dest_file}",
            dfu_pkg_file
        ])

Am I right that this can't be done through libusb? To clarify, if you want to use DFU at the moment, you need to

either use an old Python,
or hack your way around the new tool yourself.

I haven't seen the new nrfutil available on pip, and I don't like sending users to a website to download something manually.

micolous commented 1 year ago

You'd need the (new) tool to do the DFU step; it may be possible to re-implement it, but it is probably not worth the hassle.

The main Python-side problem is the nordicsemi.lister module (from the nrfutil package on PyPI) which could be replicated with libusb, because that looks like it's just searching for connected USB serial devices. It looks like they have a Win32 implementation as well, but chances are that libusb could do that too.

kaczmarczyck commented 11 months ago

Partial fix under review, the pip dependency is gone. Now you still need the nrfutil binary in version 6.

JohannesBloecker commented 9 months ago

I managed to get the deploy.py running with the 7.7.0 binary from Nordic and python 3.11 on OSX:

What I had to change was that the Nordic binary wants the version option like this:

nrfutil --version

and then gives a very detailed multi-line version info with first line nrfutil 7.7.0 (c86dfaf 2024-01-23).

So for quick testing:

I added a new PROGRAMMERS entry "nordicdfu_nrfutil_binary"

Added the following new block:

if self.args.programmer == "nordicdfu_nrfutil_binary":
  assert_python_library("intelhex")
  assert_mandatory_binary("nrfutil")
  nrfutil_version = self.checked_command_output(["nrfutil", "--version"])
  nrfutil_version = nrfutil_version.removeprefix("nrfutil ")
  if not nrfutil_version.startswith("7."):
      fatal("You need to install nrfutil binary 7.0 or above. "
            f"Found:\n{nrfutil_version}\n")

and further down:

elif self.args.programmer in ("pyocd", "nordicdfu", "nordicdfu_nrfutil_binary", "none"): ... if self.args.programmer == "nordicdfu" or self.args.programmer == "nordicdfu_nrfutil_binary":

Ugly hack so I won't PR but that was it, script ran through and I managed to deploy to the nRF52840 dongle.

joostd commented 8 months ago

Another temporary workaround that I used to flash a nrf52840_dongle_dfu using macos (14.3.1):

$ brew install pyenv pyenv-virtualenv
$ pyenv --version
pyenv 2.3.36
$ pyenv install 3.10.13
$ eval "$(pyenv init -)"
$ eval "$(pyenv virtualenv-init -)"
$ pyenv virtualenv 3.10.13 venv
$ pyenv activate venv
(venv) $ python3 -V
Python 3.10.13
(venv) $ pip3 install --no-dependencies nrfutil==6.1.7
(venv) $ nrfutil version
nrfutil version 6.1.7

Then proceed according to the installation guide.

I still have an issue with configuring the device with an attestation certificate, but at least the dongle can then be used as a FIDO2 key.

kaczmarczyck commented 8 months ago

Thanks for contributing your workarounds.

Whats the attestation certificate issue you run into?

google / OpenSK