Unable to install PyNaCl on Windows on Arm64

[prayaas-a]

When you try to install PyNaCl using pip (with Python 3.11 which is native to Arm64 on Windows), pip tries to build PyNaCl from scratch which doesn't seem possible on most Windows systems given the dependency on "make". This is because the Windows on Arm64 version of pip has only "win_arm64" and "any" as compatible tags, so the prebuilt wheels for x86 and x64 don't work.

Could you consider publishing a prebuilt wheel for the win_arm64 target as well? I believe Visual Studio 2019 includes support for cross-compiling for Arm64, but you should be able to use VS2022 if not. It looks like libsodium isn't publishing Arm64 binaries for Windows yet, but it should be a fairly trivial task for them considering they're using MSVC 2019 and 2022. I can also open an issue with them if that's going to be a hard dependency. Thanks.

[reaperhulk]

Please do file the upstream issue. We'd like to support this, although lack of arm64 windows CI with GHA makes this a bit worrisome to ship since we can't test it automatically.

[prayaas-a]

Please do file the upstream issue. We'd like to support this, although lack of arm64 windows CI with GHA makes this a bit worrisome to ship since we can't test it automatically.

Thanks for the response. I have created an upstream issue here. libsodium already supports building for an Arm64 target, but they haven't published binaries yet.

As far as GitHub Actions, yeah, that's a bit problematic. I'm not sure how well it integrates into GitHub Actions but you should be able to use Azure VMs and/or pipelines as they've begun offering Windows on Arm64 images.

[reaperhulk]

The reliability of CI drops off pretty rapidly as we add multiple CI services (and we already use both GHA and CircleCI), but could you point me at where pipelines offers a microsoft-hosted arm64 windows runner? I haven't been able to find anything about native arm64 windows there.

[prayaas-a]

Yes, that's a real concern. Ideally you don't want to have to deal with complex processes with moving parts. I see that Microsoft does not currently offer Microsoft-hosted Arm64 agents; however, you can create a Windows Arm64 VM on Azure, connect to it, and install the agent software on it manually. It can then be used by Azure Pipelines like any other build agent/pool.

When you create a VM on Azure, choose one of the Arm options for size (I selected Standard D2pdsv5). For the image, the "Windows 11 Enterprise - Gen 2" options appears to support Arm64. You can also click "See all images" and search for Arm to find one. Once the VM is set up, you can install the x64 version of the Azure Pipelines agent on it as usual. I can confirm that it works fine.

[reaperhulk]

Self-hosted isn’t a viable option for us since it doesn’t provide the security boundary of ephemerality nor the maintenance properties we need (it’s not practical for us to monitor and patch a persistent instance). So while that would hypothetically work it’s not a path we’re willing to pursue. Supporting this really does need some support from a CI provider (and hey, Azure pipelines and GitHub just happen to be owned by the company that also produces Windows! 😄)

[echoix]

I happened to stumble upon this when looking why PyNaCl had to be compiled in our docker images since no adequate wheel was available, and maybe this could help both of us. Have you ever taken a look at https://cibuildwheel.readthedocs.io/en/stable/? It seems to be a tool that is there to help specifically with the problem of building for multiple platforms. In the case of Windows arm64, it is indeed not easily available for CI. We have to keep in mind that Windows server 2019 and 2022 don't have any ARM releases at all. And we have to remember that we learned (as a public) in Nov 2021 that Qualcomm had an exclusivity for Windows on ARM, explaining the limited devices available. All that to say, that the approach cibuildwheel has in that case to help out with cross compiling is pretty appropriate, and probably way faster anyway. Because of this cross-compiling, and the fact explained above that a legal Windows on ARM version doesn't exist to be emulated, it is not really possible to "test" the Windows arm64 wheels that are built.

[alex]

We're aware of cibuildwheel, but it doesn't solve the problem of the fact that we will not distribute wheels that we cannot test.

If MS has made it impossible to actually have arm64 in CI, then that problem needs to be directed at them, we're not going ship untested things.

[mihir1739]

I tried building paramiko that has a dependency on pynacl. Although the build for cryptography was successful, and I have even installed OpenSSL. Yet the build for pynacl fails with a cryptic error message. Can anyone please help. Error messages:

Building wheels for collected packages: pynacl
  Building wheel for pynacl (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pynacl (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [41 lines of output]
      C:\Users\mihir\AppData\Local\Temp\pip-build-env-coskp_n9\overlay\Lib\site-packages\setuptools\_distutils\dist.py:261: UserWarning: Unknown distribution option: 'tests_require'
        warnings.warn(msg)
      running bdist_wheel
      running build
      running build_py
      creating build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\encoding.py -> build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\exceptions.py -> build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\hash.py -> build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\hashlib.py -> build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\public.py -> build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\secret.py -> build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\signing.py -> build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\utils.py -> build\lib.win-arm64-cpython-313\nacl
      copying src\nacl\__init__.py -> build\lib.win-arm64-cpython-313\nacl
      creating build\lib.win-arm64-cpython-313\nacl\pwhash
      copying src\nacl\pwhash\argon2i.py -> build\lib.win-arm64-cpython-313\nacl\pwhash
      copying src\nacl\pwhash\argon2id.py -> build\lib.win-arm64-cpython-313\nacl\pwhash
      copying src\nacl\pwhash\scrypt.py -> build\lib.win-arm64-cpython-313\nacl\pwhash
      copying src\nacl\pwhash\_argon2.py -> build\lib.win-arm64-cpython-313\nacl\pwhash
      copying src\nacl\pwhash\__init__.py -> build\lib.win-arm64-cpython-313\nacl\pwhash
      creating build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_aead.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_box.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_core.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_generichash.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_hash.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_kx.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_pwhash.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_scalarmult.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_secretbox.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_secretstream.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_shorthash.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\crypto_sign.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\randombytes.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\sodium_core.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\utils.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\bindings\__init__.py -> build\lib.win-arm64-cpython-313\nacl\bindings
      copying src\nacl\py.typed -> build\lib.win-arm64-cpython-313\nacl
      running build_clib
      error: [WinError 193] %1 is not a valid Win32 application
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pynacl
Failed to build pynacl