search

piwheels / packages

Issue tracker for piwheels package issues
https://github.com/piwheels/packages/issues
20 stars 5 forks source link

Package issue: hugo #445

Open agriyakhetarpal opened 2 months ago

agriyakhetarpal commented 2 months ago

Package name

hugo

Package version

all

PyPI URL

https://pypi.org/project/hugo

piwheels URL

https://www.piwheels.org/project/hugo/

Python version

I am aware this is the issue tracker for a Python package index specifically for Raspberry Pi

I have checked for duplicate issues

I am the maintainer

More information

Hi, there! I currently maintain https://github.com/agriyakhetarpal/hugo-python-distributions, which aims to provide pip-installable binaries for Python 3.8 and later (which are community-enabled and unofficial at this time) for the extended version of the Hugo static site generator for several platforms and architectures, and wanted to expand support for armv7l via piwheels (xref: https://github.com/agriyakhetarpal/hugo-python-distributions/issues/91). The issue with the current broken builds from the source distribution on piwheels is that the package requires some external, non-PyPI dependencies to compile the binary:

  1. Golang – most likely armv6l should work from https://go.dev/dl/, since they don't publish armv7l binaries anymore, I'm not sure.
  2. Git, for Golang to be able to download and compile the dependencies for the upstream project from GitHub and other code repositories
  3. A C compiler, because CGO_ENABLED is set to 1 to build the extended version of Hugo. Either Clang or a fairly recent GCC will work. Zig can be used as well, to enable cross-compilation.

Without these, the wheel builds will fail, as they have been doing so currently. Native compilation should be trivial after having access to these, but there might be a few bugs that I would be happy to fix with the packaging infrastructure that I have currently implemented. I don't have access to a CI server for building armv7l wheels, but I came here because I can receive help on getting the builds successful plus I'm willing to cooperate by providing a recipe or pre-installation scripts to deal with the above three points. I don't have a Raspberry Pi either right now, so if there is a manylinux equivalent of a Docker image available where one can build with emulation that I can be directed to, that would be great – I'm sure that the package build time won't pass the 3 hour CI time limit in any way; in general, it should not take more than an hour to build via emulation, considering that wheels for aarch64x, s390x, and ppc64le; all pass under 45 minutes. Native compilation takes 3 to 5 minutes without caches being preserved.

bennuttall commented 2 months ago

We do have golang and a C compiler available. I think there might be an issue with golang dependencies as we block git access to HTTP(S) for security reasons.

The latest build is failing with:

KeyError: 'armv7l'

Though I've not looked too closely to what this is.

See https://www.piwheels.org/logs/0000/1179/5137.txt

If there's a way we can build these separately, outside the automated piwheels build procedure, perhaps we can import these wheels.

bennuttall commented 2 months ago 
piwheels@piwheels50:~$ pip3 wheel hugo==0.125.4 --no-deps --extra-index-url https://www.piwheels.org/simple
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting hugo==0.125.4
  Using cached hugo-0.125.4.tar.gz (21 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Traceback (most recent call last):
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.11/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-7450rfeb/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-7450rfeb/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-7450rfeb/overlay/local/lib/python3.11/dist-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 34, in <module>
      KeyError: 'armv7l'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
agriyakhetarpal commented 2 months ago

Thanks for pointing this out, @bennuttall – I think the error is coming from this dictionary in the code:

import platform

HUGO_ARCH = {
    "x86_64": "amd64",
    "arm64": "arm64",
    "AMD64": "amd64",
    "aarch64": "arm64",
    "x86": "386",
    "s390x": "s390x",
    "ppc64le": "ppc64le",
}[platform.machine()]

that I use, where I need to add an additional target to build against armhf first to enable building against it, corresponding to the linux/arm duo with GOARM set to 7. I can push that and other related changes, and hope for another rebuild! Edit: just did!

bennuttall commented 2 months ago

Raspberry Pi 1 and Zero are armv6; the others are armv7. If you're on a 32-bit OS (i.e. not aarch64, which piwheels does not support atm), it's armv6 / armhf userland. Generally speaking, our armv6 and armv7 wheels are the same and will work on all Pi models. Otherwise someone could prep an SD card on a Pi 3/4/5 and insert it into a Pi 1/Zero and it wouldn't work.

agriyakhetarpal commented 2 months ago

Makes sense to me. I think armv7 would be the better choice to add, based on the upstream HugoReleaser configuration. In my case, this is the extended version instead of being the regular one, so I'm not sure how the compilation will go, but it is only through the logs that we will be able to debug.

agriyakhetarpal commented 2 months ago

Hi again, @bennuttall! I see that with my recent release (v0.125.5) from a few hours ago, this failed with a quite strange error log: https://www.piwheels.org/logs/0000/1183/9026.txt – at the stage where it was installing the build-time dependencies with OSError: [Errno 28] No space left on device? I can't find any related issue(s) about this in this repository's tracker or that for the main one, so I guess it would be best to tag you here?

Edit: never mind, found a few issues. I guess triggering another build would be great :) If there is a Docker image or anything I can use to debug builds locally, that would be quite helpful so that I can not rely on doing releases and fix things beforehand.

agriyakhetarpal commented 1 month ago

I think there might be an issue with golang dependencies as we block git access to HTTP(S) for security reasons.

On another run, I think this note came up with these logs: https://www.piwheels.org/logs/0000/1183/9631.txt fatal: transport 'https' not allowed when doing the git clone. Is there a mitigation or a feature flag available to disable this behaviour, since I am not sure if there is a way to not go with the git clone route? ☹️