Add Power(ppc64le) support to ClamAV

[snehagg24]

I am working on testing ClamAV on ppc64le architecture and would like to work on adding official Power support. I am able to build ClamAV successfully on Power and I will be sharing more updates as I test it. Can anyone guide me on the process for adding official support for additional arch? I would like to know how the binaries/images are released so that I can contribute required changes for Power. Any pointers would be really helpful. @micahsnyder @ragusaa @m-sola

[ragusaa]

Hi,

Thank you for your interest in clam.

As far as making ppc64le officially supported, we would have to add that to our testing setup. I'll see how feasible that is.

Are you interested in having us generate installation packages for, or just to have changes integrated into the clamav codebase so that it will not have to be patched when checked out?

We are happy to look at any pull requests you have, and will be happy to help get them merged into our codebase.

Thanks, Andy

[snehagg24]

@ragusaa We would like to have full support for ppc64le including installation packages/installers. I can work on contributing required code changes in your codebase. I am currently testing cross-compilation in this workflow https://github.com/Cisco-Talos/clamav/blob/main/.github/workflows/cmake.yml. However, I noticed that this workflow is only building and testing clamAV(its not released through the workflow). How are the official images/binaries being released?

[micahsnyder]

Hi @snehakpersistent, I'm the project lead.

TL;DR: we could probably build ppc64le docker images, but I don't think we will test the ppc64le docker images with our test pipeline, or build and test ppc64le RPM and DEB packages.

The official images/binaries we build are built and tested using an internal Jenkins server. The Jenkinsfile in the clamav repository kicks off the build and test pipelines, but those downstream Jenkinsfiles pipelines are internal-only.

Essentially what we advertise as "supported" on docs.clamav.net are platforms we are able to test with internally using either real hardware (for macOS), VM's (for FreeBSD), or Docker containers (for Linux). It is also limited by what we're willing to put the energy into to set up and maintain. For example, Solaris used to be supported, but we dropped support for it because it is difficult to work with, and is not commonly used.

Adding new docker architectures is something that is already requested, specifically for aarch64. It is something I am open to. I suspect that it wouldn't be hard to include ppc64le while we're at it. I did some local testing and had trouble building the aarch64 Docker image under alpine:latest, but was able to convert the image to use debian:11-slim and that build appeared to work correctly for aarch64 and ppc64le, outside of a test timeout failure caused by very slow builds of the libclamav-rust test executable (it currently counts towards the test timeout, which needs to change).

Building clamav install packages for additional platforms is more work, though. Our RPM & DEB packages are built using static libraries for the third party dependencies. At present, we compile those dependency static library in docker containers for a specific Linux versions on x86_64 or i686 docker hosts. To build all the dependencies, we use a tool called Mussels to run build scripts for each of the dependencies, in order, for each required platform. We don't have a ppc64le docker host. To build the dependencies and ClamAV we'd probably have get a ppc64le docker host, or cross-compile everything. Cross-compiling everything would mean figuring out Mussels "recipe" scripts to cross compile each library dependency for ppc64le and then cross-compiling clamav as well. Setting that up would be time consuming. I don't want to do that.

Instead of cross-compiling, I think it would be more practical to use QEMU to emulate a ppc64le that hosts docker, or use QEMU+docker to host ppc64le docker containers for use in our internal build/test environment. And that would make it possible to not only build but also test the release materials. But it would also be time consuming. We have a lot in our backlog and so it is I not something I want to prioritize at this time.

Unfortunately, outside of helping us with instructions for how to set up a ppc64le docker container environment, I don't think there is anything you can do to enable this yourself. I don't think it is ideal that most of our tests and test infrastructure isn't public, but there's no way we're going to make our Jenkins server public (Jenkins and it's plugins are a constant pile of CVE's). And we're too invested in Jenkins groovy scripts to transition fully to GitHub Actions.

-Micah

[snehagg24]

Thanks @micahsnyder for sharing the details. I was already working on adding Power support to the current build/test workflow so I went ahead and created PR https://github.com/Cisco-Talos/clamav/pull/635. You can check the output here https://github.com/snehakpersistent/clamav/runs/7334433848?check_suite_focus=true (Its failing due to some test failures. However, compilation works.) Let me know your inputs on this. Also, to build docker image for ppc64le, QEMU + docker buildx would be a good idea. The overall approach should be as follows: 1) Set up qemu: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes 2) Docker buildx: docker buildx build --platform "linux/ppc64le" -f <Dockerfile path> -t <Image:Tag> .

Additionally, sharing few more options to get ppc64le resources: 1) Ppc64le VM: https://openpower.ic.unicamp.br/minicloud/ (We can work on giving you access) 2) Ppc64le container: Using ppc64le ubuntu image ppc64le/ubuntu I'm willing to work on any changes required for adding official Power support. Let me know what you need from my end to proceed on this.

[snehagg24]

@micahsnyder any update on this?

[micahsnyder]

I was already working on adding Power support to the current build/test workflow so I went ahead and created PR https://github.com/Cisco-Talos/clamav/pull/635.

Your solution to automated testing with github actions is interesting, but it appears that emulating ppc64le on github's free ubuntu-latest node is way too slow. Maybe there is another resource for open source projects to run ppc64le natively for testing? Or if https://openpower.ic.unicamp.br/minicloud/ is faster, maybe we could install a github runner on there?

Also, to build docker image for ppc64le, QEMU + docker buildx would be a good idea.

As I said earlier, we have a lot on our backlog and I can't work on this right now or task any of my teammates to work on it. I'm sorry. And after seeing the run times for PR #635, I am concerned this will also be very slow. It's just not something we can investigate right now.