Circle CI: Enable testing for ARM64 platform

[odidev]

Hi Team, I have been working on running and testing Falco on arm64 platform. Circle CI allows to run jobs on arm64 architecture as can be seen here.

Would you be interested to enable testing for arm64 platform on Circle CI? Would be happy to help with it.

[FedeDP]

Hi @odidev ! Is Falco actually running on arm64? I know that there have been quite a few changes, mostly to kmod or eBPF probe, to support arm64, but i guess it was never really tested.

Btw I am surely interested! /cc @jasondellaluce @leogr

[leogr]

Although I have never tried, Falco should run on arm64 but with some caveats (as @FedeDP said).

So, I like the idea of adding a job for testing it with arm64. However, since it may not work initially, the job should not be set as required to merge PRs. Also, we should first introduce arm64 testing in libs

Also, I'd prefer using Prow in our test-infra (instead of CircleCI, which we likely abandon one day). @zuc: you already experimented with arm64 on Prow, didn't you?

In any case, @odidev, your help would be really appreciated! :pray:

[zuc]

Also, I'd prefer using Prow in our test-infra (instead of CircleCI, which we likely abandon one day). @zuc: you already experimented with arm64 on Prow, didn't you?

@leogr Yep, and I second what you are saying about the value of spending some effort on Prow compared to CircleCI 👍

@odidev If I can be of any help in giving you a bit more context on our Prow CI infrastructure or on anything else related to this, please just ask (either here or on slack)!

[FedeDP]

@odidev i had success in running Falco on 5.10.96-90.460.amzn2.aarch64; therefore i think this is even more important now, to avoid future regressions.

[zuc]

Hi @odidev - yesterday we ran our first full Falco build on ARM64 (see https://prow.falco.org/view/s3/falco-prow-logs/logs/arm-build-falco/1494368411055034368).

While there's some stuff still to be carried out to make the pipeline stable and reliable, this is a nice first step. Feel like helping out? We'd be more than happy to welcome your contribs on this!

[odidev]

Sure, Would be happy to help. Assuming that the above build log is for the file https://github.com/falcosecurity/test-infra/blob/master/config/jobs/arm-build/arm-build.yaml in test-infra repository, would like to see testing command make test added to it. I was facing issue while running make all command which builds complete project due to pre-build cloudtrail and json plugins not available for ARM64 platform and had raised issue for it, would appreciate if you could take a look into that as well.

Issue link: https://github.com/falcosecurity/falco/issues/1821

[FedeDP]

Hi @odidev ! I think we can try to avoid those tests for now on arm64 platform, given that they're unsupported. I see that there is a SKIP_PLUGINS_TESTS here: https://github.com/falcosecurity/falco/blob/cb51522423001745029a237fc352d33d27399fdf/test/run_regression_tests.sh that, when set to true, should do the trick; are you willing to try and eventually open a PR on test-infra to set that flag to true? Thank you :)

[odidev]

I have tried running the regression tests with SKIP_PLUGINS_TESTS flag set as true on ARM64 platform and they are passing successfully. Could we add these tests for ARM64 platform on Prow?

[FedeDP]

Nice! In https://github.com/falcosecurity/test-infra/pull/679 i am enabling build of plugins on arm64 too. That should fixed any issue :) About arm-build job, i think we can skip testing phase at the moment: i mean, nobody really uses that job and it will go away as soon as we enable circleCI build for arm64 for Falco.
What we still miss is:

• proper drivers build for arm64
• circleCI enabled to build Falco artifacts for arm64

[odidev]

We would be happy to help in enabling CircleCI build for arm64 for Falco.

[FedeDP]

Hi! We have some steps:

• Push falco-builder image for arm64 (BUT, falco-builder image downloads some deps from downloads.falco.org, and they are amd64 only. We should find and upload arm64 deps too!)
• Fix falco-driver-loader to support arm64 artifacts download (basically, we should update this line: https://github.com/falcosecurity/falco/blob/master/scripts/falco-driver-loader#L204, like this, pseudocode:

  if arch == "aarch64"
  "${DRIVERS_REPO}/${DRIVER_VERSION}/aarch64/${FALCO_KERNEL_MODULE_FILENAME}"

• Then, we are able to enable CI for Falco; the CI should use the falco-builder image on an arm machine to build and push Falco for aarch64, in a similar way than what we do with build/centos7.

I think we really need the first and third steps right now; the first step is also needed for the driverkit builder image that is also needed to build drivers for arm64.

If you are willing to help, we all would be very very glad :)

EDIT: you can find list of deps on https://download.falco.org/?prefix=dependencies/. Moreover, some comments give useful tips here: https://github.com/falcosecurity/falco/blob/master/docker/falco/Dockerfile#L39

[FedeDP]

I should've found all the required deps, except for:

• libmpx -> Intel only
• libcilkrts -> Intel only
• libasan (?)

Today, i am gonna try finding libasan, then we should be able to open a PR to push the new falco builder (and driverkit builder) images (ofc we first need to upload these deps to downloads.falco.org :) )

EDIT: ok, we just miss libasan0 for gcc 4.8. I cannot find it, and it was not supported for multiple arch back then. I guess gcc 4.8 for arm64 didn't have a dep on it.

[FedeDP]

So, deps for arm64 are now uploaded to downloads.falco.org.
I am currently working in porting driverkit builder image to support arm64 too. Falco builder should work fine on arm64. We should also port other images:

• [x] builder image -> easy; already tested on arm64 and it builds
• [x] driver-loader image -> easy; should work just fine once we have the Falco image for arm64
• [x] Falco -> requires same steps as driverkit builder image; i am on it!
• [x] local -> requires same setps as driverkit builder image
• [x] no-driver -> requires minor changes (basically this one: https://github.com/falcosecurity/falco/blob/master/docker/no-driver/Dockerfile#L14)
• [x] tester -> requires minor changes
• [x] ubi -> requires minor changes

I think you can start working on the circleCI for Falco, assuming that there will be a Falco docker builder image for arm64. As soon as the Falco image is pushed, we should then be ready to test it.

WDYT?

[odidev]

This is a great step. We would like to point out a two things:

1) The builder image was able to build for us only after a minor tweak that is changing llvm-toolset-7 to llvm-toolset-7.0 in lines https://github.com/falcosecurity/falco/blob/7aed3b6d0113a0011d286da51f40533423d6f3d2/docker/builder/Dockerfile#L23 and https://github.com/falcosecurity/falco/blob/7aed3b6d0113a0011d286da51f40533423d6f3d2/docker/builder/Dockerfile#L28 as it was not able to find llvm-toolset-7 for arm64 on our local system. This could be ignored as it worked for you.

2) Falco image tries to download Falco binary from https://download.falco.org/?prefix=packages/deb/stable/ in line https://github.com/falcosecurity/falco/blob/7aed3b6d0113a0011d286da51f40533423d6f3d2/docker/falco/Dockerfile#L82 and since there is no binary available for arm64 there we had build it from source but that takes up lot of space and time when docker image is built. Would it be possible to make arm64 binary available there so that it takes lesser time and space to build?

I will work on adding a new CircleCI job for falco for arm64, It would have to follow the syntax given here.

[FedeDP]

Hi!

1. Weird, it built successfully for me :( Well, i'll give it another look :)
2. Yep, sure. We will provide prebuilt binary for Falco. Indeed, that's part of CircleCI business :) We do not only want CI to test builds, but also to publish dev and release packages for falco.

In the end, for now please assume that the relevant packages will be there in your CircleCI job; they will be there once we fix up everything indeed!

EDIT: see here for relevant jobs:

• publish-dev: https://github.com/falcosecurity/falco/blob/master/.circleci/config.yml#L341
• publish-deb-dev: https://github.com/falcosecurity/falco/blob/master/.circleci/config.yml#L371 etc etc...

We will need to also duplicate those, to publish arm64 too.

[FedeDP]

Update: i opened a draft PR on driverkit to support arm64 builds.
I am still fighting against a couple of build issues (namely Ubuntu, Debian and AmazonLinux2 builders are failing at the build stage). In the meantime, i am working on a PR on Falco to port all the images + falco-driver-loader script for arm64 too.

[FedeDP]

@odidev i opened a wip PR on Falco to port all docker images to be multiplatform ready: https://github.com/falcosecurity/falco/pull/1990/files; also, fixed the falco-driver-loader script to support multiple architectures.

[odidev]

That is great. Here is a sample CircleCI job to build Falco for arm64:

https://github.com/odidev/falco/commit/99d26b50ad0ba56ea3752821245af8b5f07f439d.

Kindly take a look. We can test it out once we have Falco docker builder image for arm64.

[FedeDP]

@odidev That's huge! Thanks man! I am reviewing directly on your commit!

EDIT: finished my review!

[odidev]

I have made the modifications based on your review: https://github.com/odidev/falco/commit/c51d05a77424b013f4a4913007af1f3f9f0a252d .

Kindly take a look.

[FedeDP]

I left a comment; aside from that, it LGTM! After you fixed that last small issue, you can open the PR to discuss it further! Thanks!

[FedeDP]

I left a couple of more comments to fix issues. Note that we will probably need to update publish-deb script (https://github.com/falcosecurity/falco/blob/master/scripts/publish-deb#L41) and possibly publish-rpm one (i am not sure about the latter).

[odidev]

I have opened a PR to add Circle CI jobs for arm64: https://github.com/falcosecurity/falco/pull/1997 .