search

zephyrproject-rtos / zephyr

Primary Git Repository for the Zephyr Project. Zephyr is a new generation, scalable, optimized, secure RTOS for multiple hardware architectures.
https://docs.zephyrproject.org
Apache License 2.0
10.65k stars 6.52k forks source link

Support testing with ARM FVP in the CI #43852

Open stephanosio opened 2 years ago

stephanosio commented 2 years ago

Summary

Support running the Zephyr tests using the ARM FVP (fixed virtual platform; emulation software) in the CI.

Problem description

Some boards and ARM features (e.g. Ethos neural coprocessors) can only be tested using the ARM FVP emulation platform (QEMU does not currently support emulating them).

Proposed change

  1. Include ARM FVP as part of the CI Docker image private CI Docker image.

    • We need to check if ARM FVP EULA allows this. this is allowed since we are not redistributing anything.

  2. Update CI workflows to run tests using the ARM FVP included in the CI Docker image.

    • Set ARMFVP_BIN_PATH environment variable.

List of required FVPs

carlocaione commented 2 years ago

For emulation FVP should be more advanced and truthful to the hardware than QEMU, so maybe we can avoid all the usual icount issues with QEMU.

povergoing commented 2 years ago

While the annoying issue may be You are not permitted to redistribute the 'Fixed Virtual Platforms (FVPs)' Software (in whole or in part) or any derivatives thereof.. I am not an expert on Licenses, but it's a kind of dangerous thing I think. I already contacted the relevant guy half a year ago. But still no response...

povergoing commented 2 years ago

For emulation FVP should be more advanced and truthful to the hardware than QEMU, so maybe we can avoid all the usual icount issues with QEMU.

That's true, I 100% agree that including FVP into Docker img. Anyway, I'll try again. Fortunately, some high-level one in Arm is aware of this thing, so let's see.

stephanosio commented 2 years ago

@povergoing The clause you quoted falls under (b) if you are receiving a Non-Commercial Use License or version (as applicable) of the Arm Tools:, which I suppose is referring to those who have received a non-commercial license of the normally paid tools.

The (a) if you are receiving the Arm Tools free of charge: does not seem to explicitly prohibit redistribution, but it does not expressly permit redistribution either.

Weichen81 commented 2 years ago

Just some personal suggestion: another thing that need to be considered is that the instance to accept "EULA" must be a human, so the click or enter "OK" behaviors should not be done by a "BOT". For Cortex-A series, an architecture level FVP "Armv-A Base RevC AEM FVP" (https://developer.arm.com/tools-and-software/simulation-models/fixed-virtual-platforms/arm-ecosystem-models) can be download without "EULA". So maybe you can integrate the URL in docker image, but not FVP binary. But for Cortex-R, current we still haven't a proper way to bypass "EULA" click.

povergoing commented 2 years ago

@povergoing The clause you quoted falls under (b) if you are receiving a Non-Commercial Use License or version (as applicable) of the Arm Tools:, which I suppose is referring to those who have received a non-commercial license of the normally paid tools.

The (a) if you are receiving the Arm Tools free of charge: does not seem to explicitly prohibit redistribution, but it does not expressly permit redistribution either.

I am also very confused. This is totally out of my knowledge... But IMO, since the only way to get the FVP is to "register on arm website, accept "EULA" then download" (Arm wants to record who uses FVP?), there must be some restrictions on redistribution. In another world, if the binary is included in CI docker IMG or zephyr SDK, that means anyone can download FVP without accepting the EULA so that Arm cannot record.

Or is there any way that the host machine keeps the FVP binary instead of the docker IMG, mapping the FVP binary path to docker when running CI? In this way, we use FVP as a service and certainly do not redistribute FVP, so we don't need to care about the restrictions of redistribution.

stephanosio commented 2 years ago

Or is there any way that the host machine keeps the FVP binary instead of the docker IMG, mapping the FVP binary path to docker when running CI?

There is a long-term plan to implement something like that to get testing with the proprietary toolchains working in the CI, but it will take some time to get that implemented (no timeline yet).

povergoing commented 2 years ago

Hi @stephanosio @carlocaione, Update here: the FVP_BaseR_AEMv8R on page can download without any logins or registers.

The tricky part is how we avoid redistribution? That means the docker file should not contain the fvp bins (since anyone can easily get the docker image). Can we download the FVP at runtime triggered by some event? like, run ci -> trigger download fvp -> run ci test? Any suggestions?

stephanosio commented 2 years ago

Thanks for the update.

Can we download the FVP at runtime triggered by some event?

While that should work, doing so would incur a significant amount of traffic to the ARM website and also, more importantly, will slow down the CI depending on the file provider server load.

I think the best way to handle this would be to create a private CI Docker image, based on the public one, that is only available to the authenticated Zephyr CI runners (and of course, cannot be downloaded publicly).

povergoing commented 2 years ago

I think the best way to handle this would be to create a private CI Docker image, based on the public one, that is only available to the authenticated Zephyr CI runners (and of course, cannot be downloaded publicly).

Agreed, looks like it is the best way. So, is there anything we can help with? :)

stephanosio commented 2 years ago

Agreed, looks like it is the best way. So, is there anything we can help with? :)

I will implement a proof-of-concept this week and let you know if there are any issues. Thanks.

stephanosio commented 2 years ago

Test CI run using a private CI docker image that contains ARM FVP: https://github.com/zephyrproject-rtos/zephyr-testing/actions/runs/2215429286 https://github.com/zephyrproject-rtos/zephyr-testing/actions/runs/2216000834

stephanosio commented 2 years ago

@povergoing Testing with the latest ARM FVP available on the website, I am seeing a lot of failures:

cc @SgrrZhf @carlocaione

povergoing commented 2 years ago

Test CI run using a private CI docker image that contains ARM FVP: ~https://github.com/zephyrproject-rtos/zephyr-testing/actions/runs/2215429286~ https://github.com/zephyrproject-rtos/zephyr-testing/actions/runs/2216000834

Amazing work, I didn't expect you have done it so quickly 🥇

@povergoing Testing with the latest ARM FVP available on the website, I am seeing a lot of failures:

I see the failures. @SgrrZhf and I will fix them as soon as possible.

May I ask if those failures will block the CI or something for others? Can we see the results if we push a PR for fixing them in the GitHub action? That is, can we verify our fixes at upstream CI right now?

stephanosio commented 2 years ago

May I ask if those failures will block the CI or something for others?

@povergoing We can merge #45099 after these failures are fixed.

Can we see the results if we push a PR for fixing them in the GitHub action?

You can try opening a PR with the fixes to zephyr-testing to see if they pass.

povergoing commented 2 years ago

You can try opening a PR with the fixes to zephyr-testing to see if they pass.

understood, thanks!

stephanosio commented 2 years ago

Filed bug reports for the failures reported in https://github.com/zephyrproject-rtos/zephyr/issues/43852#issuecomment-1107855478:

45108

45110

45111

microbuilder commented 1 year ago

Required to enable CI testing of the Ethos-U55 in this PR, which is currently build only: https://github.com/zephyrproject-rtos/zephyr/pull/42580

carlescufi commented 1 year ago

Required to enable CI testing of the Ethos-U55 in this PR, which is currently build only: #42580

See: https://github.com/zephyrproject-rtos/zephyr/blob/f9b5138097de701cfbf99fd7546b5f4bf94ce5f3/samples/modules/tflite-micro/tflm_ethosu/sample.yaml#L8

zephyrbot commented 8 months ago

Hi @stephanosio,

This issue, marked as an RFC, was opened a while ago and did not get any traction. Please confirm the issue is correctly assigned and re-assign it otherwise.

Please take a moment to review if the issue is still relevant to the project. If it is, please provide feedback and direction on how to move forward. If it is not, has already been addressed, is a duplicate, or is no longer relevant, please close it with a short comment explaining the reason.

Thanks!

stephanosio commented 7 months ago

ARM FVP licence has been updated to not prohibit redistribution and therefore it is now possible to include it in the public Docker image, as done in https://github.com/zephyrproject-rtos/docker-image/pull/170.

CI will start using ARM FVP for the supported boards once the new Docker image is mainlined, after which this issue can be closed.