Closed parkerberb-nfv closed 6 years ago
parkerberb-nfv
commented
6 years ago This requires an upstream (?) change to Buildroot before it will work on its own. the package dmidecode has a dependency on x86 that is not needed. Deleting that dependency in dmidecode/Config.in allows the autotools to properly build dmidecode for arm. Not sure if that is the proper solution, but it works for me.
schmittjoseph
commented
6 years ago @parkerberb-nfv Thank you for the pull request!
Requesting @mastacontrola and @Sebastian-Roth to help review this
schmittjoseph
commented
6 years ago Thread note: When this is merged in, we should also update our published docker image and our Jenkinsfile to handle arm32 and arm64 builds.
mastacontrola
commented
6 years ago I'm going to merge the pull and build a new docker image for it.
I'll also, strictly, build the arm and arm64 builds just to see it all work. Then I'll work on adding support for the different binaries to allow using arm based elements.
Is there any reason the arm and arm64 can't use root.ext4.xz?
parkerberb-nfv
commented
6 years ago I don't know if it was just a configuration thing I missed, but the servers refused to boot anything that was not cpio.gz
mastacontrola
commented
6 years ago that is weird. I have 0 arm based devices, so I'll see if I can get the build to do ext.xz and bzImage for the kernels. I'll likely need your help in knowing if they work or not though.
parkerberb-nfv
commented
6 years ago You can make buildroot produce an ext4.xz, but the kernel cant decompress the .xz properly and it complains about "cpio magic" when its the ext4. My guess is that the kernel needs something extra baked in to support an initrd.ext4.xz. I can try testing stuff for you if you want to investigate
mastacontrola
commented
6 years ago From the sounds of it, the kernel config just isn't configured to use the xz/gz/lzmo etc... options shouldn't be too hard to figure out.
mastacontrola
commented
6 years ago So quick question, how are you building the ipxe binaries? Any special instructions to it?
parkerberb-nfv
commented
6 years ago I actually just grabbed the source and compiled it on one of my arm servers... I can send you the BLOB if you want but I never bothered to set up a cross compiler for that
mastacontrola
commented
6 years ago Okay, I'm doing a cross compiler for now. Trying to figure out the proper stuff.
parkerberb-nfv
commented
6 years ago Have you managed to build this yet? I assume you only want to have one 32-bit version, just like you do for the x86 ipxe? I ask because the target architecture at compile time is stored in the var ${buildarch} in ipxe, which is needed to differentiate arm/x86. That might be an issue, however, because if buildarch will always be arm32, then I am not sure of a way to differentiate between arm32 and arm64 from default.ipxe. AFAIK, the cpuid command is not supported on arm.
mastacontrola
commented
6 years ago I built the 64 bit binaries but had issue with the 32. I also had similar issues building the unit for arm32. Cpuid should work though I have no basis to differentiate of course. The kernels build properly for both arm. I just haven't pushed them anywhere yet. I did have to update the docker file and pushed it to docker hub.
parkerberb-nfv
commented
5 years ago @Sebastian-Roth Hi, I would be happy to contribute more. I am curious what still needs to be done? I have been happily using our forked FOG to manage our ARM servers. I haven't gotten around to updating our installation, but both my PRs were merged so I assume the latest stable release would work.
Sebastian-Roth
commented
5 years ago @parkerberb-nfv Whooops, sorry for the delay. Somehow lost track of this. Do your ARM servers are set to PXE boot all the time or do you PXE boot those only for imaging?
parkerberb-nfv
commented
5 years ago @Sebastian-Roth No worries. Our servers will always "PXE boot" by chain loading into the FOG provided iPXE image. If there is a pending task it will properly PXE boot with the arm kernel and init, otherwise it boots from disk.
Sebastian-Roth
commented
5 years ago @parkerberb-nfv Well that's interesting because I had expected the chainloading to boot from disk wouldn't work properly for ARM. But seems like it does. Which exit style do you use? Simple "exit"?
parkerberb-nfv
commented
5 years ago @Sebastian-Roth I set all this up a while ago, so please forgive me if I don't have all the details correct.
On the front end, In the configuration tab I set FOG_BOOT_EXIT_TYPE to GRUB and FOG_EFI_BOOT_EXIT_TYPE is set to REFIND_EFI. I also believe I had to compile my own ipxe image for the arm servers, but I did that following the instructions on the ipxe website. I didn't do anything special there. I also just found a refind_efi image for arm online somewhere, but these would not be too hard to build and include in the fog releases like you do with grub and refind for x86 already.
One thing I just remembered that definitely will need to be updated is allowing the user to configure their arm kernel and init on the front end. Currently the php scripts assume that if a host is not x86_64, then it must be an x86 32 bit machine. So when my arm servers PXE boot through FOG, FOG serves what it thinks is the x86 32 bit images. So I have FOG_TFTP_PXE_KERNEL_32 set to the arm kernel I built, for example. I fixed some of this on the backend (https://github.com/FOGProject/fogproject/pull/239) but I never touched the front end code. So the current state of everything only works for me because I only have 64 bit x86 machines and only 64 bit arm machines.
Another thing to note is that I am still running my forked (un-updated) version of FOG (version 1.4.4). I have not had to the time to set up a separate instance to make sure the updated upstream would still work with everything.
I am going to be swamped for the next couple of weeks, but I am happy to work on this when I am able. How about we move this conversation to a new issue for officially adding ARM support were we collect everything that needs to be done?
Sebastian-Roth
commented
5 years ago @parkerberb-nfv said:
I am going to be swamped for the next couple of weeks, but I am happy to work on this when I am able. How about we move this conversation to a new issue for officially adding ARM support were we collect everything that needs to be done?
Yeah definitely a great idea. Your answer here is a good starting point for sure. Would you mind opening the new issue and close this one here?
I have to admit that the next couple of week I am way too busy as well to work on this. But collecting the information is definitely a good start.
Adds the necessary changes to docker, build,sh, and configs to allow FOS to be built and operate on ARMv8.
NOTE: I have only tested this on arm64. I don't have any arm32 machines to test this on, and so I make no promises