Open tlaurion opened 4 years ago
Back then I posted some of my efforts trying to get KGPE-D16 working on the mailing list. Unfortunately it did not get me anywhere further than that. It's possible that libreboot still works okay for this board (given its codebase), but I'm not as familiar with libreboot's build environment as with coreboot's.
With coreboot 4.11 (4.11_branch), RAM initialization issues on KGPE-D16/KCMA-D8 are hit-or-miss, at least I can get some 100%-bootable RAM combinations on KGPE-D16 for the time being. However, it seems the real issue isn't RAM and lies in somewhere else. For example, something's not right in libpayload with this board as after looking into the code the fatal error messages I'm getting in some places (like trying to run nvramcui/coreinfo) originated from there (mostly USB related, and the functions in question had something to do with DMA). I'm not an expert in this field, but libpayload might be something worth looking into, for finding out what might have caused coreboot to not work as well as before on this board.
For now I can embed and boot some floppy images using coreboot, thanks to Mike Banon's patches, They appear to be stable, just that USB doesn't work. Other than that there's little I could achieve as I need to boot from USB to install an OS and I haven't gotten myself to successfully boot anything from USB yet (either freeze or with error messages, probably because of the aforementioned libpayload issues).
some patches for branch 4.11 are worked on to fix some AM3 board (https://github.com/agg1/coreboot/commits/master), which may be a decent alternative to KGPE-D16 and a viable option for a devs who couldn't afford a corebooted workstation for thousands of bucks
my budget, and even if i could afford a KGPE-D16 i wouldn't want it but instead industry including FOSS people should align their resource requirements so development can be done on commodity hardware such as AM3 boards or quadcore laptops with 16GiGs of RAM with acceptable electricity costs for 24/7 full load operation, which makes me wonder again why up until recently a dual core with 2GB RAM was the only option officially tested with AM3-generation commodity hardware. same holds true for the chaotic nature of documentation available (BIOS, ACPI, chipsets, UEFI) which requires almost a livetime of work to attach some peripherals, initialize memory, etc. it is a nightmare.
One major issue with these workstation/server AMD boards would be the availability of cooling solutions. Nowadays C32/G34 cooling solutions are very hard to find (although someone managed to get some others installed by hard-modding the clips). In comparison, cooling solutions for Intel sockets (115x, 1366, 2011, etc.) and mainstream AMD sockets (AM2/AM3, AM4, FM1, etc.) are much more common. It'd certainly be better if mainstream AMD motherboards of that era could be better supported, I'm looking forward to it.
Total power consumption with the corebooted AM3 board with a Phenom 2 X6 of mine is far below any of the server/workstation boards: ~140Watts total power consumption under full load (monitor, 4 hard disks in RAID, 16GiG ECC RAM DDR3 400MHz @1.35V seem rock solid and i don't see the benefit of memory I/O at 10GB/s@666MHz instead of 8GB/s@400MHz) Even if modern systems or the KGPE-D16 in question were slightly faster i had rarely seen a workstation system under full load which had met the power and efficiency target! Soon i'll test a Phenom2 X6 95Watt CPU (instead of the typical 125W) which may bring down consumption another 10 or 20 Watts, under constant full load that is. Too i am planning to introduce an option for easy undervolting with coreboot, fixing Northbridge frequency at 1.8GHz maybe with reduced voltage and fixing CPU at 1.25Volts which should be stable with all fam10h CPUs in non-TurboCore performance state so no specific low-power CPUs had to be acquired, and the power management chaos may simply be removed entirely, which was the part that contributed to breaking AM3 fam10h revE support, and it is a pain elsewhere too i might add. s2disk is functional still and system can be completely powered off and restored with it, with encrypted swap space even. Furthermore such an AM3 system can easily be cooled and remains almost dead silent, simply choose a 5Volt adapter with appropriate 120mm ventilators - it is a bliss, if you had ever enjoyed the noise of a workstation/server nearby. Sadly fam15h support on AM3 allthough present on branch 4.11 couldn't be confirmed, at least the FX 8120 CPU i had tested on the AM3 board with coreboot didn't POST at all.
Other than that i am maintaining a self-made hardened livedvd distribution booting read-only ISO only targetting these "obsolete" x86-64 systems in conjunction with coreboot which has removed certain "features" in kernel and userspace entirely and it is being stabilized after 3 years of development and testing. Since fam10h CPUs lack accelerated AES encryption for example the distro of mine ships with a crypto stack inspired by Telco scramblers or MIL-STD-188 this easily yields almost 10GiB/s of randomness and integrated with block layer encrypted disk i/o can be had at 1GiB/s per cpu core max instead, maybe with NILFS2 on top of it to be tested still, without any AES hardware acceleration required anymore. so much for the achievable performance and reliability.
Unrelated but since libpayload was mentioned i'll keep an eye on that one too for sure, i had encounted some minor issues recently which i couldn't track down yet.
Community attempts to bring the Heads's KGPE-D16 board config to coreboot 4.11 here. Any help welcome.
To all KGPE-D16 lovers/supporters: KGPE-D16 is supported under Heads on coreboot 4.11 with TPM support. ( @pietrushnic @mikebdp2 @miczyg1 @pkubaj @alexmaloteaux @ullbeking @Th3Fanbus @paulmenzel @agg1 @lss4 and others silently following up)
3 boards added (read boards configurations for more details)
That was merged under https://github.com/osresearch/heads/commit/1661e5dcb07a061317581187b15ae553ad8fc3df, thanks to @Tonux599 !
Now! What is the future for such boards? Critics say that the board is too power hungry, others that microcode updates are not existing anymore, some supporters have turned their heads away, some have sold their boards....)
I am a strong believer that some power users would find their way back into that board
How to get involvement? Someone has longer arms then me to reach out to the proper communities, have pointers to get FSF attention for suppport?
Thank you for bringing up this issue (I just stumbled over it). I am one of these "complaining" users who has been running this board in production for several years now and figured out a way to live / workaround around the aforementioned issues. Personally, I still find the board unique due to the arguments presented by @tlaurion, and I find its power consumption (at least with the vendor bios) acceptable for a workstation/server (however, not acceptable when running with coreboot and two cpu packages).
Imho, if 3mdeb is willing to take on this work and just needs funding, this is already a good starting point. Yet, collecting 30k is probably too much if you consider the remaining KGPE-D16 owners. But if the 30k would be split into separate goals it might be possible to collect enough money to reach at least some of them.
So, what could be the goals? Probably (some) of these:
@miczyg1 Have you considered starting a campaign on CrowdSupply or a similar platform? It's hard to guess how many people would be willing to back such a campaign and of course there is risk to loose the effort invested in the campaign, but imho it's worth a try.
@kuleszdl If your on the OSFW Slack join #kgpe-d16 where there is discussion on a possible hackathon being organised by 3mdeb. (I can also add you to #kgpe-d16-private where the bulk of the discussion occurred before opting for a public channel)
@Tonux599 I don't use slack and usually I prefer discussing such issues more in-depth over public issues (such as here) or on mailing lists. Yet, I am very pleased to hear about the planned hackathon which sounds like a very promising idea!
I am Duncan Townsend, CTO of Immunefi. We love what you're doing. Immunefi would like to pledge your choice of 17,750 USDC (approx 15,000 EUR) or 10 Ether (approx 22,700 EUR) towards the completion of this work. You can view the pledged funds at 0xD24F38eCe7084F8bAB4BEc23cbcC9a50566Bc38C or at kgped16pledge.immunefi.eth.
Would you like for us to raise awareness of this need throughout the crypto community? We believe this is something that everybody in crypto should be using and want to make sure this happens. Let us know how we can cross-promote and raise awareness.
Please send an email to team@immunefi.com if further coordination is required.
Hi @duncancmt, I'm Piotr Król, Founder and CEO of 3mdeb Embedded Systems Consulting, which initially tried to gather founds through NLNet.
This pledge very generous from your side. Thank you very much.
Rising awareness by respected member of crypto community would definitely will be helpful, we can also use 3mdeb marketing channels to bring enough founds to start this project. With this pledge we think we are very close to deliver something meaningful.
I think other participants here, especially @tlaurion, and members on kgpe-d16 channel of OSFW Slack can also help in promotion.
@duncancmt @tlaurion
Hello,
Below some notes from our internal meeting:
As far as we understand, the main goals for this project are:
RoT in bootblock
Platform usage via the BMC
Upstream?
Maintenance
RoT in bootblock
Platform usage via the BMC
Upstream or not?
In the case of the upstream, we need to implement all of the missing features that the coreboot 4.14 requires and try to push the board back upstream. When not upstreaming, we might implement only the features we need (e.g. C_ENVIRONMENT_BOOTBLOCK) on the 4.11 branch.
Maintenance?
We would like to set up an external meeting to discuss the above and the remaining items. The main goal of the meeting would be to confirm the minimum required scope of the work to make sure that we can reach your goals.
Proposed meeting dates:
Please let me know if any of the proposed dates are fine for you.
@pietrushnic has proposed joining my flashrom write-protect bounty to this... which I'm happy to do.
The selected meetin date is: Monday 23rd August 15:00 UTC The meeting will take place on: https://meet.3mdeb.com/kgpe-d16-refresh
Anyone interested in this project is invited to join.
Report from today's meeting below:
Slides: kgpe-d16-refresh.pdf
Notes from the discussion:
There were 12 participants. Thank you all for joining this call. The lits below:
sasha is using OpenBMC on the board
Duncan:
FSF: they probably use it as server
Thierry: used is as a build machine - accessing through the BMC
Summary:
Just to let you know, that we plan to present some next steps in the next week.
Note that there's also KCMA-D8 board, that is pretty much weaker KGPE-D16. There are also some people using it, and it would probably be quite easy to get it working (if you want to push support for KGPE-D16 anyway). This board was also supported previously along with KGPE-D16.
@pkubaj That might be a good item to add to the tasks list. Thanks!
Let's schedule the 2nd meeting at: Friday 17th September 16:00 UTC at https://meet.3mdeb.com/kgpe-d16-refresh
Slides for today's meeting attached. I will post the meeting minutes after the meting as well. kgpe-d16-refresh-v2.pdf
Report from Friday's meeting below.
Slides:
Notes from the discussion:
There were 9 (? sorry if I have missed anyone - I have not dumped the whole list) participants. Thank you all for joining this call. The list is below:
critial phases
(coreboot refresh for the KGPE-D16 board
and Firmware Root of Trust
)KCMA-D8
is not a priority for Immunefi - we would rather go with the
maintenance releases for the KGPE-D16
if there are funds left
KCMA-D8
workKCMA-D8
support - after the
KGPE-D16
will be contributed as part of this project, it should be far
easier to doimmunefi.3mdeb
on KeybaseHello. The proposed schedule is attached. From our side, we are ready to start with this exciting project!
Just wanted to mention that I worked on some raminit cleanups for AMD K8 and AMD FAM10 in 2017. Looking at the coreboot history, it appears 3 or so relevant commits from librecore were never upstreamed into coreboot. Feel free to reuse this code as it may save you a bit of work refactoring the ".c" includes unless someone has already fixed this again in coreboot: https://github.com/librecore-org/librecore/compare/41d5d16f65ddb2377016d2ba6ec0d89a6b67468c...67f0495709dbe3ce422256109f49e1e4375a1b71
Thanks for these references. We will definitely take a look at that once we get to this phase. We have decided that we take the approach as suggested by the coreboot community. That we start with the plain upstream coreboot and start picking the relevant KGPE-D16 bits to have each stage working correctly, one be one. We do not yet have a functional bootblock with serial port output, but we are getting there...
C bootblock is already there: https://twitter.com/Dasharo_com/status/1452929075153145859?s=20
I also suggest you look at baseline coreboot commits that have "Tested-by: Raptor Engineering Automated Test Stand
@zamaudio can you elaborate how this would help? I'm not sure I'm getting your point. Do you mean we should try their hardware configuration?
Hi @pietrushnic, what I meant was: if you are looking for a baseline version to base the ram code off, make sure you choose a commit that was boot tested on actual hw otherwise you may introduce a bug that will be hard to find and may have been already fixed in a different coreboot commit.
I was able to boot the latest commit of 4.11_branch
at the time (last year) on KGPE-D16 with a select few RAM combinations. Just that I wasn't able to find a working RAM combination for KCMA-D8.
I think the current state of the 4.11_branch
should still be relevant as there are very few commits directly relevant to our boards, nor did I see anything useful backported to it.
For now I think addressing the memory initialization issues should be the most important at the moment, as it prevents the boards from booting reliably. Other issues (such as USB and system stability) could follow once the board can be reliably booted.
I use two KGPE-D16's, one is my workstation, another is my server. Both are running coreboot 4.11.
Both are booting reliably and I can't really understand why people say those boards are unstable with coreboot. The initial setup was quite a pain, but after that, they are VERY stable and boot reliably 99% of times (the times where they don't boot, they instantly reboot after loading kernel, but it's quite rare).
I use two KGPE-D16's, one is my workstation, another is my server. Both are running coreboot 4.11.
Both are booting reliably and I can't really understand why people say those boards are unstable with coreboot. The initial setup was quite a pain, but after that, they are VERY stable and boot reliably 99% of times (the times where they don't boot, they instantly reboot after loading kernel, but it's quite rare).
It depends on the RAM installed, and the hardware people installed on the board, as well as what people use coreboot for (Heads or not).
In my experience USB is mostly broken with coreboot. I can't properly boot from USB sticks (for installing a Linux distro), and have to use PS/2 input devices because USB ones don't work once booted into something that has its own USB stack. It also prevented nvramcui
from running because it couldn't initialize "EHCI periodic frame table". Booting nvramcui
without any USB devices attached would report different errors that ultimately led me to a malloc
failure (which I think is the root cause of the aforementioned USB errors). As I didn't have an OS nor optical drives installed at that time, the USB issues blocked me from testing any further.
While some RAM combinations are near 100% bootable, the Memtest86+ I built into it reported wrong SPD values and detected some unexpected errors then eventually froze with a glitched screen. With less RAM installed (<32GB) it would not detect errors but the test still freezes after a while.
I don't use Heads at the moment, just coreboot in overall. Maybe coreboot+Heads can work reliably once configured properly, but coreboot as it currently is now is not going to be useful as simply getting it to boot on the board is no trivial task for many.
EDIT: Need to check if these commits mentioned previously can be applied to 4.11_branch
.
EDIT 2: Nope... It can't be cherry-picked as-is at the moment.
Ugh, that's really strange. I use USB input devices just fine and can boot from USB just fine (but I don't use Linux, maybe that's why). The coreboot-provided Memtest86+ does report errors right after starting, but AFAIK this is known issue on this board and if you use Memtest86+ booted from another source, it's apparently fine.
I also don't use Heads, since it only supports Linux.
Ugh, that's really strange. I use USB input devices just fine and can boot from USB just fine (but I don't use Linux, maybe that's why). The coreboot-provided Memtest86+ does report errors right after starting, but AFAIK this is known issue on this board and if you use Memtest86+ booted from another source, it's apparently fine.
I also don't use Heads, since it only supports Linux.
Something else to mention.
I don't know about your use case though, and I'm not really sure about the status of USB. For me it definitely was problematic.
Hi @pietrushnic, what I meant was: if you are looking for a baseline version to base the ram code off, make sure you choose a commit that was boot tested on actual hw otherwise you may introduce a bug that will be hard to find and may have been already fixed in a different coreboot commit.
Yes, this is fair point. I believe @miczyg1 can say more about his approach to selecting correct code base that will be bring up upstream. There is some limit to number of memory configurations we can test, but we will get to call for beta testers at some point I believe.
I also don't use Heads, since it only supports Linux.
@pkubaj petitiboot is able to kexec FreeBSD so why not Heads :)
petitboot is able to kexec FreeBSD kernel, but this still has some issues.
@macpijan cc
@pkubaj those are interesting insights, but it seem so far there was no commercial interest in fixing above issues. We are little bit off-topic here, but maybe 3mdeb at some point could fix those issues.
@pietrushnic The Talos board I have at home was bought (along with the CPU and some RAM) by FreeBSD Foundation for me to work on FreeBSD Ports on POWER. I explicitly requested a grant from them and it got approved. It took quite some time to get it, but I THINK it would be possible for you to request a grant to work on FreeBSD on POWER (kernel / OS side). It's another thing whether that gets approved, though. But I'd be happy if there were more OS devs for POWER on FreeBSD, since while on the ports side we're pretty good (Firefox, Libreoffice etc.), we're pretty much behind in terms of OS / kernel (no DRI, no bhyve, no loader and performance is quite behind Linux - even my KGPE-D16 can compile things faster than Talos on FreeBSD). Anyway, the point was that you MAY be able to get some sponsorship from FreeBSD Foundation regarding POWER.
@pkubaj thank you, we will look into sponsorship of FreeBSD Foundation.
Request for anyone interested in using KGPE-D16 with write-protection of flash chip: if your chip is not already listed in https://github.com/flashrom/flashrom/issues/185, please post a comment there stating chip's model.
In terms of the October deliverables:
IMM1.6 - KGPE-D16 firmware shall support resource allocator v4
task - we are still working on that and plan to move it to Phase 4 for November insteadWe plan the Phase 2 + Phase 3 release for this week.
@macpijan do you think it make sense to ask Libreboot community for testing?
hi. leah here. i'm currently in the process of building myself a d16 again, precisely because of your efforts. i will be available for testing. Tashtari in libreboot IRC also has D16 hardware and is quite enthusiastic about your work.
When your work is ready, I wish to integrate it in lbmk, which is libreboot's build system. It can download coreboot, applying any number of custom patches desired. lbmk currently uses standard coreboot 4.11
edit: iirc Bardon also has D16 hardware. also, the FSF has a testing station.
also:
FSF deploys all their hosting on a D16, but has a special D16 test stand. An entire spare D16 just for testing configurations on. Contact the sysadmin team at the FSF. Ian Kelling (iank on #fsfsys IRC) is very interested in your work, and is a current sysadmin at the FSF.
Will buy a tpm2 module and can test myself Heads for tpm1 and tpm2 measured boot configuration support, for both workstation board with AMD graphics and BMC for server board. PS: Would love to see the u-bmc needed work being a stretch funding goal? @pietrushnic: How much would that u-bmc for ast2050 chip additional work cost?
I will also try my best to bring a u-root + Heads board under Heads (being coreboot based as opposed to linuxboot based), encompassing the cpu toolset (@rminnich: amazing work!!! https://github.com/linuxboot/book/blob/master/cpu/README.md#the-u-root-cpu-command ), since I intend to use that board as my local build system and use those 32 cores and 64gb of ram collecting dust.... And generate heat in the cold weather here while doing so.
The kgpe-d16 in current state works already as a Qubes 4.1 platform, while some pushing would be needed to have network-server project cascade forwarding of external traffic to specific qubes(@rudd-o) coming from hidden services down to their actual offered service provided in services qubes, as well as being able to provide some yunohost or equivalent services locally. That's where I was in the past, and where I personally want this to go forward.
I will reenable that machine that sleeps for way too long.
do you think it make sense to ask Libreboot community for testing?
It makes perfect sense, but maybe more after a fourth phase (somewhere early December) when we can actually boot something.
How much would that u-bmc for ast2050 chip additional work cost?
I haven't really used u-bmc, but if it also needs U-Boot and Linux for AST2050 (which I assume it does) then most of the work between OpenBMC and u-bmc would be common. We have presented our view on that in this presentation: https://github.com/osresearch/heads/issues/719#issuecomment-921877456 (starting with slide 13).
I guess you could also try to use the existing kernel, but you would be stuck with the 12 years old kernel running on your BMC then.
FYI v0.1.0 was released: https://docs.dasharo.com/variants/asus_kgpe_d16/releases/
Please note you can join newsletter to get notifications.
Please let us know what we can improve.
Are there plans to push this work to upstream or will it be maintained as a fork?
@pkubaj please take a look at mileston 7. In short we definitely plan to upstream all code that would be accepted by upstream, but let's be honest timeline of that is unknown.
A heads up: Over in the mainline they're looking at possibly dropping quite a large number of AMD platforms (fam 14h, 15h, 16h?) if they can't move over to the newer resource allocater (though for 15h there may be some early PoC?). It looks like this work will need to support the new resource allocator in order to be merged.
The allocator v4 was in scope of this project and is being worked on already. There are still a couple of more features required in this last deprecation call (like the PARALLEL_MP) which were not part of the initial plan. We will do our best to implement it as part of the upstream effort, though.
Is there a separate issue somewhere for LEGACY_SMP_INIT removal/PARALLEL_MP? I've been looking into it a bit as it relates to fam14-16 deprecation.
@awokd we all have to be professional in light of what is going on mailing list 1 2. We tried to present 3mdeb position at coreboot leadership. We will work closely with community to avoid what was proposed. We also consider various backup plans, if you are intrested in discussing those please join Dasharo Matrix workspace
There was a proposition to do required work under refused grant application. If there is interest from third party to fund conjointly needed work, board could be maintained and reupstreamed, under u-bmc, coreboot and under Heads.
Edit:
Proposition from 3mdeb here
History here
~Possible funding path here~
Trail of 4.11 reported problems: https://github.com/osresearch/heads/issues/719#issuecomment-995879031
Last updates, newsletter, code source drop : https://github.com/osresearch/heads/issues/719#issuecomment-966729133