Windows Dev Kit 2023 Boot Work

[nickdepinet]

Discussion started in Issue #7, splitting to it's own issue for tracking:

I've recently gotten ahold of a windows dev kit 2023, which appears to be a surface pro 9 (5g) / sq3 in a box, and I'm attempting to boot linux on it with these patches.

Progress so far -

Booting base Ubuntu 22.04.1 LTS arm image - Hangs at: Booting a command list

Synchronous Exception at 0x0000000C1B17927C Building a "default" profile image with the image tool - Hangs at: Loading initramfs... Booting into Surface kernel... EFI stub: Booting Linux Kernel... EFI stub: Using DTB from configuration table EFI stub: Exiting boot services... Barring better suggestions, I'm going to try to extract the firmware with the script from aarch64-firmware and stuff it into the arch image above and see where that gets booting

---

1) could possibly be fixed by https://github.com/linux-surface/grub/commit/55b95a388414a3d850e24fd2ae8fac100ffd9319. So you could try to use e.g. the pre-built grub from https://github.com/linux-surface/grub-image-aarch64/releases/tag/fedora-37-2 with your Ubuntu kernel. Alternatively, just use the image from 2) which contains the patched grub, but remove the devicetree option from the boot entry. That should get you booting in ACPI mode.

2) is way before any firmware gets loaded, so I doubt that that will make any difference. Can you make sure that efi=novamap is set (press e in the grub menu to edit)? Also, this is booting with the SPX DT, and I'm not sure if that's a good idea. So as I've said above, you can remove the devicetree option in the grub config and it should hopefully get you a bit further.

You can also experiment with earlycon=efifb.

In the long run, you'll probably want to have a look at the DT of the sc8280xp (that is 8cx Gen 3; it's also used in the Thinkpad X13s) and use that as a base. Also, you might want to join the oftc/#aarch64-laptops IRC. There are a lot more experienced people who also might have some ideas.

Originally posted by @qzed in https://github.com/linux-surface/surface-pro-x/issues/7#issuecomment-1306998491

[tilator]

I actually created a script to extract these files ...

I did not know this. I have to give it a try. Do you know how long it takes to read the partition and extract those files?

[jglathe]

I actually created a script to extract these files ...

I did not know this. I have to give it a try. Do you know how long it takes to read the partition and extract those files?

Seconds.

jglathe@volterra:~$ sudo fetch_sc8280xp_fw.sh 
Found the Windows installation on /dev/nvme0n1p3.
Contents of /lib/firmware/qcom/sc8280xp/MICROSOFT/DEVKIT23/:
total 19720
-rwxr-xr-x 1 root root      534 Feb 18 23:16 adspr.jsn
-rwxr-xr-x 1 root root      731 Feb 18 23:16 adspua.jsn
-rwxr-xr-x 1 root root      537 Feb 18 23:16 battmgr.jsn
-rwxr-xr-x 1 root root      534 Feb 18 23:16 cdspr.jsn
-rwxr-xr-x 1 root root 14545260 Feb 18 23:16 qcadsp8280.mbn
-rwxr-xr-x 1 root root  3571712 Feb 18 23:16 qccdsp8280.mbn
-rwxr-xr-x 1 root root    14392 Feb 18 23:16 qcdxkmsuc8280.mbn
-rwxr-xr-x 1 root root  2035748 Feb 18 23:16 qcvss8280.mbn
Updating initramfs
update-initramfs: Generating /boot/initrd.img-6.8.0-rc4+
update-initramfs: Generating /boot/initrd.img-6.8.0-rc3+
update-initramfs: Generating /boot/initrd.img-6.7.3+
update-initramfs: Generating /boot/initrd.img-6.7.1+

[JeromeDeBretagne]

The Volterra box is using the same mainboard.

Is it really?

So maybe using the dts from Volterra might be a better fit.

I tried that at first too, but I couldn't get it to boot on the SP9 5G (I was getting reboots IIRC) hence my take at creating a dedicated minimal dts, inspired from existing sc8280xp dts files. I will have a look at it again.

regarding the screen, I would assume it needs some firmware files. all 8280.mbn files are device specific (signing), need to be extracted/copied from the Windows partition.

I was hoping to reuse most of the X13s firmware files as they are officially in linux-firmware ; it works somehow for Wi-Fi at least to start with.

I'll see if it makes a difference for the screen with the extracted firmware files from the Windows drive.

I actually created a script to extract these file

Thanks for the pointer!

[jglathe]

I have had a look at the dedicated dts. I would suggest to go from the Volterra (Blackrock) file, but not ignore the X13s file either. Your dts doesn't describe the internal display. The description from the X13s is a good start (mdss0_dp2). Volterra uses mdss0_dp3, this could also work, but needs edp as the driver (and the definition of the panel bus, like in the X13s definition). Keyboard, Touchpad, Touchscreen I would suggest to copy from X13s for a start. PCIe buses, too - you have all of them, pcie2a, pcie2b (nvme in there), pcie3a, pcie3b (WWAN in there), pcie4 (WLAN in there). The WLAN drivers are a little challenge: Linux-firmware has (sorta current) drivers, but there is no board definition. Since its the same mainboard, it shoud be the one from volterra (needs to be verified, though). The board file to use is obtainable from my repo, I will also put it in my syshacks repo to make it easier to find. The newest firmware for the WCN6855 is here. There are some other specialties, though. Like /etc/initramfs-tools/modules needs a list of drivers that are required. The sc8280xp special firmware files need to be in the initramfs. mhi drivers need to be built as modules. Some grub specifics. bringing up Bluetooth (pesky bugger, that). Setting useful MAC addresses. Most of it is described in the kernel repo, but still quite distributed. That (among other things) is why I created a bootable image that starts the ubiquity installer and does a bootstrap dance with fetching the firmware from the windows partition, updating initramfs with it. I'm running one of the volterra boxes directly from this (external ssd via USB), and it is very usable (internal nvme is even faster). But I need to make the time to do a new image, 'cause its a tedious and still manual process. And you get Ubuntu 23.10, no real choice in that. Optimal starting point would be a variant for SP9 5G with its own dts and amended scripts. I can't really test it, though.

[BarrelCauldron]

I have had a look at the dedicated dts. I would suggest to go from the Volterra (Blackrock) file, but not ignore the X13s file either. Your dts doesn't describe the internal display. The description from the X13s is a good start (mdss0_dp2). Volterra uses mdss0_dp3, this could also work, but needs edp as the driver (and the definition of the panel bus, like in the X13s definition). Keyboard, Touchpad, Touchscreen I would suggest to copy from X13s for a start. PCIe buses, too - you have all of them, pcie2a, pcie2b (nvme in there), pcie3a, pcie3b (WWAN in there), pcie4 (WLAN in there). The WLAN drivers are a little challenge: Linux-firmware has (sorta current) drivers, but there is no board definition. Since its the same mainboard, it shoud be the one from volterra (needs to be verified, though). The board file to use is obtainable from my repo, I will also put it in my syshacks repo to make it easier to find. The newest firmware for the WCN6855 is here. There are some other specialties, though. Like /etc/initramfs-tools/modules needs a list of drivers that are required. The sc8280xp special firmware files need to be in the initramfs. mhi drivers need to be built as modules. Some grub specifics. bringing up Bluetooth (pesky bugger, that). Setting useful MAC addresses. Most of it is described in the kernel repo, but still quite distributed. That (among other things) is why I created a bootable image that starts the ubiquity installer and does a bootstrap dance with fetching the firmware from the windows partition, updating initramfs with it. I'm running one of the volterra boxes directly from this (external ssd via USB), and it is very usable (internal nvme is even faster). But I need to make the time to do a new image, 'cause its a tedious and still manual process. And you get Ubuntu 23.10, no real choice in that. Optimal starting point would be a variant for SP9 5G with its own dts and amended scripts. I can't really test it, though.

Thank you for your work on this! I’ve been running portainer on my dev kit without any issues/freezes, it’s so stable :)

[shvetsnikita]

@JeromeDeBretagne

It's a bit annoying as I have to add the devicetree line upon each boot, I still need to better understand how Debian's Grub supports devicetree files, where to put them on the filesystem, etc.

AFAIK it doesn't, as this patch was never merged. But Ubuntu ship GRUB2 packages with this patch, so what I do is just take 10_linux from Ubuntu package, put it to my Debian install, and then hold grub2 packages. I know it's a bit barbaric, but if anyone have better suggestions - please post it here.

[JeromeDeBretagne]

I'll see if it makes a difference for the screen with the extracted firmware files from the Windows drive.

Slow progress on the Surface Pro 9 5G, I've been able to copy the firmware files from the Windows drive, I've put them in the right locations then in the initramfs, I can see the firmware files loading properly now in the boot logs so this is going in the right direction (using a recent Volterra / Blackrock dtb)!

However, I didn't get any display output yet, neither from the built-in display nor from the 2 USB-C ports. I need to play more and adapt the devicetree using a mix of the X13s and Volterra / Blackrock .dts files, that's not easy to guess without an actual board description...

[tilator]

How about SSH?

[jglathe]

How about SSH?

Fighting with my new image, not sure who's winning. Update tomorrow.

[JeromeDeBretagne]

How about SSH?

SSH is working fine, over Wi-Fi or Ethernet, since my first message. That's how I am doing my follow up investigations and improvements on my Debian install running on the NVMe drive.

[tilator]

How about SSH?

SSH is working fine, over Wi-Fi or Ethernet, since my first message. That's how I am doing my follow up investigations and improvements on my Debian install running on the NVMe drive.

Is it just off by default? The problem is, I don't have any kind of control to the device after booting. No screen, no keyboard (I assume because I can't see if it works or not), no SSH access either.

Ethernet is up and running, but no means to access the device.

What do I do wrong or what should I do other way to get SSH access?

[jglathe]

How about SSH?

Fighting with my new image, not sure who's winning. Update tomorrow.

Update on this: The image won, I'm respawning from the (working) 6.6.0 image and do my mods there - again. Can't get it to do my bidding for unknown reasons, something systemd /snap related I'm afraid.

• extend root partition suddenly doesn't work anymore
• snap is really... unfriendly but comes up eventually
• (worst) this buggered user setup (ubiquity) decides to crash and you have to kinda do parts of it again
• sshd thinks it doesn't have to talk to me

All the firmware / driver setup works nicely, but what use is it without the GUI you're putting up with. So back to the drawing board I guess.

[jglathe]

Is it just off by default? The problem is, I don't have any kind of control to the device after booting. No screen, no keyboard (I assume because I can't see if it works or not), no SSH access either.

Ethernet is up and running, but no means to access the device.

What do I do wrong or what should I do other way to get SSH access?

@JeromeDeBretagne uses a different installation AFAICT. With the Ubuntu image, yes sshd is actually not only off, you have to install it. And you need a user that is already set up.

[tilator]

yes sshd is actually not only off, you have to install it. And you need a user that is already set up.

Which is all impossible without SSH or any other kind on UI.

[jglathe]

Which is all impossible without SSH or any other kind on UI.

yep. me wrestling with a new image and cursing.

[JeromeDeBretagne]

@JeromeDeBretagne uses a different installation AFAICT. With the Ubuntu image, yes sshd is actually not only off, you have to install it. And you need a user that is already set up.

Indeed, I'm using the official Debian netinst image(s) with my custom .dtb file so far, the built-in screen works during the whole installation. Then I boot the SP9 5G from the installed system and I can ssh into it (as I still don't get any screen working yet) as it keeps the Wi-Fi network parameters from the installation phase.

[jglathe]

Argh. This debugging work continues (found a few, still some to go). But, @tilator you can edit the image and do the installations you might need yourself.

What you need: a Raspberry pi 4 or some other aarch64 device with a console or tty. I have used a pi 4b for the initial images, worked nicely. And was a good fallback for the next try. An external SSD whith USB (c) to nvme or SATA adapter. This one will be written a few times. SSD is way faster than USB sticks, in my experience.

The easiest way would be to just burn the image to the SSD, mount it on the pi4, do the editing. Usually you need a console that is chroot'ed into the file system, and one that copies over things from outside. If you want to do the edits in the image file, though, and burn it later, this is the way to do it:

loopf="$(sudo losetup -f)"
sudo losetup "$loopf" ./Ubuntu_Desktop_23.10_wdk2023_6.6.0.img 
sudo partprobe "$loopf"
sudo mount "$loopf"p2 /mnt/
sudo mount "$loopf"p1 /mnt/boot/efi/
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /mnt/$i; done 

This mounts the image file via loop and all its partitions. You can do a sudo chroot /mnt/ and do almost everything in there that doesn't require running services. apt works, snap not really. Installing ssh should be no problem, compromising security by setting a root password either. You can also do update-initramfs and update-grub. When you're done, you're exiting the file system and need to properly unmount it:

for i in /mnt/dev /mnt/dev/pts /mnt/proc /mnt/sys /mnt/run; do sudo umount -l $i; done
sudo umount "$loopf"p1
sudo umount "$loopf"p2
sudo losetup -d "$loopf"

The image can be written to an USB stick (16GB minimum) or SSD and is bootable. For writing to the device I use the disks utility. You can also do this on an x86 machine but you need binfmt and qemu for this.

[jglathe]

The new image for Kernel 6.7.1, Ubuntu 23.10 can be found here. It's the "secure" image, no root pw changed, no sshd installed. It installs pretty smoothly, fetches the sc8280xp firmwares from the Windows partition, extends the rootfs, has cmdline option to see which firmware is loaded, has current WLAN firmware and a board file good for the WDK module, and uses bootmac to generate constant mac addresses. A suggested hack for the SP9 5G will follow. But not today.

[JeromeDeBretagne]

Quick update on my side, I have finally been able to get display working on the SP9 5G, re-using a mix of .dts / .dtb files. Display output is working for external screens plugged into one of the 2 USB-C ports at the moment, the bottom one to be specific.

Compared to my past attempts, I have realized that the 4 .jsn files were missing next to the .mbn firmware files extracted from the Windows partition (they were not in the same folder) so I've fixed it today and that's certainly what was missing previously.

Let's the fun continue!

[jglathe]

Display output is working for external screens plugged into one of the 2 USB-C ports at the moment, the bottom one to be specific.

Yay :fireworks: Is it only the bottom one? Curious, I have seen a dependency here on the WDK, too. It's on the list. And I guess that would be USB-C 0. Do you have a display with an USB-C connector, or do you use an adapter?

the 4 .jsn files were missing next to the .mbn firmware files extracted from the Windows partition

I copied them because they looked interesting. These json files define service endpoints and qmi_instance_id for apparrently the user space side of things, what is running on those co-processors. The packages are alsa-ucm-conf protection-domain-mapper qrtr-tools Sound doesn't work to the DP devices yet. Those co-processors are part of what makes the USB-C multiplexer tick, though, so maybe they were not starting up properly without it.

Can you please push your dts so that I can use it as base for further tweaks? I would suggest to use a separate branch for this. According to Gus' repo the code name would be Arcata, so maybe use that name for the dts too?

[tilator]

Argh. This debugging work continues (found a few, still some to go). But, @tilator you can edit the image and do the installations you might need yourself.

Thanks. Unfortunately this Surface is the only proper ARM64 device I have at the moment. I have an old Nvidia Shield TV too, but it's too hard to use it.

I did try to edit some files by just mounting the installation stick to an X64 PC with Ubuntu, but it did not work. So I'm still without any UI.

[tilator]

Quick update on my side, I have finally been able to get display working on the SP9 5G, re-using a mix of .dts / .dtb files. Display output is working for external screens plugged into one of the 2 USB-C ports at the moment, the bottom one to be specific.

Sounds good. Is it somewhere available as a disk image to give it a try?

[jglathe]

@tilator that was quite a challenge, but its working now:

This is after the box boots up (twice) and leaving it untouched, trying to login from outside.

Image is compressing now, will upload soon.

[jglathe]

@tilator @JeromeDeBretagne There it is, Image with open root access and ssh is available. Password: FsecuritY!

Would be good to get dmesg for a start.

[tilator]

Here you have a dmesg. dmesg.txt

It seems to fail loading FW from Windows. Bitlocker is not on. I did double check it.

[tilator]

BTW: SSH is accessible while it boots up first time. After automatic reboot there is no SSH because network does not come up again.

It negotiates TCP address while booting first time, but not at the second time. I did try it with two different USB-RJ45 adapters and two different routers.

The dmesg was taken after first boot before it rebooted itself.

[jglathe]

Hmm the reboot is intended, its after extend-rootfs and firmware fetcher. hmm. Thanks for the dmesg, I'll take a look

[tilator]

Hmm the reboot is intended, its after extend-rootfs and firmware fetcher. hmm. Thanks for the dmesg, I'll take a look

Would it be any sense in disabling root fs extending to find out if it causes this network trouble? Second try would be disabling FW operation.

Making it boot properly with network and SSH might help us a lot at this point.

[jglathe]

First analysis: This really looks like the same mainboard. dmesg looks normal for a boot without firmware files... the r8152 messages look like from the internal USB adapter (strange as this is). Do you have an additional dock connector on this? Can't see any messages from any other USB ethernet adapters, though. The WLAN driver is good news: It's the same board as in the Windows Dev Kit, so the loaded board file should give you some pretty useful connectivity over WLAN and Bluetooth. @JeromeDeBretagne this could help you get better connectivity, drivers are in the wdk2023_syshacks repo.

Since I assume you just tried to "turn it off and on again" and didn't get further, what I could do:

• disable extend-rootfs
• disable reboot after fetching the firmwares.

The firmwares would be loaded at next reboot, but you could set up WiFi connection via command line, no prob. This will be remembered afair.

I will do a new image with these two tweaks, but the process takes at least an hour.

[tilator]

Would it be good idea to put the missing firmware files in the filesystem before trying to boot it at all? So that it can load those files properly at the first place. This might cut many errors making it boot further.

[jglathe]

With the new config this is something you get at second boot anyway. Won't publish these files, they are not mine. New image is compressing, upload soon.

[tilator]

the r8152 messages look like from the internal USB adapter (strange as this is). Do you have an additional dock connector on this?

This device has a combined connector for power supply and some kind of addition/dock. They seem to call it "Surface connector". Maybe the message comes from it.

But the USB-RJ45 adapter I used might have Realtek chip in it too. It's an other possibility.

[jglathe]

Well the surface dock has a separate driver. I guess not included yet...

[jglathe]

File is available here. Since this will take a while: Open a new discussion or issue in my repo? If you want to grow the rootfs, its this: systemctl start growroot-almost.servicehere I'll take a look at the surface dock driver, saw an option in the kernel config.

[tilator]

Thanks. It's up and running now.

Is it normal, that according to dmesg it does try to find the FW files from

/lib/firmware/...

but according to find -iname 'qccdsp8280.mbn' only copy seems to be in

/usr/lib/firmware/...

I did not dare to reboot it yet and we have to continue tomorrow anyways.

Well - I rebooted it now. No SSH any more and I suppose it's the missing network again. I don't have time now to evaluate it more. I can continue tomorrow.

[jglathe]

Oh that's good news. The firmware for WDK2023, like for the SP9 5G, is not in mainline yet, and may never be. I used the /usr/lib/firmware/updates/ path intentionally to ensure no collisions. It is checked before /lib/firmware/. Do try to reboot, I'm curious 😈

[tilator]

I'll open a new discussion now to leave this for DEVKIT_2023 as it is supposed to be.

[JeromeDeBretagne]

Can you please push your dts so that I can use it as base for further tweaks?

For the Microsoft Surface Pro 9 5G update, here you go! I've pushed my latest .dts and .dtb files in this commit : arm64: dts: qcom: sc8280xp-sp9-5G: Enable external display

I have also cleaned up the .dts file in the previous commit to remove all the entries that don't apply/work on the Surface Pro 9 5G so I consider it the minimum .dts / .dtb files to boot and install Linux on the NVMe drive through Wi-Fi (or a USB Ethernet adapter) with external display support (working only on the bottom USB-C port). It's working fine using recent Debian netinst images based on Linux 6.7 and it is perfectly stable, yeah!

I have used the following path for the firmware files "qcom/sc8280xp/MICROSOFT/SurfacePro9/" by the way.

Sorry for the lag, I've spent a lot of time (way too much in fact) trying to get external display working on the second (top) USB-C port too, without success so far despite trying mdss0_mdp2, mdss0_dp3, etc. I wonder if the first "connector@0" section in the .dts file could apply to the Surface Connect port instead (as it seems based on USB using a different connector), but I don't have the right accessory to check this assumption.

If that's the case, getting the second (top) USB-C port to work would require to find the correct GPIOs driving the USB SBU (side band use pins) to create a new usbX-sbu-mux section I imagine. However I have no clue how to find that out :-( @merckhung @andersson @jhovold do you have any suggestions, either on Linux (I've tried gpiod, gpiomon, acpidumps, etc.) or through reverse engineering from the Windows installation? Thanks a lot if you have any tips you can share!