search

linux-surface / aarch64-arch-mkimg

Scripts to create Arch Linux AArch64 disk images (e.g. for flashing to a USB stick), adapted for the Surface Pro X
MIT License
15 stars 7 forks source link

ci: add build image #5

Closed denysvitali closed 1 year ago

denysvitali commented 1 year ago

This PR adds the ability to build the image using the CI.

Unfortuantely the CI fails, but I think there is an issue with the linux-surface Arch package repository:

debug: linux-firmware-msft-surface-pro-x-20220509.b19cbdc-1-any.pkg.tar.zst.sig: maxsize 16384
debug: linux-firmware-msft-surface-pro-x-20220509.b19cbdc-1-any.pkg.tar.zst.sig: opened tempfile for download: /var/cache/pacman/pkg/linux-firmware-msft-surface-pro-x-20220509.b19cbdc-1-any.pkg.tar.zst.sig.part (wb)
debug: linux-firmware-msft-surface-pro-x-20220509.b19cbdc-1-any.pkg.tar.zst.sig: curl returned result 0 from transfer
debug: linux-firmware-msft-surface-pro-x-20220509.b19cbdc-1-any.pkg.tar.zst.sig: response code 200
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst: url is https://pkg.surfacelinux.com/arch-aarch64/linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst: maxsize 85096454
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst: opened tempfile for download: /var/cache/pacman/pkg/linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.part (wb)
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst: curl returned result 0 from transfer
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst: response code 200
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig: url is https://pkg.surfacelinux.com/arch-aarch64/linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig: maxsize 16384
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig: opened tempfile for download: /var/cache/pacman/pkg/linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig.part (wb)
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig: curl returned result 18 from transfer
error: failed retrieving file 'linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig' from pkg.surfacelinux.com : 
debug: linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig: no more servers to retry
debug: curl_download_internal return code is -1
warning: failed to retrieve some files
error: failed to commit transaction (download library error)
 linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any downloading...

(see https://github.com/denysvitali/surface-pro-x-arch-linux-aarch64-images/actions/runs/4548499674/jobs/8019594944)

qzed commented 1 year ago

Thanks! I'm not really sure what went wrong there. Our repos essentially redirect to GitHub releases, so maybe it's an issue on GitHub's end? I was able to successfully run the script just now.

That being said, there was an issue with the latest kernel packages (should be unrelated), but those should be fixed now.

Can you try and restart the action?

Otherwise, this looks good to me. I'm a bit concerned about the size (the final image is ~1GB), but from what I've read that should still be okay.

denysvitali commented 1 year ago

Just tried to restart the GH Actions pipeline. Sadly it leads to the same result :(

qzed commented 1 year ago

I have honestly no idea what's going wrong here... I'll merge it into a branch to test it here.

denysvitali commented 1 year ago

I have the same issue locally though. Maybe I can try to debug w/ tcpdump (or run tcpdump on the CI itself).

qzed commented 1 year ago

Something is a bit off with GitHub actions today...

qzed commented 1 year ago

It seems to have worked this time around: https://github.com/linux-surface/aarch64-arch-mkimg/actions/runs/4555577441. What I changed:

I somewhat doubt that any of that should really have an impact on downloading files via curl/pacman.

For some reason, it still looks like it's running but all jobs have completed... so I guess GitHub actions is slightly broken today. I'll check if it behaves better tomorrow and will merge this then, if it does.

qzed commented 1 year ago

Also: Thank you very much for your support!

denysvitali commented 1 year ago

I'm puzzled on why it didn't work 😅

qzed commented 1 year ago

Me too.

It somehow looks like some transport issue (error 18 apparently is "transfer closed with outstanding read data remaining"), but the thing is that we don't host the files in our repo ourselves, so unless the redirect is going wrong there's nothing I can check on our end. Instead, we essentially redirect to GitHub releases, so pacman should directly download those from GitHub.

So I can only guess that it's some kind of GitHub infrastructure problem. The way actions behave right now I guess that might not be too far off.

I mostly tried the qemu action thing because I thought maybe something weird is going on there, but that was really just a shot in the dark. Again, I don't think that this really fixes it and it might be more or less random. So I'll try without that again when things are more normal.

denysvitali commented 1 year ago

Oh: https://twitter.com/alexellisuk/status/1641091515194679296

This explains it, maybe

qzed commented 1 year ago

Actions seem to be working again but it fails with the same issue. Setting up QEMU via the Docker action seems to work somehow... still absolutely no clue why. I'll push that change to this PR, wait for it to build and then merge it if it's successful.

qzed commented 1 year ago

And it failed again... with the QEMU setup change, so I guess it's random?

qzed commented 1 year ago

I've now also tried building the image via the docker container (https://github.com/linux-surface/aarch64-arch-mkimg/commit/4398651955e8d3e1b83c208e389a0218ef1a8b0c). This also seems to fail or succeed randomly. I'm really not sure how to properly debug this...

F9Alejandro commented 1 year ago

Looks like the biggest issue is with archlinux for arm mirror repos taking forever to actually load. Unfortunately using DisableDownloadTimeout doesn't seem to work properly. Possibly using "pacman --disable-download-timeout" could work?

qzed commented 1 year ago

The official Arch Linux ARM repos aren't the issue. It randomly failed with

Retrieving packages...
error: failed retrieving file 'linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any.pkg.tar.zst.sig' from pkg.surfacelinux.com : 
 linux-firmware-msft-surface-pro-x-20220509.b19cbdc-1-any downloading...
 linux-firmware-msft-surface-pro-x-qcom-20220509.b19cbdc-1-any downloading...
warning: failed to retrieve some files
error: failed to commit transaction (download library error)
Errors occurred, no packages were upgraded.

That package stems from our repo, which in turn redirects to GitHub for the actual package downloads, so it really isn't clear to my why this sometimes fails. In addition, I haven't been able to reproduce this locally, making this even more difficult to debug. Hopefully with the new firmware package, things will work more reliably.

F9Alejandro commented 1 year ago

huh for me it is mainly just issue after issue with alarm repo mirrors completely cutting out where I am.

qzed commented 1 year ago

I've made some changes on top of this PR (setting up QEMU and building the image via Docker) in the feature/ci branch and I've been re-running the CI action a couple of times. Unfortunately, it still fails fairly randomly. And with different files, e.g. the latest failure was:

error: failed retrieving file 'linux-surface-6.3.5-1-aarch64.pkg.tar.zst.sig' from pkg.surfacelinux.com : 
 linux-firmware-msft-surface-pro-x-20230515.51290942-1-any downloading...
 linux-firmware-msft-surface-pro-x-qcom-20230515.51290942-1-any downloading...
warning: failed to retrieve some files
error: failed to commit transaction (download library error)
 linux-surface-6.3.5-1-aarch64 downloading...
Errors occurred, no packages were upgraded.

I'll give your mirror a try on the off-chance that it somehow is an issue with the official mirrors.

If that doesn't work, I'll probably merge that branch anyways and set it to only build on certain tags. That way it doesn't fail constantly and I can re-run it manually until it does succeed. Having a pre-built image is probably worth that.

F9Alejandro commented 1 year ago

Looks like the build passed and there is a .img file in the artifacts section.

qzed commented 1 year ago

I still don't trust it though. These failures are random, so I'll try rebuilding the exact same commit a couple more times over the next days. We'll see after that.

F9Alejandro commented 1 year ago

I still don't trust it though. These failures are random, so I'll try rebuilding the exact same commit a couple more times over the next days. We'll see after that.

By chance do you have a discord? It would be easier to message on there then on here. I am having some issues with missing some packages, and wifi not working properly. (Got it working was using WPA2/WPA3 with AES instead of WPA/WPA2 with TKIP & AES)

qzed commented 1 year ago

By chance do you have a discord?

We have a Matrix space (https://matrix.to/#/#linux-surface:matrix.org) and an IRC channel (libera.chat/#linux-surface) for stuff like this.

qzed commented 1 year ago

And I'm not sure how well wifi works yet, in particular there are issues like https://github.com/linux-surface/surface-pro-x/issues/3 (e.g. some channels that are valid here in Germany but disallowed in the US won't work). I suspect there could be more issues similar to this or issues applying only to specific setups.

qzed commented 1 year ago

I still don't trust it though. These failures are random, so I'll try rebuilding the exact same commit a couple more times over the next days. We'll see after that.

Attempt 4 failed just now. Similar download library error again.

F9Alejandro commented 1 year ago

I still don't trust it though. These failures are random, so I'll try rebuilding the exact same commit a couple more times over the next days. We'll see after that.

Attempt 4 failed just now. Similar download library error again.

Just took a look at the logs and noticed it seems to be an issue with a stable download of surfacelinux packages. Could possibly need a more reliable host or have it go through through the package manager of github.

qzed commented 1 year ago

Server-side, our repo is essentially a just script that redirects to github releases. So any package archives being downloaded are actually being downloaded directly from github servers. That's why I can't really debug this well, since I have no insight to those.

denysvitali commented 1 year ago

Are we maybe rate-limited by GitHub?

Should we try to install one package, sleep and then install the other twos?

qzed commented 1 year ago

I somehow think that would manifest differently, but we might as well try.

qzed commented 1 year ago

So I've added a retry mechanism with some backoff-timeout in https://github.com/linux-surface/aarch64-arch-mkimg/commit/0ed1b0a6a94aec3951b485ebcd1e6da84fb59fd1. I'm hoping that that is a good enough workaround, but I haven't been able to trigger that mechanism to test it yet.

qzed commented 1 year ago

Okay, it triggered just now and it looks like it's working. I'll clean up that branch over the next days and then merge it.

F9Alejandro commented 1 year ago

I will also work on migrating over to a better dedicated host for the mirror hosting. Also when trying to use my mirror it has one extra portion instead of the old https://mirror.fangshdow.trade/ it is now https://mirror.fangshdow.trade/archlinuxarm/ I am hosting a few other projects that need to be mirrored based on availability.

qzed commented 1 year ago

Thanks @denysvitali and others for your help. I've merged this (with extensions) in #9.

qzed commented 1 year ago

Images will now be built on each tag release and published as proper release (see e.g. https://github.com/linux-surface/aarch64-arch-mkimg/releases/tag/v0.1.3 in a couple of minutes).