search

lineageos4microg / docker-lineage-cicd

Docker microservice for LineageOS Continuous Integration and Continous Deployment
https://hub.docker.com/r/lineageos4microg/docker-lineage-cicd
GNU General Public License v3.0
502 stars 196 forks source link

Have the builds stopped? #418

Closed khao-soi closed 1 year ago

khao-soi commented 1 year ago

It looks like the current image builds stopped after DRG a few days ago: https://download.lineage.microg.org/?sort=time&order=desc

D3SOX commented 1 year ago

Yup, noticed the same. Wasn't sure where the right place to report this was.

khao-soi commented 1 year ago

Yup, noticed the same. Wasn't sure where the right place to report this was.

I'm not sure if this is the right place, either. Let's find out :)

D3SOX commented 1 year ago

I think @bananer has access to the server. Can you check and start a new round if needed?

Carlos-Rocha commented 1 year ago

I wonder if this channel is monitored

mddvul22 commented 1 year ago

I don't have a lot of money, but in the past I've offered to make a monetary donation to help improve the build environment/process. That offer still stands. I'm guessing that others might also be willing.

ultr41337h4xor commented 1 year ago

Missed 2 updates for Lineage already, because I'm waiting to do a dirty install of this... Does this happen often here? I read somewhere that updates are twice a month, but I'm guessing the builds made this way = OTA updates after install?

khao-soi commented 1 year ago

I read somewhere that updates are twice a month

With the addition of LOS 20 this has been reduced to once a month, since the build server takes more than 14 days to create all images. More here: https://github.com/lineageos4microg/docker-lineage-cicd/issues/335#issuecomment-1436082195

I guess we have to be a little patient and wait till the next round starts.

Revival8697 commented 1 year ago

I like how if the build process is interrupted, they just ignore all unbuild process and wait for the next cycle. 🫥

khao-soi commented 1 year ago

Maybe this project needs another volunteer or two. I have no experience with android builds but I know some docker, if that helps.

ivangal2 commented 1 year ago

I am also volunteering, know some docker too..

ivangal2 commented 1 year ago

So I believe we should wait if there is a next round of builds in the beginning of June. If there is no next round or it ends the same way as the last round, we probably should start thinking what to do with the project. Bananer is probably not active in the project any more (I wrote him email 3 weeks ago, no response). What do you guys think, what are our options?

khao-soi commented 1 year ago

I've emailed bananer as well. No response yet. Is there anyone else with access to CI/CD pipelines?

khao-soi commented 1 year ago

@Exodusnick @spolack Maybe you guys can help?

bananer commented 1 year ago

I've been busy with other parts of my life. Builds should restart tomorrow.

Yeah, I could use some help. Thanks to everybody who volunteered already. There is a lot of trust involved with accessing the build server and I'm not sure how to vet people. Suggestions welcome...

khao-soi commented 1 year ago

I'm not sure how to vet people

How about a short get-to-know video call? I would have no problem in revealing some infos about me to you, to see if it could make sense. I've emailed you past friday. Just reply to that, if you think thats a reasonable idea.

khao-soi commented 1 year ago

Builds are running.

ivangal2 commented 1 year ago

I was thinking to provide us with the build logs via ELK stack (I can help with setting this up). And make it either public or accessible to a couple of trusted users, so we can investigate problems by ourselves if you are busy. Unfortunately, it does not solve the problem of fixing the actual builds if they got broken, but at least, whoever is interested, can check what is going on. Because like this time, builds got broken, no one responded for 3 weeks and no one knew what was going on, if the project died, etc.

And the problem of fixing the builds may possibly be solved if the build scripts are published to GIT and if they cease to work, we can check the Kibana and anyone can create a pull request to fix it. And admins can just approve/merge it then.

A bit of brain storming, the general idea is to spread the responsibility among more people/everyone so the whole infrastructure does not depend on a single person.

petefoth commented 1 year ago

I'd be happy to help checking on build progress and / or debugging failed builds. Also happy to undergo whatever vetting process you think is appropriate :)

LuccoJ commented 1 year ago

Builds are running.

If they stop like last time around the letter D, though... I mean, bananer hasn't shown up in a month...

levinuss commented 1 year ago

I mean, bananer hasn't shown up in a month...

You have seen https://github.com/lineageos4microg/docker-lineage-cicd/issues/418#issuecomment-1570116855 right? :)

StuartJMackintosh commented 1 year ago

I'm not sure how to vet people. Suggestions welcome...

I offer to help with this. Drop me a DM and we can have a chat.

jlHertel commented 1 year ago

Also happy to provide help. Feel free to DM me and we can chat

JQuilty commented 1 year ago

Also volunteering to help.

ivangal2 commented 1 year ago

Builds are running fine, we are on letter 'e' now, further than the last time, so far so good.

Great that there are so many volunteers among us. We can start to brainstorm how to utilize the volunteers to help the project become more resilient. Does anyone have an experience with managing volunteers on an open source project?

JQuilty commented 1 year ago

So, I can confirm at least that the LOS20 builds for gtaxl4wifi have been built and flash just fine. I don't know how much further it got, but up to that point I can confirm it's building properly.

D3SOX commented 1 year ago

Seems like guacamole was skipped. Can someone check why?

FintasticMan commented 1 year ago

According to the messages in the Matrix server, the build for guacamole failed.

StuartJMackintosh commented 1 year ago

Lemonadep downloaded today and upgrade seems to work so far 👍

Thank you !

bstnbuck commented 1 year ago

Seems like the current run (only 19.1?) contains failures. On several builds, only the recovery and boot images have been updated. The OS image, on the other hand, has not. Please check for 19.1 gauguin as soon as possible. The last update was on 16.04. (2 months ago!) and I need the update, because the previous one contains errors.

Other examples are kirin or h850.

Update

It seems that gauguin and almost all 19.1 models from h850 onwards have -not- received an update. Run for 18.1 models has started. Here all seem to run through.

khao-soi commented 1 year ago

I don't think anyone else has gotten access, yet. So its up to @bananer

awfm9 commented 1 year ago

It seems that builds for sunfish are quite behind. I really rely on this project to avoid Google services. Any chance to get an updated build? Couldn't really get the past one to work, either. Maybe because of the version mismatch between recovery and image.

khao-soi commented 1 year ago

Any chance to get an updated build?

You can always build it yourself. The build process is documented in the readme and the LOS20 specifications can be found in #335.

In general the situation at the moment is not really satisfiying. Some builds fail and we don't get any info for weeks, nor can anyone do anything about it. We really need more maintainers.

LuccoJ commented 1 year ago

I'm looking at https://matrix.to/#/#microg-lineage-os-builds:matrix.domainepublic.net and it doesn't seem like the sunfish build has failed, but rather, it hasn't started at all. Last time it appears is April 14. I don't think it's queued for running, either, as it appears all queued builds have ended since there's nothing being built now after "zl1".

bstnbuck commented 1 year ago

@LuccoJ Did the process for gauguin fail or skip?

LuccoJ commented 1 year ago

@bstnbuck respectfully I am not a Matrix-relaying bot. You can click on the link I gave and easily reach the Matrix room whose history is public and accessible by anyone.

bstnbuck commented 1 year ago

Pretty much all missing images (19.1, 18.1) seem to be skipped (at least no fail). I support the suggestion for more maintainers.

vanMacG commented 1 year ago

New image for 18.1 klte was built and works fine 👍

LuccoJ commented 1 year ago

@vanMacG that's correct, and it's Thursday 15th's news: I point this out because if one looked at the builds bot on Matrix like I've indicated, it's clear that it first built LOS 20 images, then LOS 19.1, then LOS 18.

It's absolutely possible that some were skipped entirely, and I think I've seen an example or two (not sure what they were now), but, spoilers! Completed build for gauguin (Sun, Jun 11). It definitely didn't just blanket skip all 19.1 and 18.1 images.

petefoth commented 1 year ago

@LuccoJ wrote

looking at https://matrix.to/#/#microg-lineage-os-builds:matrix.domainepublic.net and it doesn't seem like the sunfish build has failed, but rather, it hasn't started at all.

Comparing the output there for the build runs, we see on Fri, Apr 14 2023

Completed build for redfin
Starting build for river
Failed build for river
Starting build for sagit
Completed build for sagit
Starting build for sunfish
Completed build for sunfish
Starting build for surya
Completed build for surya

On Fri, Jun 9 2023 we see

Starting build for redfin
Completed build for redfin
Starting build for renoir
Completed build for renoir
Starting build for river
Completed build for river
Starting build for sagit
Completed build for sagit
Starting build for sake
Completed build for sake
Starting build for starlte
Completed build for starlte
Starting build for star2lte
Completed build for star2lte
Starting build for surya
Completed build for surya

It looks like renoir, sake and starlte were added to the list of devices to build 20.0 for. Looking in the https://download.lineage.microg.org/ directory, renoir and sake previously had 19.1 builds, starlte seems to be a new device.

Is it possible that, when the list of devices to build for was modified, sunfish was deleted (accidentally or deliberately)?

Time to go hunting for the list of devices to build, and it's commit history

An alternative thought is that the Failed build for river on April 14th affected what happened afterwards. I don;t yet know how the builds are performed, or how the list of devices to build for is managed Time to go digging :)

FintasticMan commented 1 year ago

The list of devices to build is taken from the LineageOS hudson automatically AFAIK, from which, sunfish hasn't been removed recently.

The builds should be done just with the scripts in this repo, each LineageOS version one after the other.

petefoth commented 1 year ago

Thanks for that. So I guess we would need to look at the detailed logs from Hudson (or whichever build tool the lineageos4microg uses) to find out why the build for sunfish wasn't started in the latest build run. I guess those are not publicly available.

In this comment to an old issue from 2021, we see a list of devices to build for. The final comment from @bananer says:

I would like to move the scripts to a publicly visible repo in the future.

I guess the relevant repo is not yet publicly visible. If it were, it would allow others to help debug build failures (and support @bananer in other ways)

khao-soi commented 1 year ago

Making the build logs publically available would be a good start. I don't think there is anything sensitve in it. Let's hope we'll get some response soon.

awfm9 commented 1 year ago

The list of devices to build is taken from the LineageOS hudson automatically AFAIK, from which, sunfish hasn't been removed recently.

It has been excluded from the build on two occasions now, so it doesn't seem like a random occurrence to me.

You can always build it yourself. The build process is documented in the readme and the LOS20 specifications can be found in #335.

Could someone provide complete instructions on how to produce a build that is 1:1 the same as the release I can download, please? It's not obvious from the readme, especially with regards to which add-ons to include and how, where to put the extra manifest files, what parameters to use to fully mirror it.

petefoth commented 1 year ago

how to produce a build that is 1:1 the same as the release I can download,

Not possible. The official builds will be signed with the official LOS signing keys. You don't have access to those keys. You can either generate your own keys, and sign your build with those, or sign the build with test keys, or not sign the build at all. Whichever option you choose, your build will differ from the released builds that you can download .

Apart from that, pretty much everthing you need do know is documented in the README: https://github.com/lineageos4microg/docker-lineage-cicd#readme

awfm9 commented 1 year ago

Not possible. The official builds will be signed with the official LOS signing keys. You don't have access to those keys. You can either generate your own keys, and sign your build with those, or sign the build with test keys, or not sign the build at all. Whichever option you choose, your build will differ from the released builds that you can download .

As I have already expressed in my previous reply, it is not obvious from this README how to include all the add-ons and parameters that are in the downloadable release. Proper step by step instructions with full commands would be much appreciated.

giovEra commented 1 year ago

Not possible. The official builds will be signed with the official LOS signing keys. You don't have access to those keys. You can either generate your own keys, and sign your build with those, or sign the build with test keys, or not sign the build at all. Whichever option you choose, your build will differ from the released builds that you can download .

As I have already expressed in my previous reply, it is not obvious from this README how to include all the add-ons and parameters that are in the downloadable release. Proper step by step instructions with full commands would be much appreciated.

About "how to include all the add-ons and parameters that are [used]", see here.

Obviously, the values of some parameters need to be adjusted (e.g., paths)

cashflys commented 1 year ago

For hotdogb the directory date was updated, but the contents did not change from April. It appears the build process is dead for this device even though it is still supported on LineageOS proper. I agree this project isn't dead, but it is not currently being developed. I think the autopilot is just getting lucky sometimes. I truly hope this changes but so far not looking good for this particular device. I'm going to also look into building locally. However, I am guessing the build server scripts are using the same processes described in the README. If the build server once worked, and is now failing on my device, there is likely something that changed, and as I am not a developer, I doubt I will be able to fix that.

I believe that might be at the heart of the issue, which this thread morphed into. No new builds, OK... I'll build my own, someone please tell me how.... Well, it is more complicated than just following the published tutorials, as unless I am very much in error, if the scripts as designed were working, the build server should be working. If it was a minor fix, then it would likely have already happened (several iterations of lineageOS have occurred since the build for hotdog broke). Therefore even if I was able to put together a functional build environment, I would likely have to start testing with versions prior to the official source failing. Let's assume I can, however, when I move on and it fails, what then? Maybe a developer that has real skills can do this, pick a failed build from Lineage_mG and go build the last working one, then try the failed release. If they can't figure it out, it is very unlikely referring non-Devs to the readme for constructing a roll-your-own of MicroG will succeed for anyone, regardless of Docker skill levels as this is build debugging and well.... I only wish I was that bright.

C

Pulled from the June 5th Matrix:

_{ "content": { "body": "Failed build for hotdogb", "msgtype": "m.text" }, "origin_server_ts": 1685989110268, "room_id": "!kKoGaPHPftqoYWDDCu:matrix.domainepublic.net", "sender": "@lineage-microg-bot:matrix.domainepublic.net", "type": "m.room.message", "unsigned": {}, "event_id": "$mWOaO49Tc3Ea9IohnLG10RO8kCiCqUxcr5_lvp6CBc0", "user_id": "@lineage-microg-bot:matrix.domainepublic.net" }__

petefoth commented 1 year ago

I have had a look at the June build failures. The builds failed for the following devices

  1. deadpool, wade, m5: these are all TV boxes, and failed with the same error relating to a missing file in the partner_gms repo: raised #436
  2. guacomole, guacomoleb, hotdog, hotdogb: all failing with the same error in file device/oneplus/sm8150-common/sepolicy/private/camera_helper_app.te
  3. sunfish, Taimen, TP1803, walleye : all failed with missing file errors relating the same file hardware/google/pixel/lineage_health/device.mk Interesting
  4. TP1803 different failure also related to the (new?) health HAL

Re: 3 & 4 - it is interesting a: that they are close together in the list of build targets, and b:that there was a change in LOS which introduced the health HAL, and was made after the build run had started, but before the builds on sunfish, Taimen, TP1803, walleye started. So for these devices, 3 & 4 we can wait until the next build run (starting tomorrow) and see if the builds succeed

Re: 2. Again they are close together in the list of build targets: maybe some changes were made upstream that we hadn't picked when the builds for those devices started. Again, we'll wait and see what happens in tomorrow's build run

Raised #436 to track the TV box failures

Raised #435 to track the others

petefoth commented 1 year ago

The June build failures are being addressed: see #435 and #436

LaPeste commented 1 year ago

I can confirm that this time the build for hotdogb has successfully completed. Thank you all for the great work! 🎉🎉