petefoth
commented
10 months ago @bananer I'd be interested to know your thoughts on this if you have time
Closed petefoth closed 7 months ago
petefoth
commented
10 months ago @bananer I'd be interested to know your thoughts on this if you have time
petefoth
commented
10 months ago Sat 02 Dec 23
st3rox
commented
10 months ago Looks like this now only leaves the current build and the last previous major version build So there's no way to roll back a month
petefoth
commented
10 months ago Looks like this now only leaves the current build and the last previous major version build So there's no way to roll back a month
It should leave the two most recently built versions, which would normally be the same version but previous month. I have vague memories of wrongly triggering a build of an earlier major version for one device, but I cant remember which.
st3rox
commented
10 months ago Looking at bramble, the only builds available are this month's build and the 19.1 build from 12/16/2022
Same for enchilada
petefoth
commented
10 months ago OK
It looks like there's a bug in the cleanup code which happens here that doesn't do the right thing when a device is moved to the next Android version. Not sure whether the fix is in that sell code or in the clean-up.py Python code.
It's not new: in previous build runs we will have had 2 x 20.0 builds and 1 x 19.1 build. It will happen a lot in this build run, as a lot of devices have moved from 19.1 to 20.0 :(
I don't have the time (or the skills) to start debugging either. Short term I will do the following:
petefoth
commented
10 months ago On the build server, for those devices which
have 17.1, 18.1 or 19.1 builds still present
I have deleted the 1?.1 files, so that after this month's build we will have the two most recent 20.0 builds. On completion of the build for each device, the device zips directory on the build server will be rsynced to the download server
This applies to the following devices:
h990 haydn heart hotdog hotdogb instantnoodle instantnoodlep joan kane kebab kiev lake lancelot lemonade lemonadep lemonades lisa lmi ls997 lynx m5 m5_tab m52xq marlin mars martini mata merlinx messi Mi439 Mi8917 Mi8937 miatoll nairo nash nio nx563j nx606j nx609j nx611j nx619j nx651j nx659j ocean odroidc4 odroidc4_tab onclite oriole panther payton pdx214 pdx215 PL2 polaris pro1 pro1x pstar pyxis racer radxa0 radxa0_tab raven redfin renoir rhode river rosemary rs988 sagit sailfish sake sargo Spacewar starlte star2lte sunfish surya sweet taimen tangorpro tissot TP1803 troika ursa us996 us996d us997 vela violet vs995 wade walleye X00TD xmsirius xz2c Z01R z2_plus zippo
Next for the devices which were built for 19.1 last month, which should all be built for 20.0 this month (discovery,gauguin,gemini,kirin,liber,mermaid,natrium,parker,pioneer,vayu,voyager). When this month's build completes, we will have the new 20.0 build and last month's 19.1 build which is correct.
That leaves the following problems to solve:
akari alioth apollon bardockpro beckham berlin berlna beryllium beyond0lte beyond1lte beyond2lte beyondx blueline bonito bramble channel cheeseburger cheryl chiron coral crosshatch crownlte d1 d2s d2x davinci devon dipper dre DRG dubai dumpling enchilada equuleus evert f62 fajita flame FP3 grus gta4xl gta4xlwifi gts4lv gts4lvwifi guacamole guacamoleb h830 h850 h870 h872 h910 h918 h932) I will think about 1, and maybe ask other s for ideas on 2.
@st3rox Thanks for spotting this. It's not a new problem, but it's more urgent with only keeping two builds instead of three
petefoth
commented
10 months ago More investigation:
the shell code here
we should be keeping DELETE_OLD_ZIPS (currently 2, previously 3) builds of the current lineage version plus 1 version of previous versions. It seems that, if old versions are present ,we are keeping DELETE_OLD_ZIPS -1 (currently 1, previously 2) builds of the latest.
Time to stop now before I do any more damage :)
petefoth
commented
10 months ago Damn! Removing the old builds didn't fix the problem for h990: no we only have today's build, lineage-20.0-20231207-microG-h990. Last month's lineage-20.0-20231207-microG-h990 is gone!
st3rox
commented
10 months ago
- In the meantime, my actions in the previous post mean that we no longer have any builds of older lineage versions. I don't know how much of a problem that really is, and whether there is any need to keep old versions lying around (certainly LineageOS don't do it)
I didn't realize lineageOS doesn't keep old versions
I see no problem with getting rid of previous versions to keep storage usage down
For future version increases my suggestion would be keeping the last previous version build for some time until the current version is daily driver quality
ctag
commented
10 months ago I'd like to echo the other offers to help buy more space. Or offer space on my home storage server.
I just got bit by an update that breaks calling, and no clear way to access the next previous update file.
st3rox
commented
10 months ago I'd like to echo the other offers to help buy more space. Or offer space on my home storage server.
I just got bit by an update that breaks calling, and no clear way to access the next previous update file.
If your device has a/b partitions you can switch to the other partition to get back to your previous install
ctag
commented
10 months ago Thank you for the troubleshooting advice, st3rox. When I change the active slot the system fails to boot with a warning about corruption. If there's a user step to enable a/b booting I must have not set it up.
For the topic of having older builds available, would a solution like the link below be permissible to this project's hosting provider? I only downloaded one phone's listing as an example, but I could keep the last handful of builds available for all the phones.
If the main repo kept the old sha files too, then users could confirm what they download from me is correct. Those sha files are small, but I understand that keeping more of them is still the opposite of freeing up space..
https://mirror.berocs.com/download.lineage.microg.org/guacamole/
petefoth
commented
9 months ago I'd like to echo the other offers to help buy more space. Or offer space on my home storage server.
For the topic of having older builds available, would a solution like the link below be permissible to this project's hosting provider?
Thanks for these offers, but I don't think that's the way to go. We definitely have enough storage to keep the two most recent builds, and I think that will be enough. The fact that, for some devices we now only have one build is down to a bug either in the code that keeps a specified number of builds - previously three, changed to two for this build run - or in my understanding of that code. I hope to resolve that before the next build run. From then on, two builds should be available for each device, and I don;t think we need any more than that
petefoth
commented
9 months ago The right thing is happening with the 18.1 builds: before this build run, two builds - the October & November builds - were available for the 18.1 devices; after this month's build we still have two builds -the November & December builds
vanMacG
commented
9 months ago The right thing is happening with the 18.1 builds: before this build run, two builds - the October & November builds - were available for the 18.1 devices; after this month's build we still have two builds -the November & December builds
Not for every device. klte i.e. has only december builds now...
petefoth
commented
9 months ago Not for every device.
kltei.e. has only december builds now...
Good spot!
Looking at the logs, I think we had a 17.1 build for that device. So it looks like the 'bug' is in the handling of devices where a build of a previous Android / LOS version was present. Probably not helped by my removing all the 17.1 builds I could find - see my earlier comments
vanMacG
commented
9 months ago I don't have proof for this (scrrenshots), but as far as I remember, there was no 17.1 build before. There was the current monthly 18.1 build including recovery and boot image and corresponding checksums. Additionally there was the previous monthly 18.1. image and corresponding checksum (no recovery and boot). For me, this is fine. I always keep the previous image simply on the smartphone... So no blame ❤️
petefoth
commented
9 months ago At some point before the January build run, I will check how many / which devices now have two builds present and which have only one.
I don't see the point of keeping builds for old Android versions once a device has been successfully 'promoted` to a higher Android version (except for devices which were only promoted in the most recent build run.
Ideally we should have two builds for each device available. These will usually be the latest Android version. For recently promoted devices, we will have the most recent build which is the new version, and the previous build of the previous version. So users of those devices have a month to check that the new Android version is working OK before the old Android version disappears.
petefoth
commented
8 months ago See Tidy up old 17.1, 18.1 & 19.1 builds on both servers #525
After the January build run we have 263 zip files for 230 separate devices, so only 30 or our build targets have two builds as they should
df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/vdb1 590G 249G 311G 45% /mnt/archivedevices a5y17lte - debx have two builds, ether - zl1 have only one. Look at the build log - nothing obvious (and I've tidied up the 18.1 device logs 😄 )
May be a consequence of the cleanup though I don't think so.
I'm going to do another 18.1 build run with DELETE_OLD_ZIPS (and LOGS) set to 3 instead of 2, and see what happens.
If there's no change then ????
vanMacG
commented
8 months ago I'm going to do another 18.1 build run with DELETE_OLD_ZIPS (and LOGS) set to 3 instead of 2, and see what happens.
With the re-run it is again like I wrote in a comment above:
I don't have proof for this (scrrenshots), but as far as I remember, there was no 17.1 build before. There was the current monthly 18.1 build including recovery and boot image and corresponding checksums. Additionally there was the previous monthly 18.1. image and corresponding checksum (no recovery and boot).
petefoth
commented
8 months ago I'm going to do another 18.1 build run with DELETE_OLD_ZIPS (and LOGS) set to 3 instead of 2, and see what happens.
With the re-run it is again like I wrote in a comment above:
I don't have proof for this (screenshots), but as far as I remember, there was no 17.1 build before. There was the current monthly 18.1 build including recovery and boot image and corresponding checksums. Additionally there was the previous monthly 18.1. image and corresponding checksum (no recovery and boot).
Thanks. This is caused by our post-build.sh which rsyncs the zip directories on the build and download servers: as currently coded, that is removing from both servers any files that are not present on the other. As I deleted the .img and .sha256sum files on the one of the servers in my initial tidy-up, they are missing from both :)
I'll go and learn a bit more about how rsync works :smile
petefoth
commented
8 months ago OK I've made some changes to the post-build.sh (removing the --delete and --delete-excluded args from the call to rsync) which seem to have the desired effect. Test build onklteactivexx resulted in two zips, and the full set of .img and .sha2456sum files. (In the test build DELETE_OLD_ZIPS is set to 3, so we have two complete builds (20240121 & 20240122), and the partial 20240118 build (no .img and their sha256sum files).
For the February build run will keep this change, and keep DELETE_OLD_ZIPS at 2. This should give leave us with two complete builds per device. We can see how much space that takes up on the download server, and make a decision whether to go back to 3 builds in the March build run.
petefoth
commented
8 months ago All the 20.0 devices built so far in the February 24 build run now have 2 builds present :)
petefoth
commented
7 months ago All the 20.0 devices built so far in the February 24 build run now have 2 builds present :)
Except for devices which were successfully built for the first time (radxa0, odroidc4, m5)
To ensure we keep two builds available for these devices (and any devices which get built for the first time in future build runs, we should change out rsync logic so the --delete and --delete-excluded args are only set if there is only a single build present in the zips directory.
ArchangeGabriel
commented
7 months ago There is another exception, pdx203 and pdx206 that had been disabled since July were enabled again during last month, and rebuild for the first time in February. They now have three images: https://download.lineage.microg.org/pdx203/ and https://download.lineage.microg.org/pdx206/.
petefoth
commented
7 months ago OK the idea of keeping only two builds, and deleting builds for devices which are no longer supported is too simplistic. Please see new issue 'Build retention strategy #573' for a strategy which is more flexible, and more friendly for devices which are no longer officially supported by LOS
I'd be very interested in your comments there. (I'll close this issue now)
petefoth
commented
7 months ago The free disk space numbers after the February build run, with two builds stored per device
Filesystem Size Used Avail Use% Mounted on
/dev/vdb1 590G 491G 70G 88% /mnt/archiveThe figures include the new archive folder containing the final builds for devices which are no longer supported by LOS (see Build retention strategy #573 ), which takes up a total of 24G. Even if we removed the archive, we would still not have space to keep three builds per device rather than two. This is mainly due to the fact that we now keep several img files for each device build
We are currently using 99% of the available disk space on the download server (554GB used, 6.2GB free, out of 590GB). This caused a couple of problems with the latest build run, where copying of some files from build server to download server failed due to lack of disk space (see #505 and had to be completed manually after some unneeded files were removed.
The lack of space is party due to the fact we now publish loads of .img files in addition to the main ROM zip files and recovery images - see #405 & #467.
Zipping up these images (see #483) caused the 'Update recovery` function of the Updater to fail (see #487), so was reverted - See PR #490.
Adding more disk space is problematic for a number of reasons. So I plan to reduce the number of builds we make avaiable from three for each device to two (by setting
DELETE_OLD_ZIPS=2in thedocker runcommand). Unless there are any strong objections, I will do this for the next build run which starts early Friday morning. If anyone has any concerns about this, please let me know before then.