Build retention strategy

[petefoth]

1. We don't have enough disk space to retain 3 builds for every device that is officially supported by LOS
2. We should aim to keep the two most recent builds for officially supported devices
3. When a device is 'promoted' from one Android version to a higher Android version (e.g. 20.0->21.0) we should keep the last build of the lower Android version, alongside the two most recent builds of the higher version, until the higher Android version is deemed to be stable. Definition of 'stable' to be decided :)
4. It would be good to keep the last build of devices which are no longer supported by LOS for as long as we can without running out of space. So, when a device 'drops off` the list of supported devices we should remove all but the mst recent build for rhat device
5. Many older builds - for 18.1 & 20.0 devices which have been 'dropped' by LOS (and 19.1 builds for one device monet which was dropped by LOS before a 20.1 build was made - are still present on the build server, even though they have been removed from the download server.
6. If space runs out on the current download server, builds for 'dropped' devices could be stored in an alternative location. This alternative - and 'archove server'? - does not need to provide OTA updates, sone fhere will be no more builds for these devices.
7. Initial implementation for this strategy will involve a manual process. Eventually, the strategy should be automated

[ArchangeGabriel]

This seems very good to me. :)

[petefoth]

We have a number of old builds available on the build server., for devices which are no longer supported by LineagOS - see below

There are two or three builds per device. We should thin them out so there is only one build per device - the latest. We should then rsync the directories to a new archive folder at https://download.lineage.microg.org/

Builds

``` d800/lineage-18.1-20220815-microG-d800.zip d800/lineage-18.1-20220901-microG-d800.zip d800/lineage-18.1-20221001-microG-d800.zip d801/lineage-18.1-20220815-microG-d801.zip d801/lineage-18.1-20220901-microG-d801.zip d801/lineage-18.1-20221001-microG-d801.zip d802/lineage-18.1-20220815-microG-d802.zip d802/lineage-18.1-20220901-microG-d802.zip d802/lineage-18.1-20221001-microG-d802.zip d803/lineage-18.1-20220815-microG-d803.zip d803/lineage-18.1-20220901-microG-d803.zip d803/lineage-18.1-20221001-microG-d803.zip d850/lineage-18.1-20220815-microG-d850.zip d850/lineage-18.1-20220901-microG-d850.zip d850/lineage-18.1-20221001-microG-d850.zip d851/lineage-18.1-20220815-microG-d851.zip d851/lineage-18.1-20220901-microG-d851.zip d851/lineage-18.1-20221001-microG-d851.zip d852/lineage-18.1-20220901-microG-d852.zip d852/lineage-18.1-20221002-microG-d852.zip d855/lineage-18.1-20220901-microG-d855.zip d855/lineage-18.1-20221002-microG-d855.zip f400/lineage-18.1-20220902-microG-f400.zip f400/lineage-18.1-20221002-microG-f400.zip jasmine_sprout/lineage-18.1-20220903-microG-jasmine_sprout.zip jasmine_sprout/lineage-18.1-20221003-microG-jasmine_sprout.zip jason/lineage-18.1-20220903-microG-jason.zip jason/lineage-18.1-20221003-microG-jason.zip kuntao/lineage-18.1-20220903-microG-kuntao.zip kuntao/lineage-18.1-20221003-microG-kuntao.zip lavender/lineage-18.1-20220903-microG-lavender.zip lavender/lineage-18.1-20221003-microG-lavender.zip lithium/lineage-18.1-20220704-microG-lithium.zip lithium/lineage-18.1-20220718-microG-lithium.zip ls990/lineage-18.1-20220904-microG-ls990.zip ls990/lineage-18.1-20221004-microG-ls990.zip m20lte/lineage-18.1-20220904-microG-m20lte.zip m20lte/lineage-18.1-20221004-microG-m20lte.zip montana/lineage-18.1-20240119-microG-montana.zip obiwan/lineage-18.1-20221004-microG-obiwan.zip obiwan/lineage-18.1-20221015-microG-obiwan.zip platina/lineage-18.1-20221004-microG-platina.zip platina/lineage-18.1-20221015-microG-platina.zip scorpio/lineage-18.1-20220705-microG-scorpio.zip scorpio/lineage-18.1-20220719-microG-scorpio.zip suzu/lineage-18.1-20221005-microG-suzu.zip suzu/lineage-18.1-20221015-microG-suzu.zip twolip/lineage-18.1-20221005-microG-twolip.zip twolip/lineage-18.1-20221015-microG-twolip.zip vs985/lineage-18.1-20221005-microG-vs985.zip vs985/lineage-18.1-20221016-microG-vs985.zip wayne/lineage-18.1-20221005-microG-wayne.zip wayne/lineage-18.1-20221016-microG-wayne.zip whyred/lineage-18.1-20221005-microG-whyred.zip whyred/lineage-18.1-20221016-microG-whyred.zip X00P/lineage-18.1-20220720-microG-X00P.zip X00P/lineage-18.1-20220819-microG-X00P.zip X01AD/lineage-18.1-20220720-microG-X01AD.zip X01AD/lineage-18.1-20220820-microG-X01AD.zip X01BD/lineage-18.1-20220706-microG-X01BD.zip X01BD/lineage-18.1-20220720-microG-X01BD.zip YTX703F/lineage-18.1-20220905-microG-YTX703F.zip YTX703F/lineage-18.1-20221016-microG-YTX703F.zip YTX703L/lineage-18.1-20220905-microG-YTX703L.zip YTX703L/lineage-18.1-20221016-microG-YTX703L.zip monet/lineage-19.1-20230916-microG-monet.zip monet/lineage-19.1-20231007-microG-monet.zip ```

[petefoth]

We should thin them out so there is only one build per device - the latest. We should then rsync the directories to a new archive folder at https://download.lineage.microg.org/

This has now been done :)

We still have 92GB of disk space available,of the total 590G. This should easily be enough room for the remaining 18.1 builds still to be completed this month

[ArchangeGabriel]

I think you can move caprip and Spacewar to the archive (they already have only one build however).

[petefoth]

I think you can move caprip and Spacewar to the archive (they already have only one build however).

Done! Thanks

[petefoth]

Notes

Build Retention

Strategy

When a device is 'new', only one build is available, on both Download server (DS) and Build server (BS)

While a device is supported by LOS, two builds will available on DS

• Normally, both builds will be of the currently supported branch
• When a device is first 'promoted' to a higher branch (e.g.c19.1 -> 20.0),
  • one build on the DS is of the higher branch, and one is of the lower branch
  • the final build of the old branch (e.g. 19.1) is available in the archive directory on the BS (in case of 'emergencies'). It is kept until the device is promoted again (e.g. 20.0 -> 21.0), when it will be replaced by the final build of the higher branch e.g. 20.0

When support is dropped for a device

• the final two builds are available on the DS, in the normal device directory, for a limited time (i.e. until the next build run is performed).
• after that time, one build (the final build for the device) is available in the archive directory on both the DS and the BS

Use cases to handle when building

1. Device is new this build
   • create newlastbuild marker
   • rsync with delete args, leaving one build on both servers
2. Device was new last build
   • rsync without delete args, leaving two builds on both servers
   • remove newlastbuild marker
3. Current build branch is the same as the previous build branch - the normal, default case
   • rsync with delete args,
4. Device is promoted this build
   • rsync with delete args,leaving two builds - one of each branch - on both servers
   • after rsync
     • remove any existing files from the BS archive directory
     • move (copy & delete) old branch build to BS archive directory, leaving only one build in BS zips directory
5. Device was promoted last build
   • rsync without delete args, leaving two builds on both servers
6. Device is dropped this build
   • remove any existing files from the BS archive directory
   • on both servers
   • remove the penultimate build from the zips directory
   • move the zips directory to the archive directory

Cases 1-6 can be handled per device in 'pre-' or post-build.sh

Case 6 must be handled differently

New devices logic

• For device new this build, everything works: we have one build on the BS, that will get synced to DS - all is good
• It's the next build we need to worry about. Then, there will be two builds on the BS and only one on the DS. We need to remove the --delete rsync args, or the previous build will be removed from the BS during rsync
• So, on the first build we need a record that thsi was a new device, that can be read on the next round. Easiest to create a file,
  • EITHER create an empty file using touch touch newdevice
  • OR write the date of first build to the file echo "$builddate" > newdevice
• Then, on the next build, we check for the existence of newdevice and set rsync_delete_args accordingly

Promoted Devices logic

For a promoted device, on the first build, clean_up.py will leave $DELETE_OLD_ZIPS builds on the BS:

• (`$DELETE_OLD_ZIPS - 1) builds of the current branch
• one build - the final build - of the previous branch

We want to leave all of them available on the DS, so do nothing before we call rsync.

We also want the final build of the old branch in the archive directory 'just in case'

Next time we build, we want $DELETE_OLD_ZIPS builds of the current branch on both the BS and the DS. If we remove the old branch build *before* we start the build, (i.e. in pre-build.sh OR after we call rsync on the previos build) , the we will finish the build with the right number on teh BS, and - after calling rsync with*** the --delete rsync args - on the DS too: the old build will be gone there too.

So, if we do it all in post build .sh

• set rsync_delete_args="--delete --delete-excluded
• If newdevice file exists, then
  • set rsync_delete_args="" so the first build - which is not present on BS - does not get deleted from DS
• call rsync
• If this is a new device, then
  • create the newdevice file
• If this is a promoted device, then
  • move old branch build to BS archivedirectory

device_promoted=false new_device=false

there must be at least one ls "$BRANCH_NAME"*.zip file, becasue we've just built it

so how many are there

current_builds=$(ls -1 "$BRANCH".zip | wc -l) total_builds=$(ls -1 lineage-.zip | wc -l) if [ $current_builds -lt 2 ] ; then if [ $total_builds eq $current_builds] ; then new_device=true else device_promoted=true fi fi

files not matching a pattern

find . -type f -not -name "*.html"

builddate=$(date +%Y%m%d) find . -type f -not -name "$BRANCH"-$builddate*" -print

if there's only one, how many builds in total?

builds=$()

• does file exist

if test -f lineage-20.0*.zip; then echo "file exists" fi

• count builds of current branch in current directory
• current branch_num branch_num=$(echo "$BRANCH" | sed 's/lineage-//') | sed/.//) branch_num=$(echo "$branch_num" | sed 's/.//') export branchnum=20

if branch_num = 20 : then previous_branchnum=18 else previous_branchnum="$((branch_num-1))"

ls lineage-"$branch_num"*.zip | wc -l 1