Android: Update update speed

[alohapersona]

I suggest to update the update speeds values, which seem to not 100% reflect latest developments. I looked at the last 6 updates of each (based on https://divestos.org/pages/patch_history), here's the suggested changes:

GrapheneOS: ~1 day The March 2024 update was 2 days from Android upstream release, so the previous "<2 days" is technically untrue. I think "~1 day" better reflects it

DivestOS: 1-2 weeks None of the last 6 updates was delayed by more than 14 days (June 2024 update), the average is 9 days.

CalyxOS: 2-4 days, sometimes longer The June 2024 update took 10 days, but all others are less than 4 days, with 3 out of 6 in 2 days. Even when including the 10 days outlier, the average is just below 4 days.

IodeOS: 2-4 weeks, sometimes longer The March 2024 update was skipped and the June 2024 update took 32 days, but the 5 other updates were delivered with 17 to 25 days of delay, the average is 23 if you exclude the skipped update and 29 days if you include it.

/e/: 1-2 months, sometimes longer The March 2024 update was skipped and the June 2024 update is still pending. The other updates took 30-51 days, with 42 days average.

LineageOS: 1-2 weeks, sometimes longer June 2024 update had 18 days of delay, the other 5 between 7 and 12 days. Overall average 11 days

The most notable change to the current list is that CalyxOS seems to be a little faster by now (except for irregular outliers), with 3 out of 6 updates only having 2 days of delay.

[eylenburg]

Thank you, that's very useful research. I'll update it in the next few days.

[thestinger]

GrapheneOS: ~1 day The March 2024 update was 2 days from Android upstream release, so the previous "<2 days" is technically untrue. I think "~1 day" better reflects it

The Android Open Source Project release can take up to around 12 hours to get pushed to AOSP. They use Pacific time for their announcements and we use UTC so there's a substantial offset. It's usually still the same day in PST when we do the release but has rolled over into the next day in UTC. Bear in mind that our release dates incorporated into the version name are UTC and are often near the very start of the new day because we're doing a release at night in Eastern time which is the next day in UTC once it's several hours before midnight.

[thestinger]

The table and the research above is conflating partial security updates provided by the Android Security Bulletins with the full security updates provided by updating to the latest monthly, quarterly or yearly release of Android. High and Critical severity patches to the Android Open Source Project are generally but not always backported to older releases. There are multiple High and Critical Android Open Source Project patches which are only part of the latest Android 14 QPR3 monthly release.

The backports to older releases contained in the Android Security Bulletin often happen months or even more than a year after it's released in the monthly, quarterly or yearly releases. There are only monthly and quarterly releases for the current version of Android, i.e. Android 14 QPR3 has monthly releases right now and then there will be the Android 15 yearly release. There's an inherent delays of many months for some of the High and Critical severity patches for any OS that's not on the latest release. If you go by when the patch is first released, then the ASB itself is often delayed.

Additionally, Moderate and lower severity issues are NOT generally backported to older releases. Check any of the recent Android Security Bulletins and the patches that are released. You'll see they do not list Moderate and lower severity issues, because they never backported most of them and eventually gave up on it almost entirely. Many of these are still important and it includes most of the privacy patches that are fixing leaks of info not covered by the main permissions, etc. which shouldn't exist instead of security exploits such as being able to get the Wi-Fi MAC address even with no permission when it's not supposed to be possible with any permission for user installed apps.

The most notable change to the current list is that CalyxOS seems to be a little faster by now (except for irregular outliers), with 3 out of 6 updates only having 2 days of delay.

It has gotten a bit faster but it still has significantly longer delays you're missing due to their misleading release notes. It has at least 1 to 2 month delays for the Android Security Bulletin patches for the Fairphone, etc. and it has similar delays for Pixels in some cases when they get delayed for yearly updates. LineageOS had months of delays for Android 14 QPR2, which caused a long delay for full security patches on all devices and also a long delay for the baseline security patches on Pixels.

LineageOS, operating systems based on LineageOS and CalyxOS set an inaccurate Android security patch level. They ignore that the Android security patch level and Android Security Bulletins defining it include many patches that are NOT available solely by applying Android Open Source Project patches. The regular claim to be shipping all open source patches in the CalyxOS release notes is untrue, and they know it's untrue. You're going by inaccurate marketing data instead of the real data.

I've brought this up in the past:

https://github.com/eylenburg/eylenburg.github.io/issues/15

LineageOS having months of delays for QPR2 meant that they didn't have the current Android security patch level on Pixels for months but they set an inaccurate value for the field. Their split out Vendor security patch level is also inaccurate and also misleads users about what's patched or not patched since a lot of what wasn't patched is not tied to the vendor image and a lot of what's in the vendor image, firmware and elsewhere is open source contrary to what's claimed by CalyxOS in their release notes claiming to ship all open source patches by applying the ASB backports when they don't.

It's not clear what the current field is meant to convey. It appears that it's only meant to convey when the partial backports of Android Open Source Project patches are applied. The row should really be quickly renamed to saying it refers to the partial ASB backports for AOSP but should avoid claiming it refers to all ASB patches when it definitely doesn't since that requires per-device patches that are not being included.

[alohapersona]

I'm not suggesting that GrapheneOS is slow, I think the 1 day on average is indeed a very good number, considering the time it takes Google to upload and also that it takes significant time to actually compile Android.

---

You are right that the table does not properly define the details of which parts are updated and how proper the updates are done. The old value was clearly referring to delay between release of upstream and claimed inclusion in update. I was only suggesting to update the numbers based on latest developments, which I think is an easy fix and independent of the other topics you brought up.

[thestinger]

@alohapersona The data you've gathered above is different for GrapheneOS and non-GrapheneOS. For GrapheneOS, it's referring to when the full Android Security Bulletin and Pixel Security Bulletin patches are applied. For the others, it's referring to when they ship partial Android Security Bulletin patches without the Pixel Security Bulletin patches in most cases. Most Pixel Security Bulletin patches are relevant to more than Pixels. There are also a huge number of unlisted patches for the Android Open Source Project with Moderate or lower severity, which are only provided through the latest monthly, quarterly and yearly releases. ASB only lists High severity and above. Some High severity and above AOSP patches are not backported as part of the ASB for months or in rare cases aren't ever backported. The table is making the situation for LineageOS-based operating systems and CalyxOS seem much better than it is, and changing it as proposed above will make that issue worse.

Here are different levels of security patches:

1) Subset of the Android Security Bulletin patches backported to older releases of the Android Open Source Project (currently 12, 13 and 14)

This only provides most High and Critical severity AOSP userspace patches. It does not provide nearly any of the Moderate or lower severity patches.

Additionally, these backports are made to the initial yearly release of the OS. They're not made to the monthly or quarterly releases with substantial changes. This means when LineageOS didn't port to Android 14 QPR2 for several months, there were some ASB AOSP patches which didn't apply cleanly and had to be manually ported. In some cases, these are skipped when they don't apply cleanly.

LineageOS, CalyxOS, etc. set their Android security patch level based on applying these patches even though that's not how it's defined and is missing half of the Android Security Bulletin patches. They announce this as shipping the security patches. Therefore, getting the data from them is unreliable unless this is how you're defining the field and it's not really clear how it's useful to define it this way where users are missing High and Critical severity ASB patches but yet are told they aren't.

This is what the table is currently portraying as providing the security patches, which is simply wrong.

2) Subset of the Android Security Bulletin patches including non-hardware-specific kernel patches

Android Security Bulletins list generic kernel patches in the 2nd section and it's not inherently backported for each device through the OS providing ASB AOSP backports. This is easy to do, but still doesn't provide all of the ASB patches.

3) Full Android Security Bulletin patches

This requires that the full hardware-related ASB patches are available for firmware and drivers. LineageOS and /e/OS often don't ship vendor code and firmware for devices.

4) Full vendor patches

This requires shipping the latest Android Open Source Project release and all of the vendor patches from the vendor's own bulleting/releases, such as Samsung's bulletin

For Pixels, 4 and 5 are the same. For other devices, they aren't.

5) Full vendor + Pixel Security Bulletin patches

Pixel Security Bulletins mostly list patches relevant to other devices using similar hardware such as Snapdragon, Exynos, Broadcom/Qualcomm/Samsung GNSS, Broadcom/Qualcomm Wi-Fi, Qualcomm/Samsung cellular, the same drivers for touchscreens, etc. If you look through them, you'll see it's mostly hardware related patches for hardware used beyond Pixels. However, there are also Android Open Source Project patches listed in these. I can give several recent High severity examples we reported ourselves: a High severity Bluetooth bug exploitable in proximity to the device and a High severity issue for device admin API wipes and other wipes being interruptible before and AOSP patch. Neither of those is in the ASB despite being AOSP patches with High severity. This shatters people's misconceptions about how it works.

[alohapersona]

The data you've gathered above is different for GrapheneOS and non-GrapheneOS. For GrapheneOS, it's referring to when the full Android Security Bulletin and Pixel Security Bulletin patches are applied. For the others, it's referring to when they ship partial Android Security Bulletin patches without the Pixel Security Bulletin patches in most cases.

Just to clarify: You are saying that GrapheneOS shipped ASB patches earlier than 1 day on average and I wrongly took the numbers for PSB and that's not correctly reflected in the table? That's great, would you suggest to write ~0 days instead?

[thestinger]

Just to clarify: You are saying that GrapheneOS shipped ASB patches earlier than 1 day on average and that's not correctly reflected in the table? That's great, would you suggest to write ~0 days instead?

GrapheneOS essentially always ships ASB YYYY-MM-01 patches in less than 1 day. Shipping the YYYY-MM-05 patches requires having all the hardware-specific patches listed in the ASB which for Pixels generally requires the monthly, quarterly and yearly releases. We take around 2 days to get quarterly and yearly releases in Alpha, and a couple more to get them into Stable. The AOSP release is often delayed by up to 24 hours and we're not including that as part of the delay. The monthly, quarterly and yearly updates are also sometimes 1 week after the ASB release. We do an ASB AOSP patch release right away before the monthly, quarterly and yearly release is available. This month, we did an early ASB AOSP patch release since we thought the main release might be delayed but then it came out anyway so we had the ASB AOSP patch release available in Alpha hours before we replaced it with the full monthly release. They're not the same thing, and I think the ways in which they're not the same are being missed by most people.

Since Pixels have moved away from Snapdragon to Tensor, there are often no ASB YYYY-MM-05 patches relevant to them but you still need the monthly, quarterly and yearly releases for their full YYYY-MM-05 patch level, they're just mostly listed in the Pixel Update Bulletin now since the scope of the ASB is small and doesn't include Exynos, Broadcom/Qualcomm Wi-Fi, touchscreens, etc. They're not always consistent about what goes into ASB or Pixel Update Bulletin. Most patches in the Pixel bulletin are relevant beyond Pixels yet other devices don't have to patch them to claim to have the latest patch level, but on Pixels that's part of what the patch level includes.

The current value of less than 2 days is correct for us shipping the full quarterly and yearly releases, if it refers to a production build being available to users in any release channel. It takes often LineageOS many months to ship the quarterly and yearly releases for any devices and they're supporting a lot of devices without them.

I do not understand what the field is meant to mean. GrapheneOS is being treated differently than the other operating systems right now, because GrapheneOS is being held to the standard of shipping full security patches but other operating systems are often having long delays for that which are being ignored, in the current table and what you've listed above.

[eylenburg]

So @thestinger would it be correct if I changed the name of the row to say something like "Security update speed for Partial Security Updates (ASB)", then using the numbers @alohapersona mentioned and changing GrapheneOS to "same day"?

[thestinger]

@alohapersona By ASB patches, do you mean the full ASB patches which include certain driver and firmware patches or only the AOSP patches? If you mean only the AOSP patches, do you mean only the YYYY-MM-01 ones or also the YYYY-MM-05 including kernel patches available via AOSP?

If this field is referring to the Android Security Bulletin patches, it should say so, and it should < 1 day for GrapheneOS. Even if it's referring to Android Security Bulletin patches, that doesn't mean that the numbers currently in the table or shown above are correct for other operating systems. AOSP userspace backports are not the full ASB patches. There are also the generic AOSP kernel patches and then the majority of the 2nd section of driver/firmware updates. These updates require shipping all the vendor updates, which is not done by /e/OS or LineageOS for many of the devices they support. For many devices they leave the vendor and firmware images which were present when the OS was installed, meaning no patches for those even if the vendor releases the patches.

[thestinger]

@eylenburg The current row should be renamed to Android Security Bulletin patches for AOSP because it appears to refer to a subset of the patches. There should be other rows because that's not nearly all of the security patches. It omits Moderate and lower severity AOSP patches and non-AOSP patches. The situation even for Android Security Bulletin patches for AOSP is an inconsistent mess for the other operating systems so the numbers above are not strictly correct since not all devices end up applying all the kernel patches and there are components from AOSP in vendor, etc.

[thestinger]

Here's my initial suggestion:

1) Rename current security patch row to AOSP subset of Android Security Bulletin patches with < 1 day for GrapheneOS 2) Add a new row for full Android Security Bulletin and vendor patches with < 2 days for GrapheneOS, and put it as yellow for each other OS stating that it's a complex situation. CalyxOS up to 1 to 2 months of delays for this based on Android 13 and 14, while most other listed operating systems have more. Perhaps they won't have 2 months of delays again as they did for Android 13 but that remains to be seen. LineageOS had months of delays for Android 14 QPR2. I'm unsure what their Android 14 QPR3 situation is at the moment.

[alohapersona]

@thestinger my data is derived from https://divestos.org/pages/patch_history. This is talking about ASB and proclaimed support by each OS to have it included. According to the table, GrapheneOS included the ASB 2024-03-01 in a build on 2024-03-06, that is 2 days after the ASB was published 2024-03-04 (due to the weekend), so my calculation resulted in 2 days of delay for that month.

Now the question is: Did GrapheneOS have an update before 2024-03-06 that did include the ASB 2024-03-01? If so, you are right, the table is wrong and puts GrapheneOS to a higher standard. If not, and the ASB 2024-03-01 update was provided more than 24 hours after ASB 2024-03-01 was published, < 1 day for GrapheneOS would be incorrect, that's why I suggested ~ 1 day.

While it's certainly possible to rename the row, I'd like to point out that the row already has a link to https://divestos.org/pages/patch_history, which explains a few of your points already and allows to understand how the numbers were derived. Yet a rename to something like you suggested or maybe something shorter and easier understood like "AOSP security update delay", might be a reasonable option.

[thestinger]

my data is derived from https://divestos.org/pages/patch_history. This is talking about ASB and proclaimed support by each OS to have it included. According to the table, GrapheneOS included the ASB 2024-03-01 in a build on 2024-03-06, that is 2 days after the ASB was published 2024-03-04 (due to the weekend), so my calculation resulted in 2 days of delay for that month.

The data does not show when the Android Security Bulletin was provided but rather when the AOSP portion of the patches in the initial YYYY-MM-01 section were applied. I've explained above why that does not include all ASB patches. It's easy to confirm it does not from the bulletins themselves.

Now the question is: Did GrapheneOS have an update before 2024-03-06 that did include the ASB 2024-03-01? If so, you are right, the table is wrong and puts GrapheneOS to a higher standard. If not, and the ASB 2024-03-01 update was provided more than 24 hours after ASB 2024-03-01 was published, < 1 day for GrapheneOS would be incorrect, that's why I suggested ~ 1 day.

You're wrong. It was less than 24 hours after it was published. You're mixed up about when they release it in the day and how time zones work. You're also wrong about what the Android Security Bulletin includes and what the other operating systems are shipping, which has been explained above. This release shipped the monthly release of Android with the full security patches for March. It did not only ship the AOSP subset of the Android Security Bulletin patches, which are not the full ASB patches. It did not only ship the ASB patches. It shipped the monthly AOSP release in less than 24 hours after it was available, which was also less than 24 hours after the ASB backports to Android 12, 13 and 14 were available. You're consistently making inaccurate and biased claims about GrapheneOS with your proposals here. It's clear that you're biased against it, and therefore are using a bogus definition of Android Security Bulletin patches where operating systems NOT shipping them are credited with doing so.

Pacific time has a 7 hour offset from UTC. Our release times are UTC and theirs are Pacific. AOSP publishing code at 5pm Pacific means our release will show it as being the next day.

While it's certainly possible to rename the row, I'd like to point out that the row already has a link to https://divestos.org/pages/patch_history, which explains a few of your points already and allows to understand how the numbers were derived. Yet a rename to something like you suggested or maybe something shorter and easier understood like "AOSP security update delay", might be a reasonable option.

The data is missing times and time zones. It's also not differentiating between completely different things.

This is not correct data for Android Security Bulletin patches, which include more than the backports...

GrapheneOS ships the full security patches, not only the backports. The backports are often posted at 3pm Pacific and the full patches at 7pm Pacific. The full patches sometimes take a full day to push. We shipped them in under 24 hours from when both were published on that month... You're just confused about when it was actually published (they had substantial delays for both the backport tags and monthly release tags) and the fact that their times are in Pacific time and ours are in UTC... Our times would look nicer if we used Pacific or Eastern but we haven't been trying to make it appear better... but sure, if you start spreading misinformation about GrapheneOS as you are here we will consider it.

[alohapersona]

You're wrong. It was less than 24 hours after it was published.

I haven't done this myself, I'm not the author of https://divestos.org/pages/patch_history, I am just using the data from there. That table claims there is 2 days between ASB publication and GrapheneOS release. That's why I asked if it was less than 24 hours but still displayed as 2 days in the table, which I understand can happen because of timezones.

I'm trying to understand what you say here to provide the possibility to make an accurate assessment and fair comparison possible. Saying "You're wrong" when I ask a question is not really making it easier for me to understand the discrepancies between your numbers and the one from the table.

[thestinger]

@alohapersona

I haven't done this myself, I'm not the author of https://divestos.org/pages/patch_history, I am just using the data from there. That table claims there is 2 days between ASB publication and GrapheneOS release. That's why I asked if it was less than 24 hours but still displayed as 2 days in the table, which I understand can happen because of timezones.

The data goes solely by date but doesn't take into account when the AOSP release was actually done. It can be delayed by up to a day and will be shown as being a day earlier here. It also doesn't take into account the 7 hour difference of Pacific vs. UTC.

As stated earlier, this also does not show when Android Security Bulletin patches were shipped. You're portraying Android Security Bulletin patches as only including the AOSP patches when they include significantly more. These are also not the full generic Android security patches, or the full hardware specific ones beyond that. The current row is simply incorrect.

[thestinger]

LineageOS not shipping Android 14 QPR2 for months meant they didn't ship the full ASB patches for Pixels for months since they had to update to Android 14 QPR2 patches for that. Applying the AOSP ASB backports did not provide the current patch level on Pixels, but they incorrectly marked it as doing so. CalyxOS and each LineageOS-based OS sets an inaccurate patch level and makes inaccurate statements about this. DivestOS doesn't really fix LineageOS dealing with this in an incorrect way.

Shipping only the AOSP subset of ASB patches is not providing the ASB patches and should not result in a higher Android security patch level beyond updating 2024-07-05 to 2024-08-01 if you ship all the ASB AOSP patches for August 2024.

I spent a long time writing what I wrote above. It took a lot of time, which was taken away from other things. Since you're ignoring most of what I wrote, I no longer think you simply misunderstand the topic.

[eylenburg]

I'll re-open this for now because the current row is not 100% correct, especially as it's not distinguishing between the different types of updates. I will need to re-read this discussion and make some of the suggested changes.

[thestinger]

The current row should be renamed to AOSP subset of Android Security Bulletin and the data slightly corrected. However, it really only shows part of the picture for various reasons. It doesn't communicate how the device-related patches in the 2nd half of each ASB get shipped, if they do at all since /e/OS and LineageOS don't ship them for a lot of their supported devices at all. It's also missing the fact that full security patches require the latest monthly/quarterly/yearly updates. Additionally, you need to keep up with the vendor's releases which means falling behind on quarterly or yearly releases results in missing many of the High/Critical severity patches for Pixels or another device launching the new quarterly/yearly version in a reasonable time. LineageOS falls many months behind and CalyxOS has fallen around 2 months behind in the recent past.

The issue is that this heavily varies by device. Pixels require the alternate OS to always keep up to ship updates. Some other device ship yearly updates quickly too. Devices not shipping the latest OS release make it harder for the alternate OS to do it but it is possible, especially if they don't build the vendor image, etc. If the vendor doesn't ship the releases, then the firmware/driver code used from them will be missing the patches. For example, Fairphone is consistently 1 or 2 months behind on the Android Security Bulletin patches so CalyxOS, LineageOS, etc. on it is at least that far behind at all times for the non-AOSP half of the ASB.

This is one of the most basic requirements for GrapheneOS hardware support but is currently only satisfied by Pixels. Every other OEM doesn't ship the monthly/quarterly updates and typically has major delays for yearly updates. Most also do a bad job shipping the backported patches. The table currently lists hardware brands that are supported but doesn't have a hint about why some operating systems would choose not to support hardware rather than just not having time to do it.