Question on update size

[aaronovz1]

Does it seem correct that I would get an update size of 50MB when the only change made between distro versions was to add one small file? The statistics show that 800 files changed and 1 new file. Does this sound like it is not deterministic? Or does adding one file really cause a chain reaction of 800 file changes?

Happy to provide logs and gather any other information to get to the bottom of this.

[matthewrsj]

Logs would be helpful.

How do you know there was only supposed to be one file addition? If you are using swupd in Clear Linux several packages are updated each release, which would explain 800 file changes.

[aaronovz1]

I am not using Clear. I am using meta-swupd with my own Yocto based OS. By one file change, I mean I changed one recipe to add a random <1KB file. When I ran the update on the client I checked the received bytes on the network interface and saw it downloaded 50MB, and swupd reported 1 new file and over 800 changes. I was just curious if this is normal. I was hoping updates would be much smaller due to bsdiff and that I could roll out small security updates of only a couple MB's.

Let me know what kind of logs might be useful if this doesn't seem normal.

[phmccarty]

@aaronovz1 That does not sound normal.

Before you are creating a new build (version VER) with swupd-server, is the /var/lib/update/LAST_VER file updated correctly? LAST_VER is the last build version swupd-server should compare against for VER. The manifests will generate with expected changes only if LAST_VER is set properly.

If LAST_VER looks correct, check the manifests to determine whether they have changed appropriately according to your changes. Most swupd-client update operation hinges on comparing differences between manifests, so if you are seeing client updates for many files you did not update, something may be off with manifest creation.

[aaronovz1]

I can see that LAST_VER is being updated to latest version number each time I bump OS_VERSION. I have noticed something interesting which makes me think you're right, either about image creation or manifest creation not working correctly. Any new image I generate seems to always say that 803 files changed and I get a message at the end of "swupd update" command which says:

803 files were not in a pack

Just for reference I am using the patches from pull #47 and #48. I don't know if this is causing issues or whether it is how I have my image configured through Yocto and meta-swupd. I am going to try reverting back to vanilla swupd-server and swupd-client and see if there is any difference.

I am not sure if you are familiar with the meta-swupd but I have also noticed there are a lot of hash mismatches on the very first update from 1 to 2.

[aaronovz1]

FYI switching back to vanilla swupd made no difference. There always seems to be about 800 files that change every update and which are "not in a pack".

[aaronovz1]

I have uploaded the logs/output. They are available here: https://www.dropbox.com/s/t0g9u8i4mnza9pe/swupd_logs.tar.gz?dl=0

Appreciate if someone with a deeper understanding of the manifest files can take a look and see if there is anything obvious. I see some "Bad package runtime state detected" messages. I am using the "stateful" option of meta-swupd so changes can be made in /etc, I don't know if this causes an effect.

[aaronovz1]

@phmccarty did you have a chance to look over the logs?

[phmccarty]

@aaronovz1 Sorry for the delay. I have downloaded the logs tarball and will inspect it today.

[phmccarty]

@aaronovz1 I analyzed the swupd-server logs, but I didn't see anything obviously wrong there. And indeed, I see that LAST_VER was set correctly to "1" before building "2". So, no issues there.

I then analyzed the manifests (from www/VER/) and the fullfile tarballs (from www/VER/files/) to determine the scope of what in fact changed between builds "1" and "2".

There were 805 files that changed in total, spread over multiple bundles, and 1 new file was added to the dss bundle (/opt/seeingmachines/dss/versiontest). Looking at the changed files, they are mostly binary ELF files with different hashes but with the same build-id. Using the same build-ids may be an artifact of how Yocto works (I don't know), but because the file hashes differ, the binaries appear to have been rebuilt, even though you only added a new file to a single bundle.

With all the evidence I can collect from your swupd_logs.tar.gz tarball, I think swupd-server is working as expected. For some reason, many binaries were rebuilt in the process of preparing the chroots for swupd-server input for build "2".

[pohly]

@aaronovz1 swupd-server always creates a pack-os-core-from-0.tar which contains the entire rootfs, so in the update repo it is normal to have at least one fairly large file.

Do you have SWUPD_DELTAPACK_VERSIONS set when producing the update? That's how meta-swupd configures against which previous versions it computes deltas. If you don't use that, then swupd-client will have to download pack-os-core-from-0.tar instead of downloading just the changed files.

But that doesn't explain why swupd-server deems the files as changed. Did you redo the build from scratch, i.e. remove tmp and the sstate cache, i.e. rebuild all binaries again? That would likely cause this, as rebuilding from source is not currently fully reproducible.

FWIW, I am currently adding automated tests for meta-swupd using the refkit distro and infrastructure and so far it is working as expected.

[aaronovz1]

@pohly I wasn't aware of that variable and it seems like there is a mismatch in the documentation because I had thought the variable was named SWUPD_DELTAPACKS along with another variable called SWUPD_N_DELTAPACKS as outlined here. I will update the variable you mention and see if that makes a difference.

I didn't delete tmp or sstate but I do have a question regarding sstate cache. Can multiple Yocto projects use the same sstate cache? I have a number of projects, some using Morty some using Pyro, etc and building different operating systems variations within each project. Only one project is using SWUpd currently, but they all point to to the same DL_DIR and SSTATE_DIR directories. This was to save space on servers as each cache uses over 100GB. Could this be a cause for problems in a SWUpd environment?

[pohly]

On Wed, 2017-09-20 at 15:59 +0000, Aaron wrote:

@pohly I wasn't aware of that variable and it seems like there is a mismatch in the documentation because I had thought the variable was named SWUPD_DELTAPACKS along with another variable called SWUPD_N_DELTAPACKS as outlined here. I will update the variable you mention and see if that makes a difference.

You were probably following the Wiki, which unfortunately is out-dated. The actual "documentation" are the comments in the swupd-image.bbclass

• sorry for that.

At this point I am not sure how useful a separate Wiki page is. I'm tempted to just delete it entirely.

I didn't delete tmp or sstate but I do have a question regarding sstate cache. Can multiple Yocto projects use the same sstate cache?

Yes.

Could this be a cause for problems in a SWUpd environment?

No, shouldn't matter.

[aaronovz1]

@pohly, Setting SWUPD_DELTAPACK_VERSIONS = "1" didn't make a difference. I also moved the project to its own sstate cache just in case. I tried a much simpler test by simply bumping DISTRO_VERSION and building the new image outside of our CI system to rule out anything else. I still get over 700 files reported as changed. Couple questions:

1. Is there a particular spot that DISTRO_VERSION should be in? Currently I have it under my meta layer in conf/distro/mydistro.conf (edit: also tried local.conf, no difference)
2. Have you tried meta-swupd under the Pyro branch of Yocto?
3. Do you have a minimal working example of how a project should be setup?
4. What is the "delta" folder under each version generated from swupd? It's always empty for me.

[pohly]

On Thu, 2017-09-21 at 14:00 -0700, Aaron wrote:

@pohly, Setting SWUPD_DELTAPACK_VERSIONS = "1" didn't make a difference. I also moved the project to its own sstate cache just in case. I tried a much simpler test by simply bumping DISTRO_VERSION and building the new image outside of our CI system to rule out anything else. I still get over 700 files reported as changed.

What you can do to narrow this down is to look at the changed files in the old and new update. Locate a modified file in the Manifest.full, get the .tar from the old and new update, unpack, compare content and attributes. There are some pitfalls which need to be avoided:

• both image-prelink and image-mklibs can rewrite files during rootfs construction, and if I remember correctly, the result was not reproducible
• one has to use useradd-staticids to ensure that uids/gids don't change

   Couple questions:

  Is there a particular spot that DISTRO_VERSION should be in? Currently I have it under my meta layer in conf/distro/mydistro.conf.

You mean OS_VERSION? OS_VERSION needs to be set anew each time you start a build, i.e. local.conf or auto.conf. DISTRO_VERSION can be in your distro config.

Have you tried meta-swupd under the Pyro branch of Yocto?

Not explicitly with Pyro. I used it with some older version of Yocto and now again with current master. As Pyro is somewhere between those two ends, it should work.

Do you have a minimal working example of how a project should be setup?

https://github.com/ostroproject/ostro-os has it enabled, but that's not exactly "minimal" :-/

What is the "delta" folder under each version generated from swupd? It's always empty for me.

It contains binary diffs when using the SWUPD_DELTAPACK_VERSIONS variable and swupd-server determines that a diff is more efficient than downloading the entire modified file.

     

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c5 5493e4bb","name":"GitHub"},"entity":{"external_key":"github/clearlinu x/swupd-server","title":"clearlinux/swupd-server","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/ass ets/143418/17495839/a5054eac-5d88-11e6-95fc- 7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent .com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed- b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/clearlinux/swupd- server"}},"updates":{"snippets":[{"icon":"PERSON","message":"@aaronov z1 in #66: @pohly, Setting SWUPD_DELTAPACK_VERSIONS = \"1\" didn't make a difference. I also moved the project to its own sstate cache just in case. I tried a much simpler test by simply bumping DISTRO_VERSION and building the new image outside of our CI system to rule out anything else. I still get over 700 files reported as changed. Couple questions:\r\n\r\n1. Is there a particular spot that DISTRO_VERSION should be in? Currently I have it under my meta layer in conf/distro/mydistro.conf.\r\n2. Have you tried meta-swupd under the Pyro branch of Yocto?\r\n3. Do you have a minimal working example of how a project should be setup?\r\n4. What is the \"delta\" folder under each version generated from swupd? It's always empty for me.\r\n\r\n"}],"action":{"name":"View Issue","url":"https://github.com/clearlinux/swupd- server/issues/66#issuecomment-331280537"}}}

[aaronovz1]

Success! The culprit was that I had both image-prelink and image-mklibs set in the local.conf. After removing those and using OS_VERSION and SWUPD_DELTAPACK_VERSIONS in local.conf, I was able to produce a very small update like expected.

Thanks to everyone for helping me resolve this.