Cloud-init 23.4 introduces hard requirement on snap store connectivity for `package_update_upgrade_install`

[MggMuggins]

Bug report

4202 introduces a hard requirement on snap store connectivity when using the following in the user/vendor data:

package_update: true
package_upgrade: true

This may or may not be a concern in cloud-init upstream, but the changes were included in 23.4 which was SRU'd to Jammy. This is a regression for users with restrictive firewalls preventing access to the snap store.

Steps to reproduce the problem

1. Create a vm with userdata containing:

   package_update: true
   package_upgrade: true

2. Enable Proposed in the vm

3. Install cloud-init 24.1.3-0ubuntu1~22.04.4

   sudo apt-get install cloud-init -t jammy-proposed

4. Create a lxd ACL preventing access to the snapd network requirements

   
   lxc network acl create stop-snap
   snap_reqs=(
   api.snapcraft.io
   storage.snapcraftcontent.com
   canonical-lgw01.cdn.snapcraftcontent.com
   canonical-lcy01.cdn.snapcraftcontent.com
   canonical-lcy02.cdn.snapcraftcontent.com
   canonical-bos01.cdn.snapcraftcontent.com
   )

for name in "${snap_reqs[@]}"; do for ip in $(dig +short "${name}"); do lxc network acl rule add stop-snap egress action=reject destination="${ip}" done done

```sh
lxc network set lxdbr0 security.acls.default.ingress.action=allow
lxc network set lxdbr0 security.acls.default.egress.action=allow
lxc network set lxdbr0 security.acls="stop-snap"

5. Rerun cloud-init:

   sudo cloud-init clean --logs --reboot

6. Cloud-init fails as snap refresh can't get out to api.snapcraft.io.

7. Downgrade to cloud-init 23.1.2-0ubuntu0~22.04.1:

sudo apt-get install cloud-init=23.1.2-0ubuntu0~22.04.1

8. Rerun cloud-init:

   sudo cloud-init clean --logs --reboot

9. Cloud-init succeeds (no hard requirement on snap store)

Environment details

• Cloud-init version: 23.4-0ubuntu1~22.04.1 and later
• Operating System Distribution: Ubuntu 22.04
• Cloud provider, platform or installer type: Any

[blackboxsw]

@MggMuggins thank you for filing this bug and improving cloud-init.

We originally thought we may treat this as won't fix for a couple of reasons:

1. This feature package_upgrade: true should attempt to upgrade all packages installed by any active package manager present within the environment 2.. This particular environment is unique in that the sandboxed limited network which doesn't permit an available package manager to run properly and there is no way in snap tooling or local environmental artifacts to query the status of outbound connectivity so that cloud-init can determine it shouldn't call snap refresh.
2. This "supplemental feature" was released in 23.4 and is already consumed and used in stable releases, so removing that feature now as part of a new SRU would break admins which now depend on snap refresh happening as part of package_upgrade: true

What we originally thought here was that workarounds would be possible by modifying the user-data provided to the machines to prevent the side-effect of calling snap refresh due to the #cloud-config package_upgrade: true option:

User-data workaround 1: Instead of this user-data for a snap-limited network environment

#cloud-config
package_upgrade: true

Provide the alternative which focus on apt upgrades and avoid triggering snap refresh

#cloud-config
runcmd: [apt-get upgrade --assume-yes]

But, instead of this non-specific workaround above, we've come around to a proposal of a more specific way that user-data can inform cloud-init that it shouldn't call snap refresh due to the blanket package_upgrade: true in cloud-config.

One can basically set snap refresh --hold to tell snappy not to automatically refresh packages in general, this is enough of an indicator to cloud-init that snap refreshes and generally unwanted it shouldn't attempt to invoke this command.

What this requires from cloud-init is a preliminary call to snap get system refresh.hold to see if this configuration is set. This functionality will be provided in this TBD PR.

To configure this with user-data add the appropriate snap command to your user-data:

#cloud-config
package_upgrade: true
snap:
  commands:
    00: snap refresh --hold