Upgrade from before the refactoring (rev114)

[deusebio]

Upgrade from rev114 does not work and the charms goes into an error state.

Steps to reproduce

1. Deploy the bundle

series: jammy
applications:
  tls-certificates-operator:
    charm: tls-certificates-operator
    channel: latest/edge
    revision: 27
    num_units: 1
    to:
    - "3"
    options:
      ca-common-name: Canonical
      generate-self-signed-certificates: true
    constraints: arch=amd64
  zookeeper:
    charm: zookeeper
    channel: 3/edge
    revision: 114
    num_units: 3
    to:
    - "0"
    - "1"
    - "2"
    constraints: arch=amd64
machines:
  "0":
    constraints: arch=amd64
  "1":
    constraints: arch=amd64
  "2":
    constraints: arch=amd64
  "3":
    constraints: arch=amd64
relations:
- - zookeeper:certificates
  - tls-certificates-operator:certificates

(make sure all units goes up fine)

2. Run pre-upgrade actions juju run zookeeper/leader pre-upgrade-check --format yaml, make sure it completes successfully
3. Run juju refresh zookeeper (that should update to revision 121)

Expected behavior

The upgrade should go through nicely, with no error.

Actual behavior

There are three problems here:

1. The first refresh sets the units into failed because of the following error:

   juju.worker.dependency "uniter" manifold worker returned unexpected error: preparing operation "upgrade to ch:amd64/jammy/zookeeper-121" for zookeeper/0: failed to download charm "ch:amd64/jammy/zookeeper-121" from API server: Get https://10.178.146.204:17070/model/de79be2c-6cc3-4401-8d85-2a27c9c80c7e/charms?file=%2A&url=ch%3Aamd64%2Fjammy%2Fzookeeper-121: cannot retrieve charm: ch:amd64/jammy/zookeeper-121

2. The upgrade actually fails (not sure why) but the charms says

   unit-zookeeper-1: 13:09:27 ERROR unit.zookeeper/1.juju-log upgrade:2: Not all application units are connected and broadcasting in the quorum

3. When update status runs, the charm goes into active/idle state, but there is a service down+the upgrade has failed (the user should see this in a blocked status)

I'm reporting the point 1) into the juju channels, but 2) and 3) I believe they pertain to the charm here.

Versions

Operating system: Ubuntu 22.04

Juju CLI: 3.1.7

Juju agent: 3.1.6

Charm revision: rev114 to re121

LXD: 5.20

Log output

Juju status

Model  Controller  Cloud/Region         Version  SLA          Timestamp
tests  overlord    localhost/localhost  3.1.6    unsupported  13:23:14Z

App                        Version  Status  Scale  Charm                      Channel      Rev  Exposed  Message
tls-certificates-operator           active      1  tls-certificates-operator  latest/edge   27  no
zookeeper                           active      3  zookeeper                  3/edge       121  no

Unit                          Workload  Agent  Machine  Public address  Ports  Message
tls-certificates-operator/0*  active    idle   3        10.178.146.51
zookeeper/0*                  active    idle   0        10.178.146.65
zookeeper/1                   active    idle   1        10.178.146.52
zookeeper/2                   active    idle   2        10.178.146.162

Machine  State    Address         Inst id        Base          AZ  Message
0        started  10.178.146.65   juju-c80c7e-0  ubuntu@22.04      Running
1        started  10.178.146.52   juju-c80c7e-1  ubuntu@22.04      Running
2        started  10.178.146.162  juju-c80c7e-2  ubuntu@22.04      Running
3        started  10.178.146.51   juju-c80c7e-3  ubuntu@22.04      Running

logs.txt

Additional context

[github-actions[bot]]

https://warthogs.atlassian.net/browse/DPE-3563

[deusebio]

PR #116 fixes point 2 and 3.

Point 1 is more general and applies to any charm downloaded from the store. Interestingly, the failed units and logs did not show up when using a local charm from this PR.

I will raise the discussion/bug on a juju channel.