Charms are failing basic hooks with ImportError after X-B Upgrade - Forgot to 'juju upgrade-series <m> prepare'

[dashmage]

After performing a Xenial to Bionic upgrad, a handful of charms have entered 'Error' state while running basic hooks:

nova-compute-kvm/30 active idle 32 10.55.65.18 Unit is ready
ceilometer-agent/23 active idle 10.55.65.18 Unit is ready filebeat/30 error idle 10.55.65.18 hook failed: "update-status" landscape-client/14 active idle 10.55.65.18 System successfully registered lldpd/6 active idle 10.55.65.18 LLDP daemon running neutron-openvswitch/23 active idle 10.55.65.18 Unit is ready nrpe-host/21 active idle 10.55.65.18 icmp,5666/tcp ready
ntp/25 error idle 10.55.65.18 123/udp hook failed: "update-status" rsyslog-forwarder-ha/31 unknown idle 10.55.65.18
sysconfig-compute/3 error idle 10.55.65.18 hook failed: "update-status"
telegraf/28 error idle 10.55.65.18 9103/tcp hook failed: "update-status" trilio-data-mover/109 error idle 10.55.65.18 hook failed: "update-status"

Upon reviewing the juju unit logs all charms are encountering an ImportError:

2020-03-18 13:32:04 DEBUG juju.worker.uniter.operation executor.go:59 running operation run update-status hook 2020-03-18 13:32:04 DEBUG juju.machinelock machinelock.go:156 acquire machine lock for uniter (run update-status hook) 2020-03-18 13:32:04 DEBUG juju.machinelock machinelock.go:166 machine lock acquired for uniter (run update-status hook) 2020-03-18 13:32:04 DEBUG juju.worker.uniter.operation executor.go:90 preparing operation "run update-status hook" 2020-03-18 13:32:04 DEBUG juju.worker.uniter.operation executor.go:90 executing operation "run update-status hook" 2020-03-18 13:32:04 DEBUG update-status Could not find platform independent libraries 2020-03-18 13:32:04 DEBUG update-status Could not find platform dependent libraries 2020-03-18 13:32:04 DEBUG update-status Consider setting $PYTHONHOME to [:] 2020-03-18 13:32:04 DEBUG update-status Fatal Python error: Py_Initialize: Unable to get the locale encoding 2020-03-18 13:32:04 DEBUG update-status ImportError: No module named 'encodings' 2020-03-18 13:32:04 DEBUG update-status 2020-03-18 13:32:04 DEBUG update-status Current thread 0x00007fb437a56740 (most recent call first): 2020-03-18 13:32:05 ERROR juju.worker.uniter.operation runhook.go:132 hook "update-status" failed: signal: aborted (core dumped)

---

Imported from Launchpad using lp2gh.

• date created: 2020-03-18T13:40:20Z

• owner: whereisrysmind

• assignee: None

• the launchpad url

[dashmage]

(by whereisrysmind) Note: these are all reactive charms and are having problems with their virtual environments

ubuntu@CMOOSCHSTUP7132:/var/lib/juju/agents$ source unit-ntp-25/.venv/ bin/ include/ lib/ share/
ubuntu@CMOOSCHSTUP7132:/var/lib/juju/agents$ source unit-ntp-25/.venv/bin/activate (.venv) ubuntu@CMOOSCHSTUP7132:/var/lib/juju/agents$ python Could not find platform independent libraries Could not find platform dependent libraries Consider setting $PYTHONHOME to [:] Fatal Python error: Py_Initialize: Unable to get the locale encoding ImportError: No module named 'encodings'

Current thread 0x00007f68373e0740 (most recent call first): Aborted (core dumped)

[dashmage]

(by peter-sabaini) As the venv setup is handled by layer-basic, I wonder if the defect is really there. It looks like there is an invalid (old or incomplete) venv around.

I have tried to repro this but was unable to do so I'm afraid.

Ryan: would you be able to provide more context on the issue?

• Steps to repro
• Logs or maybe even sosreports

Cheers!

[dashmage]

(by whereisrysmind) This occurred when I ran a series upgrade, but forgot to prepare juju beforehand.

The solution involved the following workaround:

for CHARM_DIR in /var/lib/juju/agents/unit-ntp-25/charm \ /var/lib/juju/agents/unit-sysconfig-compute-3/charm \ /var/lib/juju/agents/unit-bcache-tuning-32/charm \ /var/lib/juju/agents/unit-hw-health-18/charm \ /var/lib/juju/agents/unit-trilio-data-mover-109/charm \ /var/lib/juju/agents/unit-telegraf-28/charm \ /var/lib/juju/agents/unit-filebeat-30/charm; do echo $CHARM_DIR; rm -rf $CHARM_DIR/wheelhouse/.bootstrapped $CHARM_DIR/../.venv; cd $CHARMDIR; JUJU_CHARM_DIR=$CHARM_DIR PYTHONPATH=$CHARM_DIR/lib python3 -c 'import charms.layer.basic; charms.layer.basic.bootstrap_charm_deps()'; done

[dashmage]

(by janitor) [Expired for Telegraf Charm because there has been no activity for 60 days.]

[dashmage]

(by janitor) [Expired for NTP Charm because there has been no activity for 60 days.]