hiccups when contacting nova.clouds.archive.ubuntu.com

[lourot]

Since at least 2021-01-14 the cot_sink and cot_charm_upgrade always fail in the same way: the install hook of a charm, always a different one, seems to have a hiccup when contacting nova.clouds.archive.ubuntu.com

Last occurence: http://osci:8080/job/zaza_cot_runner/95/

23:12:35 2021-02-03 23:12:35 [ERROR] Units in error state: swift-proxy/2
23:12:35 2021-02-03 23:12:35 [ERROR] Juju log for swift-proxy/2
23:12:37 2021-02-03 23:12:35 [WARNING] unknown delta type: id
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 DEBUG juju.worker.dependency engine.go:564 "hook-retry-strategy" manifold worker started at 2021-02-03 23:11:37.84600905 +0000 UTC
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 INFO juju.worker.leadership tracker.go:217 swift-proxy leadership for swift-proxy/2 denied
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 DEBUG juju.worker.leadership tracker.go:224 swift-proxy/2 waiting for swift-proxy leadership release
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 DEBUG juju.worker.logger logger.go:93 reconfiguring logging from "<root>=DEBUG" to "<root>=INFO"
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-swift-proxy-2
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.8.7-focal-amd64
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 INFO juju.worker.meterstatus runner.go:89 skipped "meter-status-changed" hook (missing)
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 INFO juju.worker.uniter uniter.go:302 unit "swift-proxy/2" started
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 INFO juju.worker.uniter uniter.go:581 resuming charm install
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 INFO juju.worker.uniter.charm bundles.go:79 downloading cs:~openstack-charmers/swift-proxy-95 from API server
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:37 INFO juju.downloader download.go:111 downloading from cs:~openstack-charmers/swift-proxy-95
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:38 INFO juju.downloader download.go:94 download complete ("cs:~openstack-charmers/swift-proxy-95")
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:38 INFO juju.downloader download.go:174 download verified ("cs:~openstack-charmers/swift-proxy-95")
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:38 INFO juju.worker.uniter uniter.go:314 hooks are retried false
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:38 INFO juju.worker.uniter.storage resolver.go:125 initial storage attachments ready
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:11:38 INFO juju.worker.uniter resolver.go:147 found queued "install" hook
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:12:33 WARNING install E: Failed to fetch http://nova.clouds.archive.ubuntu.com/ubuntu/pool/main/d/dnspython/python3-dnspython_1.16.0-1build1_all.deb  Temporary failure resolving 'nova.clouds.archive.ubuntu.com'
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:12:33 WARNING install E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:12:33 ERROR juju.worker.uniter.operation runhook.go:136 hook "install" (via explicit, bespoke hook script) failed: exit status 100
23:12:37 2021-02-03 23:12:36 [ERROR]     2021-02-03 23:12:33 INFO juju.worker.uniter resolver.go:143 awaiting error resolution for "install" hook

[ajkavanagh]

Yes, I've seen that too. It's very strange as it didn't used to happen; I'm assuming an infrastructure change somewhere has led to DNS resolution failures (internally) as it actually resolves to another address that is usually continually available.

[freyes]

We haven't seen this in a while now, I will close the issue.