During a load test, all charms received a stop, but only scarpe-config got stuck on installing agent.
The charm was running fine and debug-log clearly showed that update-status take place.
It's just the status that wasn't updating.
App Version Status Scale Charm Channel Rev Address Exposed Message
scrape-config n/a waiting 1 prometheus-scrape-config-k8s edge 39 10.152.183.17 no installing agent
Unit Workload Agent Address Ports Message
scrape-config/0* maintenance idle 10.1.79.214
To Reproduce
Deploy the cos-lite load test.
Wait.
Environment
Juju 2.9.34
MicroK8s 1.24/stable (v1.24.10)
Relevant log output
# juju show-status-log scrape-config/0
24 Feb 2023 17:28:44-05:00 juju-unit idle
24 Feb 2023 17:28:44-05:00 workload active
25 Feb 2023 18:01:03-05:00 workload maintenance stopping charm software
25 Feb 2023 18:01:03-05:00 juju-unit executing running stop hook
25 Feb 2023 18:01:06-05:00 juju-unit idle
25 Feb 2023 18:01:10-05:00 juju-unit executing running start hook
25 Feb 2023 18:02:00-05:00 juju-unit idle
25 Feb 2023 18:02:00-05:00 workload maintenance
# two days later, after manually running juju config, the status became correct again
27 Feb 2023 13:42:40-05:00 juju-unit executing running config-changed hook
27 Feb 2023 13:42:40-05:00 workload maintenance Updating scrape jobs and alert rules for all metrics consumer
27 Feb 2023 13:42:40-05:00 workload active
27 Feb 2023 13:42:41-05:00 juju-unit idle
Additional context
For some reason, the stop wasn't followed by upgrade-charm - only start. Twice.
unit-scrape-config-0: 2023-02-25 18:01:03 DEBUG unit.scrape-config/0.juju-log Emitting Juju event update_status.
unit-scrape-config-0: 2023-02-25 18:01:04 DEBUG unit.scrape-config/0.juju-log Operator Framework 2.0.0 up and running.
unit-scrape-config-0: 2023-02-25 18:01:04 DEBUG unit.scrape-config/0.juju-log Emitting Juju event stop.
unit-scrape-config-0: 2023-02-25 18:01:08 INFO juju.cmd running containerAgent [2.9.34 90e2f047763059f0b8a57941ae0907346464aee8 gc go1.19]
unit-scrape-config-0: 2023-02-25 18:01:08 INFO juju.cmd.containeragent.unit start "unit"
unit-scrape-config-0: 2023-02-25 18:01:08 INFO juju.worker.upgradesteps upgrade steps for 2.9.34 have already been run.
unit-scrape-config-0: 2023-02-25 18:01:08 INFO juju.worker.probehttpserver starting http server on [::]:3856
unit-scrape-config-0: 2023-02-25 18:01:08 INFO juju.api cannot resolve "controller-service.controller-uk8s.svc.cluster.local": lookup controller-service.controller-uk8s.svc.cluster.local: operation was canceled
unit-scrape-config-0: 2023-02-25 18:01:08 INFO juju.api connection established to "wss://10.152.183.72:17070/model/02f10d90-7242-4543-89d1-d2bf094a30bf/api"
unit-scrape-config-0: 2023-02-25 18:01:08 INFO juju.worker.apicaller [02f10d] "unit-scrape-config-0" successfully connected to "10.152.183.72:17070"
unit-scrape-config-0: 2023-02-25 18:01:09 INFO juju.api cannot resolve "controller-service.controller-uk8s.svc.cluster.local": lookup controller-service.controller-uk8s.svc.cluster.local: operation was canceled
unit-scrape-config-0: 2023-02-25 18:01:09 INFO juju.api connection established to "wss://10.152.183.72:17070/model/02f10d90-7242-4543-89d1-d2bf094a30bf/api"
unit-scrape-config-0: 2023-02-25 18:01:09 INFO juju.worker.apicaller [02f10d] "unit-scrape-config-0" successfully connected to "10.152.183.72:17070"
unit-scrape-config-0: 2023-02-25 18:01:09 INFO juju.worker.migrationminion migration phase is now: NONE
unit-scrape-config-0: 2023-02-25 18:01:09 INFO juju.worker.logger logger worker started
unit-scrape-config-0: 2023-02-25 18:01:09 INFO juju.worker.leadership scrape-config/0 promoted to leadership of scrape-config
unit-scrape-config-0: 2023-02-25 18:01:09 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-scrape-config-0: 2023-02-25 18:01:10 DEBUG unit.scrape-config/0.juju-log Operator Framework 2.0.0 up and running.
unit-scrape-config-0: 2023-02-25 18:01:10 INFO unit.scrape-config/0.juju-log Running legacy hooks/start.
unit-scrape-config-0: 2023-02-25 18:01:11 DEBUG unit.scrape-config/0.juju-log Operator Framework 2.0.0 up and running.
unit-scrape-config-0: 2023-02-25 18:01:11 DEBUG unit.scrape-config/0.juju-log Charm called itself via hooks/start.
unit-scrape-config-0: 2023-02-25 18:01:12 DEBUG unit.scrape-config/0.juju-log Legacy hooks/start exited with status 0.
unit-scrape-config-0: 2023-02-25 18:01:12 DEBUG unit.scrape-config/0.juju-log Emitting Juju event start.
unit-scrape-config-0: 2023-02-25 18:06:41 DEBUG unit.scrape-config/0.juju-log Operator Framework 2.0.0 up and running.
unit-scrape-config-0: 2023-02-25 18:06:41 DEBUG unit.scrape-config/0.juju-log Emitting Juju event update_status.
Bug Description
During a load test, all charms received a
stop
, but onlyscarpe-config
got stuck oninstalling agent
. The charm was running fine anddebug-log
clearly showed thatupdate-status
take place. It's just the status that wasn't updating.To Reproduce
Environment
Relevant log output
Additional context
For some reason, the
stop
wasn't followed byupgrade-charm
- onlystart
. Twice.