canonical / charmed-openstack-upgrader

Automatic upgrade tool for Charmed Openstack
Apache License 2.0
3 stars 12 forks source link

Segfault when exiting after unit blocked error #474

Open samuelallan72 opened 4 months ago

samuelallan72 commented 4 months ago

I observed this twice, first with 2 ceph-osd units blocked, second with one ceph-mon unit blocked:

$ cou upgrade --auto-approve

...

Running cloud upgrade...
Verify that all OpenStack applications are in idle state ✖
2024-07-02 02:29:41 [ERROR] Unit in blocked: ceph-mon/2
2024-07-02 02:29:41 [ERROR] See the known issues at https://canonical-charmed-openstack-upgrader.readthedocs-hosted.com/en/stable/reference/known-issues/
Segmentation fault (core dumped)
samuelallan72 commented 4 months ago

Also observed when ceph-osd/2 was in error:

$ cou upgrade data-plane --auto-approve --force
...
Running cloud upgrade...
Verify that all OpenStack applications are in idle state ✖
2024-07-11 04:29:50 [ERROR] Unit in error: ceph-osd/2
2024-07-11 04:29:50 [ERROR] See the known issues at https://canonical-charmed-openstack-upgrader.readthedocs-hosted.com/en/stable/reference/known-issues/
Segmentation fault (core dumped)
samuelallan72 commented 4 months ago

I tried to get a traceback, hopefully this helps:

$ PYTHONFAULTHANDLER=1 cou upgrade data-plane --force --auto-approve
...
Running cloud upgrade...
Verify that all OpenStack applications are in idle state ✖
2024-07-11 04:38:48 [ERROR] Unit in error: ceph-osd/2
2024-07-11 04:38:48 [ERROR] See the known issues at https://canonical-charmed-openstack-upgrader.readthedocs-hosted.com/en/stable/reference/known-issues/
Fatal Python error: Segmentation fault

Current thread 0x00007f8b0237c000 (most recent call first):
  Garbage-collecting
  File "/usr/lib/python3.10/asyncio/queues.py", line 164 in get
Segmentation fault (core dumped)

Maybe something like an async thread was still executing when sys.exit was called? Maybe a bug in python itself?

python3.10/now 3.10.12-1~22.04.3 is the version that apt reports is installed. Ubuntu jammy.