search

canonical / charm-microceph

Charm to deploy/manage microceph
Apache License 2.0
2 stars 9 forks source link

Removing unit does not remove node from cluster #106

Open hemanthnakkina opened 3 months ago

hemanthnakkina commented 3 months ago

Removing a microceph unit does not remove the node from microceph cluster.

Deployed sunbeam 2 node with all roles. Removed the non-primary node using [1] However microceph status on primary node shows the non-primary node (sunbeam-2) as part of cluster. 

$ sudo microceph status
MicroCeph deployment summary:
- sunbeam1 (10.121.193.100)
  Services: mds, mgr, mon, rgw, osd
  Disks: 3
- sunbeam2 (10.121.193.19)
  Services: mds, mgr, mon, rgw
  Disks: 0

[1] https://microstack.run/docs/scaling-down

sabaini commented 2 months ago

Hm, I couldn't immediately reproducing it using charm-microceph, cf. below

Do you still see this behaviour? If so, would you be able to provide steps for reproducing and logs (juju logs, microceph logs and /v/l/syslog ideally)?

juju deploy microceph -n2 --channel reef/candidate --config default-pool-size=1
juju run microceph/2 add-osd loop-spec=4G,1 --wait=2m
juju run microceph/3 add-osd loop-spec=4G,1 --wait=2m
...

ubuntu@cm-0:~/src/charm-microceph$ juju ssh 2 -- sudo microceph status
MicroCeph deployment summary:
- juju-79f944-2 (10.73.128.231)
  Services: mds, mgr, mon, osd
  Disks: 1
- juju-79f944-3 (10.73.128.25)
  Services: mds, mgr, mon, osd
  Disks: 1
Connection to 10.73.128.231 closed.
ubuntu@cm-0:~/src/charm-microceph$ juju status
Model           Controller           Cloud/Region         Version  SLA          Timestamp
microceph-test  localhost-localhost  localhost/localhost  3.5.3    unsupported  08:37:35Z

App        Version  Status  Scale  Charm      Channel         Rev  Exposed  Message
microceph           active      2  microceph  reef/candidate   73  no       

Unit          Workload  Agent  Machine  Public address  Ports  Message
microceph/2*  active    idle   2        10.73.128.231          
microceph/3   active    idle   3        10.73.128.25           

Machine  State    Address        Inst id        Base          AZ  Message
2        started  10.73.128.231  juju-79f944-2  ubuntu@22.04      Running
3        started  10.73.128.25   juju-79f944-3  ubuntu@22.04      Running
ubuntu@cm-0:~/src/charm-microceph$ juju remove-unit microceph/3
WARNING This command will perform the following actions:
will remove unit microceph/3

Continue [y/N]? y
ubuntu@cm-0:~/src/charm-microceph$ juju status
Model           Controller           Cloud/Region         Version  SLA          Timestamp
microceph-test  localhost-localhost  localhost/localhost  3.5.3    unsupported  08:39:08Z

App        Version  Status  Scale  Charm      Channel         Rev  Exposed  Message
microceph           active      1  microceph  reef/candidate   73  no       

Unit          Workload  Agent  Machine  Public address  Ports  Message
microceph/2*  active    idle   2        10.73.128.231          

Machine  State    Address        Inst id        Base          AZ  Message
2        started  10.73.128.231  juju-79f944-2  ubuntu@22.04      Running
ubuntu@cm-0:~/src/charm-microceph$ juju ssh 2 -- sudo microceph status
MicroCeph deployment summary:
- juju-79f944-2 (10.73.128.231)
  Services: mds, mgr, mon, osd
  Disks: 1