search

kubernetes / cloud-provider-openstack

Apache License 2.0
599 stars 597 forks source link

[manila-csi-plugin] Cannot extend pvc #2619

Open stevenyslins opened 3 weeks ago

stevenyslins commented 3 weeks ago

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

What happened:

When I extend the volume, my pvc get Extending Error status

image

What you expected to happen: Resize volume succeeded

How to reproduce it:

  1. Create a share type 
    
$ manila type-create test-share-type true

+----------------------+--------------------------------------+ | Property | Value | +----------------------+--------------------------------------+ | ID | 9aff85eb-dd63-4883-9a1d-a11e9de86943 | | Name | test-share-type | | Visibility | public | | is_default | - | | required_extra_specs | driver_handles_share_servers : True | | optional_extra_specs | | | Description | None | +----------------------+--------------------------------------+


2. Create a share network
Need to create a `neutron-net` and `neutron-subnet`.

$ manila share-network-create \ --name test-share-net \ --neutron-net-id 6e28ce79-6aff-42c1-8a9c-641fb8a5d370 \ --neutron-subnet-id fbeb0e31-b50c-484a-8605-1ff3cdc92e2f

+-----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +-----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | id | 11832582-0c53-4575-a9ef-4073259f03c9 | | name | test-share-net | | project_id | acb78797bb0040dbbb0ddcdcf3db8fef | | created_at | 2024-06-14T12:05:32.852194 | | updated_at | None | | description | None | | status | active | | security_service_update_support | True | | network_allocation_update_support | True | | share_network_subnets | [{'id': '9fdd2a8f-2a54-46be-86b5-0ab261c2d211', 'availability_zone': None, 'created_at': '2024-06-14T12:05:32.865126', 'updated_at': None, 'segmentation_id': None, 'neutron_net_id': '6e28ce79-6aff-42c1-8a9c-641fb8a5d370', 'neutron_subnet_id': 'fbeb0e31-b50c-484a-8605-1ff3cdc92e2f', 'ip_version': None, 'cidr': None, 'network_type': None, 'mtu': None, 'gateway': None, 'metadata': {}}] | +-----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


3. Create a storageclass

storageclass.yaml

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-manila-test-nfs provisioner: nfs.manila.csi.openstack.org allowVolumeExpansion: true parameters:

Manila share type

type: test-share-type shareNetworkID: 11832582-0c53-4575-a9ef-4073259f03c9

csi.storage.k8s.io/provisioner-secret-name: csi-manila-secrets csi.storage.k8s.io/provisioner-secret-namespace: kube-system csi.storage.k8s.io/controller-expand-secret-name: csi-manila-secrets csi.storage.k8s.io/controller-expand-secret-namespace: kube-system csi.storage.k8s.io/node-stage-secret-name: csi-manila-secrets csi.storage.k8s.io/node-stage-secret-namespace: kube-system csi.storage.k8s.io/node-publish-secret-name: csi-manila-secrets csi.storage.k8s.io/node-publish-secret-namespace: kube-system


apply storageclass yaml

$ kubectl apply -f storageclass.yaml


4. Create a persistentVolumeClaim:

pvc.yaml

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: nfs-share-pvc-test spec: accessModes:

apply pvc.yaml

$ kubectl apply -f pvc.yaml
image
  1. Extend the volume to 2Gi, the expansion will get error 
    
$ kubectl get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
nfs-share-pvc-test   Bound    pvc-b9722890-a85a-4e0e-b967-f121cd3b6464   1Gi        RWX            csi-manila-test-nfs   4m57s

$ kubectl -n default patch pvc nfs-share-pvc-test -p '{"spec":{"resources":{"requests":{"storage":"2Gi"}}}}' persistentvolumeclaim/nfs-share-pvc-test patched

$ kubectl describe pvc nfs-share-pvc-test Name: nfs-share-pvc-test Namespace: default StorageClass: csi-manila-test-nfs Status: Bound Volume: pvc-b9722890-a85a-4e0e-b967-f121cd3b6464 Labels: Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-provisioner: nfs.manila.csi.openstack.org volume.kubernetes.io/storage-provisioner: nfs.manila.csi.openstack.org Finalizers: [kubernetes.io/pvc-protection] Capacity: 1Gi Access Modes: RWX VolumeMode: Filesystem Used By: Conditions: Type Status LastProbeTime LastTransitionTime Reason Message

Resizing True Mon, 01 Jan 0001 00:00:00 +0000 Fri, 14 Jun 2024 12:28:44 +0000 Events: Type Reason Age From Message

Warning ProvisioningFailed 7m28s nfs.manila.csi.openstack.org_manila-csi-openstack-manila-csi-controllerplugin-0_f3013426-0272-462e-8349-653303248acc failed to provision volume with StorageClass "csi-manila-test-nfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded Warning ProvisioningFailed 6m41s (x6 over 7m27s) nfs.manila.csi.openstack.org_manila-csi-openstack-manila-csi-controllerplugin-0_f3013426-0272-462e-8349-653303248acc failed to provision volume with StorageClass "csi-manila-test-nfs": rpc error: code = Aborted desc = volume pvc-b9722890-a85a-4e0e-b967-f121cd3b6464 is already being processed Normal ExternalProvisioning 5m41s (x9 over 7m38s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "nfs.manila.csi.openstack.org" or manually created by system administrator Normal Provisioning 5m37s (x8 over 7m38s) nfs.manila.csi.openstack.org_manila-csi-openstack-manila-csi-controllerplugin-0_f3013426-0272-462e-8349-653303248acc External provisioner is provisioning volume for claim "default/nfs-share-pvc-test" Normal ProvisioningSucceeded 5m36s nfs.manila.csi.openstack.org_manila-csi-openstack-manila-csi-controllerplugin-0_f3013426-0272-462e-8349-653303248acc Successfully provisioned volume pvc-b9722890-a85a-4e0e-b967-f121cd3b6464 Warning ExternalExpanding 48s volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. Warning VolumeResizeFailed 43s external-resizer nfs.manila.csi.openstack.org resize volume "pvc-b9722890-a85a-4e0e-b967-f121cd3b6464" by resizer "nfs.manila.csi.openstack.org" failed: rpc error: code = Internal desc = failed to resize volume pvc-b9722890-a85a-4e0e-b967-f121cd3b6464: share 1357f637-c7f7-44cd-8919-498c7a14f574 is in error state "extending_error": extend: Share Driver failed to extend share, The share status has been set to extending_error. This action cannot be re-attempted until the status has been rectified. Contact your administrator to determine the cause of this failure. Normal Resizing 31s (x6 over 48s) external-resizer nfs.manila.csi.openstack.org External resizer is resizing volume pvc-b9722890-a85a-4e0e-b967-f121cd3b6464 Warning VolumeResizeFailed 30s (x5 over 41s) external-resizer nfs.manila.csi.openstack.org resize volume "pvc-b9722890-a85a-4e0e-b967-f121cd3b6464" by resizer "nfs.manila.csi.openstack.org" failed: rpc error: code = Unknown desc = Bad request with: [POST http://10.78.19.155:8786/v2/shares/1357f637-c7f7-44cd-8919-498c7a14f574/action], error message: {"badRequest": {"code": 400, "message": "Invalid share: Share 1357f637-c7f7-44cd-8919-498c7a14f574 status must be 'available' to extend, but current status is: extending_error."}}


**Anything else we need to know?**:

**Environment**:
- openstack-cloud-controller-manager(or other related binary) version: v1.25.3
- OpenStack version:

$ nova --version 18.3.0


- Others:
* kubernetes version : v1.24.10
* [csi-driver-nfs](https://github.com/kubernetes-csi/csi-driver-nfs/tree/v4.7.0/charts/v4.7.0) : v4.7.0
* [manila-csi-plugin](https://github.com/kubernetes/cloud-provider-openstack/tree/v1.30.0/charts/manila-csi-plugin) : v1.30.0
gouthampacha commented 3 weeks ago

hey @stevenyslins

The request failed in manila; it appears you have access to the manila CLI; could you try this:

manila message-list

(or openstack share message list)

maybe there's some detail there around why the share wasn't extended as you expected.

If there's something failing on the backend, you might need to contract your cloud provider.

stevenyslins commented 2 weeks ago

Hi @gouthampacha

My instruction is as follows, and I am also the administrator of this OpenStack installation.

Thank you for your response.

$ manila message-list
manila CLI is deprecated and will be removed in the future. Use openstack CLI instead. The equivalent command is " openstack share message list "
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| ID                                   | Resource Type | Resource ID                          | Action ID | User Message                                                                                                                                                                                                                              | Detail ID | Created At                 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| 8af43d1c-1050-4e57-8fda-763e894b2cf5 | SHARE         | 1357f637-c7f7-44cd-8919-498c7a14f574 | 008       | extend: Share Driver failed to extend share, The share status has been set to extending_error. This action cannot be re-attempted until the status has been rectified. Contact your administrator to determine the cause of this failure. | 015       | 2024-06-14T12:28:31.276346 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+

$ manila list
manila CLI is deprecated and will be removed in the future. Use openstack CLI instead. The equivalent command is " openstack share list "
+--------------------------------------+------------------------------------------+------+-------------+-----------------+-----------+-----------------+----------------------+-------------------+
| ID                                   | Name                                     | Size | Share Proto | Status          | Is Public | Share Type Name | Host                 | Availability Zone |
+--------------------------------------+------------------------------------------+------+-------------+-----------------+-----------+-----------------+----------------------+-------------------+
| 1357f637-c7f7-44cd-8919-498c7a14f574 | pvc-b9722890-a85a-4e0e-b967-f121cd3b6464 | 1    | NFS         | extending_error | False     | test-share-type | test01@generic       | nova              |
+--------------------------------------+------------------------------------------+------+-------------+-----------------+-----------+-----------------+----------------------+-------------------+
gouthampacha commented 2 weeks ago

My instruction is as follows, and I am also the administrator of this OpenStack installation.

That's great; because from the error message below; it looks like an ir-recoverrable error occurred on the backend driver's attempt to extend the share. 


$ manila message-list
manila CLI is deprecated and will be removed in the future. Use openstack CLI instead. The equivalent command is " openstack share message list "
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| ID                                   | Resource Type | Resource ID                          | Action ID | User Message                                                                                                                                                                                                                              | Detail ID | Created At                 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| 8af43d1c-1050-4e57-8fda-763e894b2cf5 | SHARE         | 1357f637-c7f7-44cd-8919-498c7a14f574 | 008       | extend: Share Driver failed to extend share, The share status has been set to extending_error. This action cannot be re-attempted until the status has been rectified. Contact your administrator to determine the cause of this failure. | 015       | 2024-06-14T12:28:31.276346 |
+--------------------------------------+---------------+--------------------------------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------+

Can you look at your manila-share log file and determine happened here? From the o/p it appears you are using the Generic driver - which creates cinder volumes and exports them as NFS/CIFS shares. Maybe the extension request failed to get communicated to cinder? Or maybe it failed during the interaction? Does this always occur?

stevenyslins commented 2 weeks ago

At 2024-06-14 12:28, I grepped the manila-share.log as follows:

# cat /var/log/kolla/manila/manila-share.log |grep -i "2024-06-14 12:28"
2024-06-14 12:28:23.263 16 INFO manila.share.manager [None req-8caafbd0-45f7-4b72-b47c-cce6eb7695f0 - - - - - -] Updating share status
2024-06-14 12:28:29.707 16 ERROR oslo.messaging._drivers.impl_rabbit [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] [23ea5473-fbd5-4cb6-a2ba-c6db0d1ee280] AMQP server on 10.78.19.158:5672 is unreachable: connection already closed. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: connection already closed
2024-06-14 12:28:30.713 16 INFO oslo.messaging._drivers.impl_rabbit [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] [23ea5473-fbd5-4cb6-a2ba-c6db0d1ee280] Reconnected to AMQP server on 10.78.19.158:5672 via [amqp] client with port 56084.
2024-06-14 12:28:30.716 16 ERROR oslo.messaging._drivers.impl_rabbit [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Failed to publish message to topic 'openstack': Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo.messaging._drivers.impl_rabbit [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true': amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Could not send notification to notifications. Payload={'message_id': 'd5a29e94-efb7-437c-8905-13f42e74b0c5', 'publisher_id': 'share.test@generic', 'event_type': 'share.extend.start', 'priority': 'INFO', 'payload': {'share_id': '1357f637-c7f7-44cd-8919-498c7a14f574', 'user_id': '9ba82382877246a28fb104f1370a3101', 'project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', 'snapshot_id': None, 'share_group_id': None, 'size': 1, 'name': 'pvc-b9722890-a85a-4e0e-b967-f121cd3b6464', 'description': 'provisioned-by=manila.csi.openstack.org', 'proto': 'NFS', 'is_public': False, 'availability_zone': 'nova', 'host': 'test@generic', 'status': 'extending', 'share_type_id': '9aff85eb-dd63-4883-9a1d-a11e9de86943', 'share_type': 'test-share-type'}, 'timestamp': '2024-06-14 12:28:29.706843', '_unique_id': 'da7580ad3b3a407da12b317398fb9f2f', '_context_user': '9ba82382877246a28fb104f1370a3101', '_context_project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_system_scope': None, '_context_project': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_domain': None, '_context_user_domain': None, '_context_project_domain': None, '_context_is_admin': True, '_context_read_only': False, '_context_show_deleted': False, '_context_auth_token': None, '_context_request_id': 'req-d06ace44-3924-4a4a-85b8-2338ff4b06e3', '_context_global_request_id': None, '_context_resource_uuid': None, '_context_roles': ['heat_stack_owner', 'reader', 'member', 'admin', 'manager'], '_context_user_identity': '9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - -', '_context_is_admin_project': True, '_context_user_id': '9ba82382877246a28fb104f1370a3101', '_context_project_name': 'admin', '_context_domain_id': None, '_context_read_deleted': 'no', '_context_remote_address': None, '_context_timestamp': '2024-06-14T12:28:29.706802', '_context_quota_class': None, '_context_service_catalog': []}: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1005, in ensure
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     ret, channel = autoretry_method()
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/connection.py", line 523, in _ensured
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return fun(*args, **kwargs)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/connection.py", line 599, in __call__
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return fun(*args, channel=channels[0], **kwargs), channels[0]
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 994, in execute_method
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     method()
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1477, in _publish_and_creates_default_queue
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     queue.declare()
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/entity.py", line 606, in declare
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self._create_queue(nowait=nowait, channel=channel)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/entity.py", line 615, in _create_queue
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self.queue_declare(nowait=nowait, passive=False, channel=channel)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/kombu/entity.py", line 643, in queue_declare
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     ret = channel.queue_declare(
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/channel.py", line 1162, in queue_declare
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return queue_declare_ok_t(*self.wait(
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/abstract_channel.py", line 99, in wait
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self.connection.drain_events(timeout=timeout)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/connection.py", line 525, in drain_events
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     while not self.blocking_read(timeout):
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/connection.py", line 531, in blocking_read
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return self.on_inbound_frame(frame)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/method_framing.py", line 53, in on_frame
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     callback(channel, method_sig, buf, None)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/connection.py", line 537, in on_inbound_method
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return self.channels[channel_id].dispatch_method(
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     listener(*args)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/amqp/channel.py", line 293, in _on_close
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     raise error_for_code(
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging amqp.exceptions.PreconditionFailed: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging During handling of the above exception, another exception occurred:
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/notify/messaging.py", line 78, in notify
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self.transport._send_notification(target, ctxt, message,
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/transport.py", line 134, in _send_notification
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self._driver.send_notification(target, ctxt, message, version,
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 694, in send_notification
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     return self._send(target, ctxt, message,
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 659, in _send
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     conn.notify_send(exchange, target.topic, msg, retry=retry)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1553, in notify_send
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self._ensure_publishing(self._publish_and_creates_default_queue,
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1383, in _ensure_publishing
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     self.ensure(method, retry=retry, error_callback=_error_callback)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1027, in ensure
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging     raise exceptions.MessageDeliveryFailure(msg)
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging
2024-06-14 12:28:31.272 16 ERROR manila.share.manager [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Extend share failed.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Traceback (most recent call last):
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 4483, in extend_share
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     self.driver.extend_share(
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 115, in wrap
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return f(self, context, *args, **kwargs)
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 660, in extend_share
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     helper.disable_access_for_maintenance(server_details, share['name'])
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 178, in wrapped_func
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return source_func(self, *args, **kwargs)
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return f(*args, **kwargs)
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 176, in source_func
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return f(self, *args, **kwargs)
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 385, in disable_access_for_maintenance
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     self._ssh_exec(server,
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 168, in _ssh_exec
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     return processutils.ssh_execute(ssh, ' '.join(command),
2024-06-14 12:28:31.272 16 ERROR manila.share.manager   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/processutils.py", line 583, in ssh_execute
2024-06-14 12:28:31.272 16 ERROR manila.share.manager     raise ProcessExecutionError(exit_code=exit_status,
2024-06-14 12:28:31.272 16 ERROR manila.share.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Command: sudo exportfs -u "<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8"
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Exit code: 1
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Stdout: ''
2024-06-14 12:28:31.272 16 ERROR manila.share.manager Stderr: "exportfs: Could not find '<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8' to unexport.\n"
2024-06-14 12:28:31.272 16 ERROR manila.share.manager
2024-06-14 12:28:31.273 16 INFO manila.message.api [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Creating message record for request_id = req-d06ace44-3924-4a4a-85b8-2338ff4b06e3
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Exception during message handling: manila.exception.ShareExtendingError: Share 1357f637-c7f7-44cd-8919-498c7a14f574 could not be extended due to error in the driver: Unexpected error while running command.
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 4483, in extend_share
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     self.driver.extend_share(
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 115, in wrap
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 660, in extend_share
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     helper.disable_access_for_maintenance(server_details, share['name'])
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 178, in wrapped_func
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return source_func(self, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 176, in source_func
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/helpers.py", line 385, in disable_access_for_maintenance
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     self._ssh_exec(server,
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/drivers/generic.py", line 168, in _ssh_exec
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return processutils.ssh_execute(ssh, ' '.join(command),
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/processutils.py", line 583, in ssh_execute
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     raise ProcessExecutionError(exit_code=exit_status,
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Command: sudo exportfs -u "<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8"
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Exit code: 1
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stdout: ''
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stderr: "exportfs: Could not find '<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8' to unexport.\n"
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 236, in wrapped
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return f(self, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/utils.py", line 579, in wrapper
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.10/site-packages/manila/share/manager.py", line 4499, in extend_share
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server     raise exception.ShareExtendingError(
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server manila.exception.ShareExtendingError: Share 1357f637-c7f7-44cd-8919-498c7a14f574 could not be extended due to error in the driver: Unexpected error while running command.
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Command: sudo exportfs -u "<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8"
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Exit code: 1
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stdout: ''
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stderr: "exportfs: Could not find '<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8' to unexport.\n"
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server
# cat /var/log/kolla/manila/manila-share.log |grep -i "1357f637-c7f7-44cd-8919-498c7a14f574"
2024-06-14 12:21:40.049 16 ERROR oslo_messaging.notify.messaging [None req-c5e13e90-6c61-4712-8dc3-3cfe675ed6c5 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Could not send notification to notifications. Payload={'message_id': '75ca8778-fa63-48a3-9a06-10630f45ecb7', 'publisher_id': 'share.test@generic', 'event_type': 'share.create.start', 'priority': 'INFO', 'payload': {'share_id': '1357f637-c7f7-44cd-8919-498c7a14f574', 'user_id': '9ba82382877246a28fb104f1370a3101', 'project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', 'snapshot_id': None, 'share_group_id': None, 'size': 1, 'name': 'pvc-b9722890-a85a-4e0e-b967-f121cd3b6464', 'description': 'provisioned-by=manila.csi.openstack.org', 'proto': 'NFS', 'is_public': False, 'availability_zone': None, 'host': 'test@generic', 'status': 'creating', 'share_type_id': '9aff85eb-dd63-4883-9a1d-a11e9de86943', 'share_type': 'test-share-type'}, 'timestamp': '2024-06-14 12:21:39.038772', '_unique_id': '971cb0b2153c4e5e86aba4e7456af06b', '_context_user': '9ba82382877246a28fb104f1370a3101', '_context_project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_system_scope': None, '_context_project': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_domain': None, '_context_user_domain': None, '_context_project_domain': None, '_context_is_admin': True, '_context_read_only': False, '_context_show_deleted': False, '_context_auth_token': None, '_context_request_id': 'req-c5e13e90-6c61-4712-8dc3-3cfe675ed6c5', '_context_global_request_id': None, '_context_resource_uuid': None, '_context_roles': ['admin', 'heat_stack_owner', 'member', 'reader', 'manager'], '_context_user_identity': '9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - -', '_context_is_admin_project': True, '_context_user_id': '9ba82382877246a28fb104f1370a3101', '_context_project_name': 'admin', '_context_domain_id': None, '_context_read_deleted': 'no', '_context_remote_address': None, '_context_timestamp': '2024-06-14T12:21:39.038699', '_context_quota_class': None, '_context_service_catalog': []}: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:23:12.179 16 ERROR oslo_messaging.notify.messaging [None req-c5e13e90-6c61-4712-8dc3-3cfe675ed6c5 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Could not send notification to notifications. Payload={'message_id': '2b94a873-1dcc-4295-af01-96cbb598b42a', 'publisher_id': 'share.test@generic', 'event_type': 'share.create.end', 'priority': 'INFO', 'payload': {'share_id': '1357f637-c7f7-44cd-8919-498c7a14f574', 'user_id': '9ba82382877246a28fb104f1370a3101', 'project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', 'snapshot_id': None, 'share_group_id': None, 'size': 1, 'name': 'pvc-b9722890-a85a-4e0e-b967-f121cd3b6464', 'description': 'provisioned-by=manila.csi.openstack.org', 'proto': 'NFS', 'is_public': False, 'availability_zone': 'nova', 'host': 'test@generic', 'status': 'creating', 'share_type_id': '9aff85eb-dd63-4883-9a1d-a11e9de86943', 'share_type': 'test-share-type'}, 'timestamp': '2024-06-14 12:23:11.170967', '_unique_id': 'cddf1670fe3340728d7e39af05a2e666', '_context_user': '9ba82382877246a28fb104f1370a3101', '_context_project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_system_scope': None, '_context_project': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_domain': None, '_context_user_domain': None, '_context_project_domain': None, '_context_is_admin': True, '_context_read_only': False, '_context_show_deleted': False, '_context_auth_token': None, '_context_request_id': 'req-c5e13e90-6c61-4712-8dc3-3cfe675ed6c5', '_context_global_request_id': None, '_context_resource_uuid': None, '_context_roles': ['admin', 'heat_stack_owner', 'member', 'reader', 'manager'], '_context_user_identity': '9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - -', '_context_is_admin_project': True, '_context_user_id': '9ba82382877246a28fb104f1370a3101', '_context_project_name': 'admin', '_context_domain_id': None, '_context_read_deleted': 'no', '_context_remote_address': None, '_context_timestamp': '2024-06-14T12:23:11.170933', '_context_quota_class': None, '_context_service_catalog': []}: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:23:41.049 16 INFO manila.share.access [None req-30525a3b-ae4e-4d17-b82f-ada3aece2c10 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Access rules were successfully modified for share instance 6acebb21-b851-466b-affe-2fcfd85efde8 belonging to share 1357f637-c7f7-44cd-8919-498c7a14f574.
2024-06-14 12:28:30.717 16 ERROR oslo_messaging.notify.messaging [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Could not send notification to notifications. Payload={'message_id': 'd5a29e94-efb7-437c-8905-13f42e74b0c5', 'publisher_id': 'share.test@generic', 'event_type': 'share.extend.start', 'priority': 'INFO', 'payload': {'share_id': '1357f637-c7f7-44cd-8919-498c7a14f574', 'user_id': '9ba82382877246a28fb104f1370a3101', 'project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', 'snapshot_id': None, 'share_group_id': None, 'size': 1, 'name': 'pvc-b9722890-a85a-4e0e-b967-f121cd3b6464', 'description': 'provisioned-by=manila.csi.openstack.org', 'proto': 'NFS', 'is_public': False, 'availability_zone': 'nova', 'host': 'test@generic', 'status': 'extending', 'share_type_id': '9aff85eb-dd63-4883-9a1d-a11e9de86943', 'share_type': 'test-share-type'}, 'timestamp': '2024-06-14 12:28:29.706843', '_unique_id': 'da7580ad3b3a407da12b317398fb9f2f', '_context_user': '9ba82382877246a28fb104f1370a3101', '_context_project_id': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_system_scope': None, '_context_project': 'acb78797bb0040dbbb0ddcdcf3db8fef', '_context_domain': None, '_context_user_domain': None, '_context_project_domain': None, '_context_is_admin': True, '_context_read_only': False, '_context_show_deleted': False, '_context_auth_token': None, '_context_request_id': 'req-d06ace44-3924-4a4a-85b8-2338ff4b06e3', '_context_global_request_id': None, '_context_resource_uuid': None, '_context_roles': ['heat_stack_owner', 'reader', 'member', 'admin', 'manager'], '_context_user_identity': '9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - -', '_context_is_admin_project': True, '_context_user_id': '9ba82382877246a28fb104f1370a3101', '_context_project_name': 'admin', '_context_domain_id': None, '_context_read_deleted': 'no', '_context_remote_address': None, '_context_timestamp': '2024-06-14T12:28:29.706802', '_context_quota_class': None, '_context_service_catalog': []}: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.78.19.158:5672 after inf tries: Queue.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for queue 'notifications.info' in vhost '/': received 'false' but current is 'true'
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server [None req-d06ace44-3924-4a4a-85b8-2338ff4b06e3 9ba82382877246a28fb104f1370a3101 acb78797bb0040dbbb0ddcdcf3db8fef - - - -] Exception during message handling: manila.exception.ShareExtendingError: Share 1357f637-c7f7-44cd-8919-498c7a14f574 could not be extended due to error in the driver: Unexpected error while running command.
2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server manila.exception.ShareExtendingError: Share 1357f637-c7f7-44cd-8919-498c7a14f574 could not be extended due to error in the driver: Unexpected error while running command.

And in the Cinder log, I cannot find any helpful information.

# cat /var/log/kolla/cinder/cinder-volume.log |grep -i "pvc-b9722890-a85a-4e0e-b967-f121cd3b6464"
<no log>

# cat /var/log/kolla/cinder/cinder-volume.log |grep -i "2024-06-14 12:28"
<no log>

Additionally, when I use the Manila CLI to extend the volume, it works.

## Extend the volume to 2Gi
## command : manila extend <share> <new_size>
$ manila extend 1357f637-c7f7-44cd-8919-498c7a14f574 2
gouthampacha commented 2 weeks ago

Thanks; from the error its apparent that exportfs gave manila an error:

2024-06-14 12:28:31.331 16 ERROR oslo_messaging.rpc.server Stderr: "exportfs: Could not find '<world>:/shares/share-6acebb21-b851-466b-affe-2fcfd85efde8' to unexport.\n"

when performing an unmount operation; which is strange because the list of exports to remove is compiled by executing exportfs: https://opendev.org/openstack/manila/src/commit/0999857c47fca78b0d0a2f8059cd5aa1527cb422/manila/share/drivers/helpers.py#L382

Does this always happen when the extension is requested via manila-csi? The reason I ask is because there is no difference in the API calls between manila-csi and the manila CLI you're using:

https://github.com/gophercloud/gophercloud/blob/3bb883c4ebe42c7eb9382406e4b3f9e1cc3743ca/openstack/sharedfilesystems/v2/shares/requests.go#L310 https://github.com/openstack/python-manilaclient/blob/85f21732176ed6218e4b8b64ac3261dd40f54c7a/manilaclient/v2/shares.py#L741-L744

To triage the exportfs behavior, I'm curious what's the operating system of the manila service image in your environment? and what version?

stevenyslins commented 1 week ago

Does this always happen when the extension is requested via manila-csi?

Yes, when using the manila-csi-plugin, the status always results in Extending Error.

To triage the exportfs behavior, I'm curious what's the operating system of the manila service image in your environment? and what version?

Our environment runs the Manila service on the following operating system:

# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

And manila image as follows:

# docker ps -a |grep -i manila
ff5a572dc04c   quay.io/openstack.kolla/manila-scheduler:2023.2-ubuntu-jammy                  "dumb-init --single-…"   2 weeks ago   Up 7 days (healthy)                 manila_scheduler
3a26a6b15310   quay.io/openstack.kolla/manila-data:2023.2-ubuntu-jammy                       "dumb-init --single-…"   2 weeks ago   Up 7 days (healthy)                 manila_data
ceaf386e0a11   quay.io/openstack.kolla/manila-share:2023.2-ubuntu-jammy                      "dumb-init --single-…"   3 weeks ago   Up 7 days (healthy)                 manila_share
086a858d35f3   quay.io/openstack.kolla/manila-api:2023.2-ubuntu-jammy                        "dumb-init --single-…"   3 weeks ago   Up 7 days (healthy)                 manila_api

Thank you for your response.