importing an ontapnas volume panics `trident-main`

[clementnuss]

Describe the bug Importing an ontapnas volume after a simulated restore of the underlying QTree makes trident-main panic.

Environment

• Trident version: 22.04.0
• Container runtime: containerd github.com/containerd/containerd v1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
• Kubernetes version: 1.24.4
• Kubernetes orchestrator: kubeadm
• Kubernetes enabled feature gates: n/a
• OS: Debian 11
• NetApp backend types: ONTAP0 NAS
• Other: Volumes are mounted through NFS

To Reproduce simulate a disaster recovery: delete a PVC with reclaimPolicy delete. after a few minutes the PV and the data is gone from the ONTAP NAS.

Now restore the QTree with your backup software (in this case Veritas NetBackup), and try importing the PVC again.

Expected behavior the PVC/PV pair should be recreated and accessible Additional context

the command used to import the PVC is the following:

/usr/local/bin/tridentctl -n kube-trident import volume ontapnas <volumeAttributes.internalName> -f pvc-data01.yaml

Logs

```logs trident-main time="2022-09-06T12:31:25Z" level=error msg="error getting quota rule" error="tree quota for /vol/trident_qtree_pool_#####_NFGLQBHYKQ/####_pvc_b429450a_43e2_4f15_b2be_aa990f635a98 not found" requestID=2c657968-fabd-46f4-ba19-b7ad2b787758 requestSource=REST trident-main time="2022-09-06T12:31:25Z" level=info msg="Imported an existing volume." handler=ImportVolume requestID=2c657968-fabd-46f4-ba19-b7ad2b787758 requestSource=REST trident-main 2022/09/06 12:31:25 http: panic serving 127.0.0.1:51240: invalid WriteHeader code 0 trident-main goroutine 2351402 [running]: trident-main net/http.(*conn).serve.func1() trident-main /usr/local/go/src/net/http/server.go:1802 +0xb9 trident-main panic({0x2c9ab20, 0xc001c8fe30}) trident-main /usr/local/go/src/runtime/panic.go:1047 +0x266 trident-main net/http.checkWriteHeaderCode(...) trident-main /usr/local/go/src/net/http/server.go:1089 trident-main net/http.(*response).WriteHeader(0xc0009e27e0, 0x0) trident-main /usr/local/go/src/net/http/server.go:1123 +0x5c8 trident-main github.com/netapp/trident/frontend/rest.(*loggingResponseWriter).WriteHeader(0x2f562a0, 0xc00175bcf8) trident-main /go/src/github.com/netapp/trident/frontend/rest/logger.go:28 +0x25 trident-main github.com/netapp/trident/frontend/rest.writeHTTPResponse({0x38453f8, 0xc0016cf830}, {0x3828710, 0xc00175bce0}, {0x2f562a0, 0xc00175bcf8}, 0xc0bded5e192f6faa) trident-main /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:81 +0x252 trident-main github.com/netapp/trident/frontend/rest.AddGeneric.func1() trident-main /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:192 +0x136 trident-main panic({0x2e4a140, 0x57d69e0}) trident-main /usr/local/go/src/runtime/panic.go:1038 +0x215 trident-main github.com/netapp/trident/storage_drivers/ontap/api/azgo.(*QuotaEntryType).Volume(...) trident-main /go/src/github.com/netapp/trident/storage_drivers/ontap/api/azgo/type-quota-entry.go:168 trident-main github.com/netapp/trident/storage_drivers/ontap/api.OntapAPIZAPI.convertQuota({{0xf, 0x0}}, {0x38453f8, 0xc0016cf830}, {{{0x0, 0x0}, {0x0, 0x0}}, 0x0, 0x0, ...}) trident-main /go/src/github.com/netapp/trident/storage_drivers/ontap/api/abstraction_zapi.go:1641 +0x2a7 trident-main github.com/netapp/trident/storage_drivers/ontap/api.OntapAPIZAPI.QuotaGetEntry({{0x38e21d0, 0xc000a60fd0}}, {0x38453f8, 0xc0016cf830}, {0xc001304270, 0xc0016cf830}, {0xc002056b80, 0x2c}, {0x33c1ee9, 0x4}) trident-main /go/src/github.com/netapp/trident/storage_drivers/ontap/api/abstraction_zapi.go:1622 +0x378 trident-main github.com/netapp/trident/storage_drivers/ontap.(*NASQtreeStorageDriver).GetVolumeExternal(0xc00361a800, {0x38453f8, 0xc0016cf830}, {0xc0038e0740, 0x37}) trident-main /go/src/github.com/netapp/trident/storage_drivers/ontap/ontap_nas_qtree.go:1410 +0xfb trident-main github.com/netapp/trident/storage.(*StorageBackend).GetVolumeExternal(0xc0000ed030, {0x38453f8, 0xc0016cf830}, {0xc0038e0740, 0x37}) trident-main /go/src/github.com/netapp/trident/storage/backend.go:557 +0x157 trident-main github.com/netapp/trident/core.(*TridentOrchestrator).GetVolumeExternal(0xc0007e8000, {0x38453f8, 0xc0016cf830}, {0xc0038e0740, 0x37}, {0xc000a304a0, 0x8}) trident-main /go/src/github.com/netapp/trident/core/orchestrator_core.go:2169 +0x41d trident-main github.com/netapp/trident/frontend/csi/helpers/kubernetes.(*Plugin).ImportVolume(0xc000500800, {0x38453f8, 0xc0016cf830}, 0xc002dd2140) trident-main /go/src/github.com/netapp/trident/frontend/csi/helpers/kubernetes/import.go:86 +0xa2a trident-main github.com/netapp/trident/frontend/rest.ImportVolume.func1({0xc001a39800, 0x1a5, 0x200}) trident-main /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:681 +0x252 trident-main github.com/netapp/trident/frontend/rest.AddGeneric({0x3828710, 0xc00175bce0}, 0xc00373da00, {0x3845938, 0xc00175bcf8}, 0xc001077820) trident-main /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:206 +0x36a trident-main github.com/netapp/trident/frontend/rest.ImportVolume({0x3828710, 0xc00175bce0}, 0xc00373da00) trident-main /go/src/github.com/netapp/trident/frontend/rest/controller_handlers.go:658 +0x85 trident-main net/http.HandlerFunc.ServeHTTP(0x0, {0x3828710, 0xc00175bce0}, 0x17) trident-main /usr/local/go/src/net/http/server.go:2047 +0x2f trident-main github.com/netapp/trident/frontend/rest.Logger.func1({0x3833f30, 0xc0009e27e0}, 0xc00373d900) trident-main /go/src/github.com/netapp/trident/frontend/rest/logger.go:44 +0x3c3 trident-main net/http.HandlerFunc.ServeHTTP(0xc00373d800, {0x3833f30, 0xc0009e27e0}, 0x580d880) trident-main /usr/local/go/src/net/http/server.go:2047 +0x2f trident-main github.com/gorilla/mux.(*Router).ServeHTTP(0xc0002f3680, {0x3833f30, 0xc0009e27e0}, 0xc00373d700) trident-main /go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf trident-main net/http.serverHandler.ServeHTTP({0x3825098}, {0x3833f30, 0xc0009e27e0}, 0xc00373d700) trident-main /usr/local/go/src/net/http/server.go:2879 +0x43b trident-main net/http.(*conn).serve(0xc0007e8500, {0x38453f8, 0xc0004d22a0}) trident-main /usr/local/go/src/net/http/server.go:1930 +0xb08 trident-main created by net/http.(*Server).Serve trident-main /usr/local/go/src/net/http/server.go:3034 +0x4e8 ```

[gnarl]

Hi @clementnuss.

From the error message it appears that the qtree was created using the ontap-nas-economy driver and that you are trying to use to use the ontap-nas economy driver to import a copy of a qtree that was originally created by ONTAP in another FlexVol. The first error in the provided log appears to be trying to find the "tree quota" for an ontap-nas-economy volume/qtree that no longer exists.

Trident does support importing volumes created by the ontap-nas driver. The ontap-nas driver is not designed to work with qtrees though. Support for qtrees only exists in the ontap-nas-economy driver.

Let us know what you are attempting to do and if I've missed anything in the above comment.

[clementnuss]

hi @gnarl

From the error message it appears that the qtree was created using the ontap-nas-economy driver and that you are trying to use to use the ontap-nas economy driver to import a copy of a qtree that was originally created by ONTAP in another FlexVol.

partially correct: it's still on the same FlexVol. the following happened:

• creation of a PVC/PV
• trident (with backend ontap-nas-economy) created the tridentvolume and the underlying storage QTree
• backup of the QTree
• deleted the PVC, which deleted the PV, the tridentvolume and the underlying QTree and Quota on ONTAP
• restored the backup (made with Veritas NetBackup) on the ONTAP NAS, in the exact location that was previously deleted (on the same FlexVol, with the same name)

  The first error in the provided log appears to be trying to find the "tree quota" for an ontap-nas-economy volume/qtree that no longer exists.

as qtree was deleted by trident, the corresponding quota was also deleted. I guess when we restore the quota isn't properly restored ? but is the quota actually mandatory ?

Trident does support importing volumes created by the ontap-nas driver. The ontap-nas driver is not designed to work with qtrees though. Support for qtrees only exists in the ontap-nas-economy driver.

we are working with the ontap-nas-economy backend all along the way.

does that make things clearer ?

[clementnuss]

hi @gnarl

isn't my issue related to #343 ? import a QTree isn't supported ?

[scaleoutsean]

import a QTree isn't supported ?

ontap-nas-economy isn't listed as supported for volume import.

https://docs.netapp.com/us-en/trident-2301/trident-use/vol-import.html