search

warm-metal / container-image-csi-driver

Kubernetes CSI driver for mounting image
MIT License
30 stars 22 forks source link

Panic after normal operation #7

Closed glennpratt closed 3 years ago

glennpratt commented 3 years ago

I had a few successful test pods operating with ephemeral image volumes, but after 24 hours, I hit a consistent panic:

 plugin I0408 18:52:05.573451       1 driver.go:93] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER                                                                                                                                                      
 plugin I0408 18:52:05.577550       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}                                                                                                                    
 plugin I0408 18:52:23.630273       1 node.go:34] request: volume_id:"csi-4f6ad6041b4625e5e585add44e98861b094b966ccc3df6cfd8e05faac735b572" target_path:"/var/snap/microk8s/common/var/lib/kubelet/pods/f0635891-7412-4d86-a6a2-82006a7244da/volumes/kubernete 
 s.io~csi/drupal-code/mount" volume_capability:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > volume_context:<key:"csi.storage.k8s.io/ephemeral" value:"true" > volume_context:<key:"csi.storage.k8s.io/pod.name" value:"drupal-58fdc8f7cd-78xq8" > volume 
 _context:<key:"csi.storage.k8s.io/pod.namespace" value:"test-env-prod" > volume_context:<key:"csi.storage.k8s.io/pod.uid" value:"f0635891-7412-4d86-a6a2-82006a7244da" > volume_context:<key:"csi.storage.k8s.io/serviceAccount.name" value:"default" > volum 
 e_context:<key:"image" value:"REDACTED:ef0f6fda48eec00356401ee8d929e7d2fb9d9b98" > volume_context:<key:"secret" value:"regcred" >                       
 plugin I0408 18:52:23.641371       1 mounter.go:99] no local image found. Pull image "REDACTED:ef0f6fda48eec00356401ee8d929e7d2fb9d9b98"                
 plugin panic: image "REDACTED:ef0f6fda48eec00356401ee8d929e7d2fb9d9b98": not found                                                                      
 plugin goroutine 81 [running]:                                                                                                                                                                                                                                
 plugin github.com/warm-metal/csi-driver-image/pkg/backend/containerd.(*mounter).getImageRootFSChainID(0xc00014c000, 0x1bd90e0, 0xc000422270, 0xc000675340, 0x1b8df80, 0xc000144fa0, 0xc000224000, 0x97, 0x1, 0xc000226480, ...)                               
 plugin     /go/src/csi-driver-image/pkg/backend/containerd/mounter.go:108 +0xe90                                                                                                                                                                              
 plugin github.com/warm-metal/csi-driver-image/pkg/backend/containerd.(*mounter).refSnapshot(0xc00014c000, 0x1bd90e0, 0xc000422270, 0xc000675340, 0x1b8df80, 0xc000144fa0, 0xc0001da0a0, 0x44, 0xc000224000, 0x97, ...)                                        
 plugin     /go/src/csi-driver-image/pkg/backend/containerd/mounter.go:130 +0xe5                                                                                                                                                                               
 plugin github.com/warm-metal/csi-driver-image/pkg/backend/containerd.(*mounter).Mount(0xc00014c000, 0x1bd90e0, 0xc000422270, 0x1b8df80, 0xc000144fa0, 0xc0001da0a0, 0x44, 0xc000224000, 0x97, 0xc000136300, ...)                                              
 plugin     /go/src/csi-driver-image/pkg/backend/containerd/mounter.go:237 +0x2c5                                                                                                                                                                              
 plugin main.nodeServer.NodePublishVolume(0xc00013dfe0, 0x1b9a3e0, 0xc00014c000, 0x1be0960, 0xc00013dfd0, 0x1bd90e0, 0xc000422270, 0xc0000ae180, 0x18, 0x18, ...)                                                                                              
 plugin     /go/src/csi-driver-image/cmd/plugin/node.go:88 +0x422                                                                                                                                                                                              
 plugin github.com/container-storage-interface/spec/lib/go/csi._Node_NodePublishVolume_Handler.func1(0x1bd90e0, 0xc000422270, 0x191ac80, 0xc0000ae180, 0x18, 0x18, 0x7f14ed854c28, 0xc00041d8c0)                                                               
 plugin     /go/pkg/mod/github.com/container-storage-interface/spec@v1.3.0/lib/go/csi/csi.pb.go:5977 +0x89                                                                                                                                                     
 plugin github.com/kubernetes-csi/drivers/pkg/csi-common.logGRPC(0x1bd90e0, 0xc000422270, 0x191ac80, 0xc0000ae180, 0xc00041d8a0, 0xc00041d8c0, 0xc000416ba0, 0x508d86, 0x18b85c0, 0xc000422270)                                                                
 plugin     /go/pkg/mod/github.com/kubernetes-csi/drivers@v1.0.2/pkg/csi-common/utils.go:99 +0x15d                                                                                                                                                             
 plugin github.com/container-storage-interface/spec/lib/go/csi._Node_NodePublishVolume_Handler(0x18a5280, 0xc000219230, 0x1bd90e0, 0xc000422270, 0xc000652600, 0x1a88310, 0x1bd90e0, 0xc000422270, 0xc00017c840, 0x28e)                                        
 plugin     /go/pkg/mod/github.com/container-storage-interface/spec@v1.3.0/lib/go/csi/csi.pb.go:5979 +0x150                                                                                                                                                    
 plugin google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000ca000, 0x1bec140, 0xc000501500, 0xc000580100, 0xc0000c21e0, 0x263c530, 0x0, 0x0, 0x0)                                                                                                          
 plugin     /go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1210 +0x522                                                                                                                                                                                   
 plugin google.golang.org/grpc.(*Server).handleStream(0xc0000ca000, 0x1bec140, 0xc000501500, 0xc000580100, 0x0)                                                                                                                                                
 plugin     /go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:1533 +0xd05                                                                                                                                                                                   
 plugin google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000138b40, 0xc0000ca000, 0x1bec140, 0xc000501500, 0xc000580100)                                                                                                                               
 plugin     /go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:871 +0xa5                                                                                                                                                                                     
 plugin created by google.golang.org/grpc.(*Server).serveStreams.func1                                                                                                                                                                                         
 plugin     /go/pkg/mod/google.golang.org/grpc@v1.34.0/server.go:869 +0x1fd                                                                                                                                                                                    
 plugin stream closed
kitt1987 commented 3 years ago

Thanks again for the tests!

From the panic logs, I found that kubelet requested an abnormal image name <key:"image" value:"REDACTED:ef0f6fda48eec00356401ee8d929e7d2fb9d9b98" >. This is the very first log of the NodePublishVolume RPC call, so I need to make sure the manifest of the test Pod is correct.

Can you help paste the manifest of Pod drupal-58fdc8f7cd-78xq8? A kubectl describe result is also OK.

kitt1987 commented 3 years ago

I'm going to close this issue as it's been inactive for a while. If it still exists, feel free to reopen it or create a new issue. Thanks.