Open Raswanth49 opened 6 months ago
2024-03-07T10:55:34.101084725Z stderr F I0307 10:55:34.100796 1 register.go:88] Reporting devices GPU-1c5009f6-3637-cd7d-4955-5eaa038e563e,10,32768,NVIDIA-Tesla V100-SXM2-32GB,false:GPU-809c5c58-ad8f-c998-2447-4ac9befe0fdb,10,32768,NVIDIA-Tesla V100-SXM2-32GB,false:GPU-e1fde3ec-1842-57b1-5862-3585e22923d1,10,32768,NVIDIA-Tesla V100-SXM2-32GB,false:GPU-cd8f49bf-1607-360b-8bf3-3defa6a58bb0,10,32768,NVIDIA-Tesla V100-SXM2-32GB,false: in 2024-03-07 10:55:34.100782062 +0000 UTC m=+600.800399211 2024-03-07T10:55:47.977734126Z stderr F I0307 10:55:47.977559 1 plugin.go:296] Allocate [&ContainerAllocateRequest{DevicesIDs:[GPU-e1fde3ec-1842-57b1-5862-3585e22923d1-3 GPU-809c5c58-ad8f-c998-2447-4ac9befe0fdb-2],}] 2024-03-07T10:55:48.374277561Z stderr F panic: runtime error: invalid memory address or nil pointer dereference 2024-03-07T10:55:48.374309502Z stderr F [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1092953] 2024-03-07T10:55:48.374313183Z stderr F 2024-03-07T10:55:48.374316238Z stderr F goroutine 234 [running]: 2024-03-07T10:55:48.374319213Z stderr F volcano.sh/k8s-device-plugin/pkg/plugin/vgpu.(NvidiaDevicePlugin).Allocate(0xc0001bc280, {0x14cff80, 0xc00045c720}, 0xc0004427c0) 2024-03-07T10:55:48.374321927Z stderr F /go/src/volcano.sh/devices/pkg/plugin/vgpu/plugin.go:313 +0x353 2024-03-07T10:55:48.374324767Z stderr F k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1._DevicePlugin_Allocate_Handler({0x1292040?, 0xc0001bc280}, {0x14cff80, 0xc00045c720}, 0xc000516960, 0x0) 2024-03-07T10:55:48.374327831Z stderr F /go/pkg/mod/k8s.io/kubelet@v0.18.2/pkg/apis/deviceplugin/v1beta1/api.pb.go:1192 +0x170 2024-03-07T10:55:48.374330155Z stderr F google.golang.org/grpc.(Server).processUnaryRPC(0xc000183040, {0x14d4e98, 0xc000322180}, 0xc000148c00, 0xc0001a6180, 0x1ce57f8, 0x0) 2024-03-07T10:55:48.374349455Z stderr F /go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:1082 +0xcab 2024-03-07T10:55:48.374354779Z stderr F google.golang.org/grpc.(Server).handleStream(0xc000183040, {0x14d4e98, 0xc000322180}, 0xc000148c00, 0x0) 2024-03-07T10:55:48.374357667Z stderr F /go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:1405 +0xa13 2024-03-07T10:55:48.374360438Z stderr F google.golang.org/grpc.(Server).serveStreams.func1.1() 2024-03-07T10:55:48.374373665Z stderr F /go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:746 +0x98 2024-03-07T10:55:48.374376854Z stderr F created by google.golang.org/grpc.(*Server).serveStreams.func1 2024-03-07T10:55:48.374379632Z stderr F /go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:744 +0xea
Facing the above error when tried to create a pod with vGPU enabled.
2024-03-07T10:55:34.101084725Z stderr F I0307 10:55:34.100796 1 register.go:88] Reporting devices GPU-1c5009f6-3637-cd7d-4955-5eaa038e563e,10,32768,NVIDIA-Tesla V100-SXM2-32GB,false:GPU-809c5c58-ad8f-c998-2447-4ac9befe0fdb,10,32768,NVIDIA-Tesla V100-SXM2-32GB,false:GPU-e1fde3ec-1842-57b1-5862-3585e22923d1,10,32768,NVIDIA-Tesla V100-SXM2-32GB,false:GPU-cd8f49bf-1607-360b-8bf3-3defa6a58bb0,10,32768,NVIDIA-Tesla V100-SXM2-32GB,false: in 2024-03-07 10:55:34.100782062 +0000 UTC m=+600.800399211 2024-03-07T10:55:47.977734126Z stderr F I0307 10:55:47.977559 1 plugin.go:296] Allocate [&ContainerAllocateRequest{DevicesIDs:[GPU-e1fde3ec-1842-57b1-5862-3585e22923d1-3 GPU-809c5c58-ad8f-c998-2447-4ac9befe0fdb-2],}] 2024-03-07T10:55:48.374277561Z stderr F panic: runtime error: invalid memory address or nil pointer dereference 2024-03-07T10:55:48.374309502Z stderr F [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1092953] 2024-03-07T10:55:48.374313183Z stderr F 2024-03-07T10:55:48.374316238Z stderr F goroutine 234 [running]: 2024-03-07T10:55:48.374319213Z stderr F volcano.sh/k8s-device-plugin/pkg/plugin/vgpu.(NvidiaDevicePlugin).Allocate(0xc0001bc280, {0x14cff80, 0xc00045c720}, 0xc0004427c0) 2024-03-07T10:55:48.374321927Z stderr F /go/src/volcano.sh/devices/pkg/plugin/vgpu/plugin.go:313 +0x353 2024-03-07T10:55:48.374324767Z stderr F k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1._DevicePlugin_Allocate_Handler({0x1292040?, 0xc0001bc280}, {0x14cff80, 0xc00045c720}, 0xc000516960, 0x0) 2024-03-07T10:55:48.374327831Z stderr F /go/pkg/mod/k8s.io/kubelet@v0.18.2/pkg/apis/deviceplugin/v1beta1/api.pb.go:1192 +0x170 2024-03-07T10:55:48.374330155Z stderr F google.golang.org/grpc.(Server).processUnaryRPC(0xc000183040, {0x14d4e98, 0xc000322180}, 0xc000148c00, 0xc0001a6180, 0x1ce57f8, 0x0) 2024-03-07T10:55:48.374349455Z stderr F /go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:1082 +0xcab 2024-03-07T10:55:48.374354779Z stderr F google.golang.org/grpc.(Server).handleStream(0xc000183040, {0x14d4e98, 0xc000322180}, 0xc000148c00, 0x0) 2024-03-07T10:55:48.374357667Z stderr F /go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:1405 +0xa13 2024-03-07T10:55:48.374360438Z stderr F google.golang.org/grpc.(Server).serveStreams.func1.1() 2024-03-07T10:55:48.374373665Z stderr F /go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:746 +0x98 2024-03-07T10:55:48.374376854Z stderr F created by google.golang.org/grpc.(*Server).serveStreams.func1 2024-03-07T10:55:48.374379632Z stderr F /go/pkg/mod/google.golang.org/grpc@v1.29.0/server.go:744 +0xea
Facing the above error when tried to create a pod with vGPU enabled.