search

LINBIT / linstor-gateway

Manages Highly-Available iSCSI targets, NVMe-oF targets, and NFS exports via LINSTOR
GNU General Public License v3.0
28 stars 6 forks source link

ISCSI EOF #15

Closed ingvarrwvw closed 1 year ago

ingvarrwvw commented 1 year ago

Hi.

After removing / adding new satellites, the gateway often stops working and gives the following error:

linstor-gateway iscsi list

Error: Get "http://localhost:8080/api/v2/iscsi": EOF

curl http://localhost:8080/api/v2/iscsi curl: (52) Empty reply from server

although at the moment there are iscsi-resources that are working and being used

linstor-gateway version 1.0.0

chrboe commented 1 year ago

Hi, thank you for the report.

I'm assuming some kind of panic happened on the backend.

Can you check your journalctl if you see any unusual error messages from linstor-gatway?

ingvarrwvw commented 1 year ago

Yes, you are right:

time="2022-12-05T10:12:53Z" level=warning msg="failed to fetch deployed resources" error="failed to fetch resource definition: 404 Not Found" 2022/12/05 10:12:53 http: panic serving [::1]:46910: runtime error: invalid memory address or nil pointer dereference goroutine 18 [running]: net/http.(*conn).serve.func1()

011/opt/hostedtoolcache/go/1.18.7/x64/src/net/http/server.go:1825 +0xbf

panic({0x97aac0, 0xe25ed0})

011/opt/hostedtoolcache/go/1.18.7/x64/src/runtime/panic.go:844 +0x258

github.com/LINBIT/linstor-gateway/pkg/iscsi.FromPromoter(0xc0002ea1c0?, 0x0, {0x0, 0x0, 0xc000130080?})

011/home/runner/work/linstor-gateway/linstor-gateway/pkg/iscsi/resource_config.go:132 +0x52

github.com/LINBIT/linstor-gateway/pkg/iscsi.(*ISCSI).List(0xc000010230, {0xaf3dc0, 0xc000482960})

011/home/runner/work/linstor-gateway/linstor-gateway/pkg/iscsi/iscsi.go:202 +0x3fa

github.com/LINBIT/linstor-gateway/pkg/rest.(*server).ISCSIList.func1({0xaf3928, 0xc0003f22a0}, 0x0?)

011/home/runner/work/linstor-gateway/linstor-gateway/pkg/rest/iscsi_list.go:12 +0x6b

net/http.HandlerFunc.ServeHTTP(0x951300?, {0xaf3928?, 0xc0003f22a0?}, 0xc?)

011/opt/hostedtoolcache/go/1.18.7/x64/src/net/http/server.go:2084 +0x2f

github.com/LINBIT/linstor-gateway/pkg/rest.(*server).routes.func1.1({0xaf3928, 0xc0003f22a0}, 0xc000482960?)

011/home/runner/work/linstor-gateway/linstor-gateway/pkg/rest/routes.go:10 +0x11a

net/http.HandlerFunc.ServeHTTP(0xc0003c0800?, {0xaf3928?, 0xc0003f22a0?}, 0xba007fdb283ac550?)

011/opt/hostedtoolcache/go/1.18.7/x64/src/net/http/server.go:2084 +0x2f

github.com/gorilla/mux.(*Router).ServeHTTP(0xc0002ac000, {0xaf3928, 0xc0003f22a0}, 0xc0003c0700)

011/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf

github.com/rs/cors.(*Cors).Handler.func1({0xaf3928, 0xc0003f22a0}, 0xc0003c0700)

011/home/runner/go/pkg/mod/github.com/rs/cors@v1.8.2/cors.go:231 +0x1c4

net/http.HandlerFunc.ServeHTTP(0x0?, {0xaf3928?, 0xc0003f22a0?}, 0xc0000d2800?)

011/opt/hostedtoolcache/go/1.18.7/x64/src/net/http/server.go:2084 +0x2f

net/http.serverHandler.ServeHTTP({0xc0004827e0?}, {0xaf3928, 0xc0003f22a0}, 0xc0003c0700)

011/opt/hostedtoolcache/go/1.18.7/x64/src/net/http/server.go:2916 +0x43b

net/http.(*conn).serve(0xc00038a280, {0xaf3dc0, 0xc0002c7a70})

011/opt/hostedtoolcache/go/1.18.7/x64/src/net/http/server.go:1966 +0x5d7

created by net/http.(*Server).Serve

011/opt/hostedtoolcache/go/1.18.7/x64/src/net/http/server.go:3071 +0x4db

ingvarrwvw commented 1 year ago

And what to do with it? how to fix? and how to understand what he can not find?

chrboe commented 1 year ago

Can you outline what exactly you mean by "removing / adding new satellites"?

How many nodes did you have before, what LINSTOR commands did you run, and how many did you have afterwards?

Samamy commented 1 year ago

You most likely deleted a linstor node but did not delete it iscsi target associated for deleted node

chrboe commented 1 year ago

Better late than never: this should be fixed in df53d0b3fd099202b3fd65c124325097319c2579 (Will be in v1.1.1).

Feel free to reopen if this issue remains.