"runtime error: invalid memory address or nil pointer dereference" on re-associating orphaned agent stacks

Bug description I'm unable to re-associate orphaned stacks that are running on an agent. Portainer is stuck on '"Association in progress". No error message is displayed in the GUI. I've already restarted Portainer as well as the entire machine. I also redeployed the agent to no avail.

Expected behavior Portainer should re-associate the orphaned stacks upon clicking the re-associating button.

Portainer Logs Portainer Side:

2023/01/07 14:33:45 server: proxy#1:R:0.0.0.0:61746=>192.168.64.2:9001: Listening
{"time":1673098444,"message":"http: panic serving 192.168.30.222:41724: runtime error: invalid memory address or nil pointer dereference\ngoroutine 216072 [running]:\nnet/http.(*conn).serve.func1()\n\tnet/http/server.go:1850 +0xbf\npanic({0x18fea60, 0x2b58d00})\n\truntime/panic.go:890 +0x262\ngithub.com/portainer/portainer/api/http/handler/stacks.(*Handler).userCanManageStacks(0xc000324150?, 0x1e15500?, 0xc0001b0110?)\n\tgithub.com/portainer/portainer/api/http/handler/stacks/handler.go:140 +0x18\ngithub.com/portainer/portainer/api/http/handler/stacks.(*Handler).stackInspect(0xc000134370, {0x1e2d470, 0xc000808d20}, 0x1e2ebc0?)\n\tgithub.com/portainer/portainer/api/http/handler/stacks/stack_inspect.go:58 +0x34e\ngithub.com/portainer/libhttp/error.LoggerHandler.ServeHTTP(0xc0005a0000?, {0x1e2d470, 0xc000808d20}, 0xc000a60270?)\n\tgithub.com/portainer/libhttp@v0.0.0-20220916153711-5d61e12f4b0a/error/error.go:23 +0x2f\ngithub.com/portainer/portainer/api/http/security.(*RequestBouncer).mwUpgradeToRestrictedRequest.func1({0x1e2d470, 0xc000808d20}, 0xc0000e7f00)\n\tgithub.com/portainer/portainer/api/http/security/bouncer.go:231 +0x168\nnet/http.HandlerFunc.ServeHTTP(0xc0003247d0?, {0x1e2d470?, 0xc000808d20?}, 0x0?)\n\tnet/http/server.go:2109 +0x2f\ngithub.com/portainer/portainer/api/http/security.(*RequestBouncer).mwAuthenticateFirst.func1({0x1e2d470, 0xc000808d20}, 0xc0000e7e00)\n\tgithub.com/portainer/portainer/api/http/security/bouncer.go:282 +0x248\nnet/http.HandlerFunc.ServeHTTP(0x1827a20?, {0x1e2d470?, 0xc000808d20?}, 0x16?)\n\tnet/http/server.go:2109 +0x2f\ngithub.com/portainer/portainer/api/http/security.mwSecureHeaders.func1({0x1e2d470, 0xc000808d20}, 0xc0004ef2c0?)\n\tgithub.com/portainer/portainer/api/http/security/bouncer.go:380 +0x204\nnet/http.HandlerFunc.ServeHTTP(0xc0000e7d00?, {0x1e2d470?, 0xc000808d20?}, 0x1b11d40?)\n\tnet/http/server.go:2109 +0x2f\ngithub.com/gorilla/mux.(*Router).ServeHTTP(0xc00058b5c0, {0x1e2d470, 0xc000808d20}, 0xc0000e7c00)\n\tgithub.com/gorilla/mux@v1.7.3/mux.go:212 +0x202\nnet/http.StripPrefix.func1({0x1e2d470, 0xc000808d20}, 0xc0000e7a00)\n\tnet/http/server.go:2152 +0x332\nnet/http.HandlerFunc.ServeHTTP(0x1b6aef0?, {0x1e2d470?, 0xc000808d20?}, 0xc000134370?)\n\tnet/http/server.go:2109 +0x2f\ngithub.com/portainer/portainer/api/http/handler.(*Handler).ServeHTTP(0xc0005d9180, {0x1e2d470, 0xc000808d20}, 0xc0000e7a00)\n\tgithub.com/portainer/portainer/api/http/handler/handler.go:223 +0xe1d\ngithub.com/portainer/portainer/api/http/offlinegate.(*OfflineGate).WaitingMiddleware.func1({0x1e2d470, 0xc000808d20}, 0xc0000e7a00)\n\tgithub.com/portainer/portainer/api/http/offlinegate/offlinegate.go:39 +0x177\nnet/http.HandlerFunc.ServeHTTP(0x0?, {0x1e2d470?, 0xc000808d20?}, 0x0?)\n\tnet/http/server.go:2109 +0x2f\ngithub.com/portainer/portainer/api/adminmonitor.(*Monitor).WithRedirect.func1({0x1e2d470, 0xc000808d20}, 0xc0000e7a00)\n\tgithub.com/portainer/portainer/api/adminmonitor/admin_monitor.go:116 +0x20c\nnet/http.HandlerFunc.ServeHTTP(0x0?, {0x1e2d470?, 0xc000808d20?}, 0x463fee?)\n\tnet/http/server.go:2109 +0x2f\nnet/http.serverHandler.ServeHTTP({0xc0004ef140?}, {0x1e2d470, 0xc000808d20}, 0xc0000e7a00)\n\tnet/http/server.go:2947 +0x30c\nnet/http.(*conn).serve(0xc0005cdcc0, {0x1e2ebc0, 0xc000040690})\n\tnet/http/server.go:1991 +0x607\ncreated by net/http.(*Server).Serve\n\tnet/http/server.go:3102 +0x4db"}

Agent Side:

2023/01/07 01:33PM DBG github.com/portainer/agent/chisel/client.go:40 > creating reverse tunnel client | local_addr=192.168.64.2:9001 remote_port=61746 server=XXXXXXXXXXXXXXXXXXXXXX:8000 server_fingerprint=XXXXXXXXXXXXXXXXXXXXXXXX
2023/01/07 13:33:58 client: Connecting to ws://XXXXXXXXXXXXXXXXXXXXXXXXXXXX:8000
2023/01/07 01:33PM DBG github.com/portainer/agent/edge/edge.go:261 > Docker runtime configuration check | engine_status=1 leader_node=false
2023/01/07 13:33:58 client: Specified deprecated MD5 fingerprint (XXXXXXXXXXXXXXXXXXXXX), please update to the new SHA256 fingerprint: XXXXXXXXXXXXXXXXXXXXXXXX
2023/01/07 13:33:58 client: Connected (Latency 16.272146ms)

Steps to reproduce the issue:

Delete environment
Create new environment
Re-deploy agent
Try to re-associate orphaned stacks via Portainer GUI

Technical details:

Portainer version: 2.16.2
Docker version (managed by Portainer): Standalone 20.10.17 (for both sides)
Platform (windows/linux): Linux

Command used to start Portainer (docker run -p 9443:9443 portainer/portainer):

services:
portainer:
image: portainer/portainer-ce:latest
container_name: portainer
restart: always
volumes:
  - /etc/localtime:/etc/localtime:ro
  - /var/run/docker.sock:/var/run/docker.sock:ro
  - ./data:/data
network_mode: "bridge"
ports:
  - 9000:9000 #HTTP WebUI - Synology acts as Reverse Proxy
  - 8000:8000 #Edge Agent Endpoint
  - 9443:9443 #HTTPS WebUI - For secure Edge Agent Comms

Browser: FireFox 108.0.1
Use Case (delete as appropriate): Using Portainer at Home
Have you reviewed our technical documentation and knowledge base? Yes

Additional context This all started after my WAN IP changed. Previously when I deployed the agents, I specified my WAN IP instead of a FQDN. I now tried to resolve this issue by redeploying the agents with a FQDN instead and went down this rabbit hole...

portainer / portainer

"runtime error: invalid memory address or nil pointer dereference" on re-associating orphaned agent stacks #8267