bug: problems with bearer authentication

[rchincha]

Discussed in https://github.com/project-zot/zot/discussions/2089

^{Originally posted by **nyabla** November 29, 2023} versions I tried on: 1.4.3 and 2.0.0-rc7 my config file: [config.json](https://github.com/project-zot/zot/files/13504377/config.json) background: i am trying to run an instance of zot with authentication handled by [authentik](https://goauthentik.io). i have a sort of proxy in front of the token endpoint of authentik in order to translate the `GET` token request into a `POST` request. The clients I tried (docker cli and podman) are both able to obtain a valid token from this endpoint. problem: zot gives a `500 Internal Server Error` response upon receiving a request with a (valid) bearer token ONLY when the `service` key under `bearer` is non-empty. if the `service` key has an empty string (`""`) then podman/skopeo get confused. what i tried: making sure that the value of the `aud` key in the jwt matches the value of `service`, similarly to the auth.docker.io endpoint.

[nyabla]

logs with podman client and service set to the same as aud in the jwt. zot 1.4.3

{"level":"info","params":{"distSpecVersion":"1.0.1-dev","GoVersion":"go1.19.3","Commit":"v1.4.3-69f0cf6bb4727884af42f38c92858fdb114104de","ReleaseTag":"v1.4.3","BinaryType":"-sync-search-scrub-metrics-lint","AccessControl":null,"Storage":{"RootDirectory":"/tmp/zot","Dedupe":true,"RemoteCache":false,"GC":true,"Commit":false,"GCDelay":3600000000000,"GCInterval":0,"StorageDriver":null,"CacheDriver":null,"SubPaths":null},"HTTP":{"Address":"0.0.0.0","Port":"5000","AllowOrigin":"","TLS":null,"Auth":{"FailDelay":0,"HTPasswd":{"Path":""},"LDAP":null,"Bearer":{"Realm":"http://localhost:8080/token","Service":"zot.local","Cert":"/etc/zot/auth.crt"}},"RawAccessControl":null,"Realm":"","Ratelimit":null},"Log":{"Level":"debug","Output":"","Audit":""},"Extensions":null},"goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:119","time":"2023-11-30T12:10:38.584005867Z","message":"configuration settings"}
{"level":"info","cpus":4,"max. open files":1048576,"listen backlog":"4096","max. inotify watches":"65536","goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:110","time":"2023-11-30T12:10:38.5846984Z","message":"runtime params"}
{"level":"warn","goroutine":1,"caller":"zotregistry.io/zot/pkg/debug/swagger/swagger_disabled.go:22","time":"2023-11-30T12:10:38.590058387Z","message":"skipping enabling swagger because given zot binary doesn't include this feature, please build a binary that does so"}
{"level":"info","module":"http","clientIP":"172.22.0.1:49190","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":283,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.23.1 (github.com/containers/image)"]},"goroutine":51,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:12:20.421826473Z","message":"HTTP API"}
{"level":"error","goroutine":52,"caller":"zotregistry.io/zot/pkg/log/log.go:25","time":"2023-11-30T12:12:21.403685258Z","message":"panic recovered"}
{"level":"info","module":"http","clientIP":"172.22.0.1:43514","method":"GET","path":"/v2/","statusCode":500,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Connection":["close"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.23.1 (github.com/containers/image)"]},"goroutine":52,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:12:21.403825678Z","message":"HTTP API"}

same but with docker cli client

{"level":"info","params":{"distSpecVersion":"1.0.1-dev","GoVersion":"go1.19.3","Commit":"v1.4.3-69f0cf6bb4727884af42f38c92858fdb114104de","ReleaseTag":"v1.4.3","BinaryType":"-sync-search-scrub-metrics-lint","AccessControl":null,"Storage":{"RootDirectory":"/tmp/zot","Dedupe":true,"RemoteCache":false,"GC":true,"Commit":false,"GCDelay":3600000000000,"GCInterval":0,"StorageDriver":null,"CacheDriver":null,"SubPaths":null},"HTTP":{"Address":"0.0.0.0","Port":"5000","AllowOrigin":"","TLS":null,"Auth":{"FailDelay":0,"HTPasswd":{"Path":""},"LDAP":null,"Bearer":{"Realm":"http://localhost:8080/token","Service":"zot.local","Cert":"/etc/zot/auth.crt"}},"RawAccessControl":null,"Realm":"","Ratelimit":null},"Log":{"Level":"debug","Output":"","Audit":""},"Extensions":null},"goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:119","time":"2023-11-30T12:16:46.390232659Z","message":"configuration settings"}
{"level":"info","cpus":4,"max. open files":1048576,"listen backlog":"4096","max. inotify watches":"65536","goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:110","time":"2023-11-30T12:16:46.390424799Z","message":"runtime params"}
{"level":"warn","goroutine":1,"caller":"zotregistry.io/zot/pkg/debug/swagger/swagger_disabled.go:22","time":"2023-11-30T12:16:46.39296521Z","message":"skipping enabling swagger because given zot binary doesn't include this feature, please build a binary that does so"}
{"level":"info","module":"http","clientIP":"172.20.0.1:50076","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":283,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.1.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"]},"goroutine":23,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:17:31.927995452Z","message":"HTTP API"}
{"level":"error","goroutine":25,"caller":"zotregistry.io/zot/pkg/log/log.go:25","time":"2023-11-30T12:17:32.711874692Z","message":"panic recovered"}
{"level":"info","module":"http","clientIP":"172.20.0.1:50086","method":"GET","path":"/v2/","statusCode":500,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Connection":["close"],"User-Agent":["docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.1.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"]},"goroutine":25,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:17:32.712075996Z","message":"HTTP API"}

docker client with empty service. i havent included a log from podman because it can't handle empty service.

{"level":"info","params":{"distSpecVersion":"1.0.1-dev","GoVersion":"go1.19.3","Commit":"v1.4.3-69f0cf6bb4727884af42f38c92858fdb114104de","ReleaseTag":"v1.4.3","BinaryType":"-sync-search-scrub-metrics-lint","AccessControl":null,"Storage":{"RootDirectory":"/tmp/zot","Dedupe":true,"RemoteCache":false,"GC":true,"Commit":false,"GCDelay":3600000000000,"GCInterval":0,"StorageDriver":null,"CacheDriver":null,"SubPaths":null},"HTTP":{"Address":"0.0.0.0","Port":"5000","AllowOrigin":"","TLS":null,"Auth":{"FailDelay":0,"HTPasswd":{"Path":""},"LDAP":null,"Bearer":{"Realm":"http://localhost:8080/token","Service":"","Cert":"/etc/zot/auth.crt"}},"RawAccessControl":null,"Realm":"","Ratelimit":null},"Log":{"Level":"debug","Output":"","Audit":""},"Extensions":null},"goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:119","time":"2023-11-30T12:20:00.097669041Z","message":"configuration settings"}
{"level":"info","cpus":4,"max. open files":1048576,"listen backlog":"4096","max. inotify watches":"65536","goroutine":1,"caller":"zotregistry.io/zot/pkg/api/controller.go:110","time":"2023-11-30T12:20:00.097871564Z","message":"runtime params"}
{"level":"warn","goroutine":1,"caller":"zotregistry.io/zot/pkg/debug/swagger/swagger_disabled.go:22","time":"2023-11-30T12:20:00.101151848Z","message":"skipping enabling swagger because given zot binary doesn't include this feature, please build a binary that does so"}
{"level":"info","module":"http","clientIP":"172.22.0.1:52258","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.1.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"]},"goroutine":19,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:20:16.185581632Z","message":"HTTP API"}
{"level":"info","module":"http","clientIP":"172.22.0.1:52262","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"User-Agent":["docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.1.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"]},"goroutine":11,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:20:16.186862918Z","message":"HTTP API"}

[nyabla]

access claim was missing in jwt. feel free to close

[rchincha]

@peusebiu can you pls check if we could handle/fail this more gracefully.

{"level":"info","module":"http","clientIP":"172.22.0.1:49190","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":283,"headers":{"Accept-Encoding":["gzip"],"Connection":["close"],"Docker-Distribution-Api-Version":["registry/2.0"],"User-Agent":["containers/5.23.1 (github.com/containers/image)"]},"goroutine":51,"caller":"zotregistry.io/zot/pkg/api/session.go:131","time":"2023-11-30T12:12:20.421826473Z","message":"HTTP API"} {"level":"error","goroutine":52,"caller":"zotregistry.io/zot/pkg/log/log.go:25","time":"2023-11-30T12:12:21.403685258Z","message":"panic recovered"}

[rchincha]

@nyabla can you give us the exact steps both on authentik and zot so that we can reproduce this panic? Understood that this is an operator error, but still in our opinion best if we handle this.

[eusebiu-constantin-petu-dbk]

@nyabla I tried to reproduce but I can't configure authentik, mainly I can not do the initial setup, it's prompting me to login with user and pass, instead of a prompt with setting up the admin account.

Tried to add a username and password in .env config but same issue.

[rchincha]

Closing this, pls re-open if needed.