search

SUSE / Portus

Authorization service and frontend for Docker registry (v2)
http://port.us.org/
Apache License 2.0
3k stars 470 forks source link

OpenID Connect OpenSSL::X509::StoreError #2221

Closed d3473r closed 4 years ago

d3473r commented 5 years ago

Description

OpenSSL::X509::StoreError is thrown when trying to use openid-connect with portus

Steps to reproduce

  1. Set up portus in rancher (kubernetes) with ingress-nginx (ssl termination in front)
  2. Try to use portus with keycloak via openid-connect
  3. OpenSSL::X509::StoreError is thrown in portus

Logs

click

``` Started GET "/users/auth/openid_connect" for xxx.xxx.xxx.xxx at 2019-08-09 08:53:27 +0000 I, [2019-08-09T08:53:27.617190 #37] INFO -- omniauth: (openid_connect) Setup endpoint detected, running now. I, [2019-08-09T08:53:27.617406 #37] INFO -- omniauth: (openid_connect) Request phase initiated. OpenSSL::X509::StoreError (system lib): vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:59:in `add_file' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:59:in `call' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:59:in `block (2 levels) in ' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:245:in `add_trust_ca_to_store' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:419:in `load_cacerts' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:252:in `load_trust_ca' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:285:in `set_context' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_socket.rb:136:in `create_openssl_socket' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_socket.rb:39:in `initialize' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_socket.rb:26:in `new' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_socket.rb:26:in `create_socket' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:752:in `block in connect' /usr/lib64/ruby/2.6.0/timeout.rb:93:in `block in timeout' /usr/lib64/ruby/2.6.0/timeout.rb:103:in `timeout' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:748:in `connect' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:511:in `query' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:177:in `query' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient.rb:1242:in `do_get_block' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient.rb:1019:in `block in do_request' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient.rb:1133:in `protect_keep_alive_disconnected' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient.rb:1014:in `do_request' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient.rb:1104:in `follow_redirect' vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient.rb:655:in `get_content' vendor/bundle/ruby/2.6.0/gems/swd-1.1.2/lib/swd/resource.rb:20:in `block (2 levels) in discover!' vendor/bundle/ruby/2.6.0/gems/swd-1.1.2/lib/swd/resource.rb:37:in `handle_response' vendor/bundle/ruby/2.6.0/gems/swd-1.1.2/lib/swd/resource.rb:19:in `block in discover!' vendor/bundle/ruby/2.6.0/gems/swd-1.1.2/lib/swd/cache.rb:4:in `fetch' vendor/bundle/ruby/2.6.0/gems/swd-1.1.2/lib/swd/resource.rb:18:in `discover!' vendor/bundle/ruby/2.6.0/gems/openid_connect-1.1.6/lib/openid_connect/discovery/provider/config.rb:7:in `discover!' vendor/bundle/ruby/2.6.0/gems/omniauth_openid_connect-0.2.4/lib/omniauth/strategies/openid_connect.rb:81:in `config' vendor/bundle/ruby/2.6.0/gems/omniauth_openid_connect-0.2.4/lib/omniauth/strategies/openid_connect.rb:146:in `discover!' vendor/bundle/ruby/2.6.0/gems/omniauth_openid_connect-0.2.4/lib/omniauth/strategies/openid_connect.rb:86:in `request_phase' vendor/bundle/ruby/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:226:in `request_call' vendor/bundle/ruby/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:188:in `call!' vendor/bundle/ruby/2.6.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/deflater.rb:34:in `call' vendor/bundle/ruby/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call' vendor/bundle/ruby/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch' vendor/bundle/ruby/2.6.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call' vendor/bundle/ruby/2.6.0/gems/rack-cors-1.0.3/lib/rack/cors.rb:95:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/tempfile_reaper.rb:15:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/etag.rb:25:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/conditional_get.rb:25:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/head.rb:12:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/http/content_security_policy.rb:18:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:232:in `context' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/session/abstract/id.rb:226:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/cookies.rb:670:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call' vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:98:in `run_callbacks' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call' vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:38:in `call_app' vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `block in call' vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `block in tagged' vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:28:in `tagged' vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/tagged_logging.rb:71:in `tagged' vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/rack/logger.rb:26:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/request_id.rb:27:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call' vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in `call' vendor/bundle/ruby/2.6.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/static.rb:127:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call' vendor/bundle/ruby/2.6.0/gems/railties-5.2.3/lib/rails/engine.rb:524:in `call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:68:in `block in call' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `each' vendor/bundle/ruby/2.6.0/gems/rack-2.0.7/lib/rack/urlmap.rb:53:in `call' vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/configuration.rb:227:in `call' vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/server.rb:660:in `handle_request' vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/server.rb:474:in `process_client' vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/server.rb:334:in `block in run' vendor/bundle/ruby/2.6.0/gems/puma-3.12.1/lib/puma/thread_pool.rb:135:in `block in spawn_thread' Processing by ErrorsController#show as HTML Parameters: {"status"=>"500"} Rendering errors/500.html.erb within layouts/errors Rendered errors/500.html.erb within layouts/errors (0.1ms) Rendered errors/_status_title.html.slim (0.0ms) Completed 500 Internal Server Error in 2ms (Views: 1.2ms | ActiveRecord: 0.0ms) ```

Deployment information

Deployment method:

Kubernetes with Rancher

yml

```yml apiVersion: apps/v1beta2 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "33" field.cattle.io/creatorId: user-v4zmk field.cattle.io/publicEndpoints: '[{"addresses":["xxx.xxx.xxx.xxx"],"port":443,"protocol":"HTTPS","serviceName":"docker-registry:ingress-b459d6b6de2edb11ff3f93111a2aa6bf","ingressName":"docker-registry:registry-ingress","hostname":"registry.example.com","allNodes":true}]' creationTimestamp: 2019-07-19T13:11:21Z generation: 40 labels: cattle.io/creator: norman workload.user.cattle.io/workloadselector: deployment-docker-registry-portus name: portus namespace: docker-registry resourceVersion: "38322291" selfLink: /apis/apps/v1beta2/namespaces/docker-registry/deployments/portus uid: b3b072db-aa26-11e9-b85c-96000014acbd spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: workload.user.cattle.io/workloadselector: deployment-docker-registry-portus strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate template: metadata: annotations: cattle.io/timestamp: 2019-08-09T08:16:13Z creationTimestamp: null labels: workload.user.cattle.io/workloadselector: deployment-docker-registry-portus spec: containers: - env: - name: PORTUS_DB_ADAPTER value: postgresql - name: PORTUS_DB_DATABASE value: portus - name: PORTUS_DB_HOST value: postgres - name: PORTUS_DB_PASSWORD value: xxxxxxxxxxxxxxxxxxxxxx - name: PORTUS_DB_USERNAME value: portus - name: PORTUS_KEY_PATH value: /secrets/portus.key - name: PORTUS_MACHINE_FQDN_VALUE value: registry.example.com - name: PORTUS_OAUTH_OPENID_CONNECT_ENABLED value: "true" - name: PORTUS_OAUTH_OPENID_CONNECT_IDENTIFIER value: portus - name: PORTUS_OAUTH_OPENID_CONNECT_ISSUER value: https://keycloak.example.com/auth/realms/my-realm - name: PORTUS_OAUTH_OPENID_CONNECT_SECRET value: xxxxxxxxxxxxxxxxxxxxxx - name: PORTUS_PASSWORD value: xxxxxxxxxxxxxxxxxxxxxx - name: PORTUS_SECRET_KEY_BASE value: xxxxxxxxxxxxxxxxxxxxxx - name: RAILS_SERVE_STATIC_FILES value: "true" image: opensuse/portus:head imagePullPolicy: Always name: portus resources: {} securityContext: allowPrivilegeEscalation: false capabilities: {} privileged: false readOnlyRootFilesystem: false runAsNonRoot: false stdin: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File tty: true volumeMounts: - mountPath: /secrets name: vol2 readOnly: true dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - hostPath: path: /root/config/docker-registry/portus/secrets type: "" name: vol2 status: availableReplicas: 1 conditions: - lastTransitionTime: 2019-08-03T09:21:30Z lastUpdateTime: 2019-08-03T09:21:30Z message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available - lastTransitionTime: 2019-07-19T13:11:21Z lastUpdateTime: 2019-08-09T08:16:17Z message: ReplicaSet "portus-565468c7fd" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing observedGeneration: 40 readyReplicas: 1 replicas: 1 updatedReplicas: 1 ```

Configuration:

portusctl exec rake portus:info

yml

```yml [schema] Selected the schema for postgresql [Mailer config] Host: portus.test.lan [Mailer config] Protocol: https:// Evaluated configuration: --- email: from: portus@example.com name: Portus reply_to: '' smtp: enabled: false address: smtp.example.com port: 587 domain: example.com ssl_tls: '' enable_starttls_auto: false openssl_verify_mode: none ca_path: '' ca_file: '' user_name: '' password: "****" authentication: login gravatar: enabled: true delete: enabled: true contributors: false garbage_collector: enabled: false older_than: 30 keep_latest: 5 tag: '' ldap: enabled: false hostname: ldap_hostname port: 389 timeout: 5 encryption: method: '' options: ca_file: '' ssl_version: TLSv1_2 base: '' admin_base: '' group_base: '' filter: '' uid: uid authentication: enabled: false bind_dn: '' password: "****" group_sync: enabled: true default_role: viewer guess_email: enabled: false attr: '' oauth: local_login: enabled: true google_oauth2: enabled: false id: '' secret: '' domain: '' options: hd: '' open_id: enabled: false identifier: '' domain: '' openid_connect: enabled: true issuer: https://keycloak.example.com/auth/realms/my-realm identifier: portus secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx github: enabled: false client_id: '' client_secret: '' organization: '' team: '' domain: '' gitlab: enabled: false application_id: '' secret: '' group: '' domain: '' server: '' bitbucket: enabled: false key: '' secret: '' domain: '' options: team: '' first_user_admin: enabled: true signup: enabled: true check_ssl_usage: enabled: true registry: jwt_expiration_time: value: 15 catalog_page: value: 100 timeout: value: 2 read_timeout: value: 120 machine_fqdn: value: portus.example.com display_name: enabled: false user_permission: change_visibility: enabled: true create_team: enabled: true manage_team: enabled: true create_namespace: enabled: true manage_namespace: enabled: true create_webhook: enabled: true manage_webhook: enabled: true push_images: policy: allow-teams security: clair: server: '' health_port: 6061 timeout: 900 zypper: server: '' dummy: server: '' anonymous_browsing: enabled: true background: registry: enabled: true sync: enabled: true strategy: initial pagination: per_page: 10 before_after: 2 Portus version: 2.5.0-dev@a1b9f2ebfeb84680a9dcd5629195e4c52815735c ```

Portus version:

2.5.0-dev@a1b9f2ebfeb84680a9dcd5629195e4c52815735c

stale[bot] commented 4 years ago

Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

Josua-SR commented 4 years ago

Has meanwhile anyone figured out what this error means?

Josua-SR commented 4 years ago

So I have figured out how this error happens. After an extreme amount of trying various configuration options and environment variables, and after reproducing the problem on master branch with the omniauth_openid_connect gem updated even, I wanted to know what exactly goes wrong that is hiding behind the benign OpenSSL::X509::StoreError (system lib): message.

So I changed the entrypoint of the container:

RUN zypper install -y strace && zypper clean -a
ENTRYPOINT ["/usr/bin/strace", "-f", "/bin/sh", "/init"]

After a long startup time and after trying to login once, I had a 120k lines logfile to read. Well, We are interested in the part closest to the StoreError we are seeing, so here is an extract of the relevant section:

[pid    61] openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 14
[pid    61] fstat(14, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
[pid    61] read(14, "\221\304\364\215\32\313}\347\362\3157\207:\204\3657n\351\31\330\330\211\7\364\270[\200\224U~\203\234"..., 48) = 48
[pid    61] close(14)                   = 0
[pid    61] stat("/srv/Portus/vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/cacert.pem", 0x7f8ce6dd3f30) = -1 ENOENT (No such file or directory)
[pid    61] openat(AT_FDCWD, "/srv/Portus/vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/cacert.pem", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid    61] close(13)                   = 0
[pid    61] futex(0x7f8cd0048ab0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid    80] <... futex resumed> )       = 0
[pid    80] futex(0x7f8cd0048ae0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid    61] <... futex resumed> )       = 1
[pid    61] futex(0x7f8cd0048ae0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid    80] <... futex resumed> )       = 0
[pid    61] <... futex resumed> )       = 1
[pid    80] futex(0x7f8cd0048ae0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid    61] futex(0x55eb446e6674, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid    80] futex(0x55eb446e6674, FUTEX_WAKE_PRIVATE, 1) = 0
[pid    80] futex(0x7f8ce68d06d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=2, tv_nsec=999998587} <unfinished ...>
[pid    61] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid    61] futex(0x55eb446e66a0, FUTEX_WAKE_PRIVATE, 1) = 0

OpenSSL::X509::StoreError (system lib):

vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:59:in `add_file'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:59:in `call'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:59:in `block (2 levels) in <class:Store>'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:245:in `add_trust_ca_to_store'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:419:in `load_cacerts'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:252:in `load_trust_ca'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_config.rb:285:in `set_context'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_socket.rb:136:in `create_openssl_socket'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_socket.rb:39:in `initialize'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_socket.rb:26:in `new'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/ssl_socket.rb:26:in `create_socket'
vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:752:in `block in connect'

So, it turns out that for some unknown reason, the default certificate catalog that is expected to ship with httpclient, /srv/Portus/vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/cacert.pem, does not exist. Why that is, is way beyond my expertise as both the ruby language as well as its tools are an unexplored world to me.

For the time being, here is a workaround to the issue that can be applied to either the v2.5 docker image, or anything built from master:

RUN ln -sv /var/lib/ca-certificates/ca-bundle.pem /srv/Portus/vendor/bundle/ruby/2.6.0/gems/httpclient-2.8.3/lib/httpclient/cacert.pem