search

pomerium / ingress-controller

Pomerium Kubernetes Ingress Controller
https://pomerium.com
Apache License 2.0
22 stars 11 forks source link

Error: overlaps with local certs: skipped #699

Closed andreyolv closed 11 months ago

andreyolv commented 1 year ago

What happened?

Deploy in kubernetes does not work

What did you expect to happen?

May it work

How'd it happen?

Deploy pomerium on kubernetes as described in https://www.pomerium.com/docs/deploy

What's your environment like?

What's your config.yaml?

apiVersion: ingress.pomerium.io/v1
kind: Pomerium
metadata:
  name: global
  namespace: pomerium
spec:
  secrets: pomerium/bootstrap
  authenticate:
    url: https://xxxxxxxxxxxx
  identityProvider:
    provider: azure
    secret: pomerium/my-azure-secret
    url: https://xxxxxxxxxxxxx
  certificates:
  - pomerium/my-domain-tls-certificate
  jwtClaimHeaders:
    X-given-name: given_name
    X-name: name
  storage:
    postgres:
      secret: pomerium/postgres-pomerium
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: linkerd-test
  namespace: linkerd-viz
  annotations:
    ingress.pomerium.io/allow_websockets: 'true'
    ingress.pomerium.io/pass_identity_headers: 'true'
    ingress.pomerium.io/policy: |
      allow:
        and:
        - claim/groups: xxxxxxxxxxxxxxxxxxxxxx
spec:
  ingressClassName: pomerium
  tls:
  - hosts:
      - '*.mydomain'
    secretName: my-domain-tls-certificate
  rules:
  - host: linkerd.mydomain
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service: 
            name: web
            port:
              number: 8084

What did you see in the logs?

...
{"level":"warn","time":"2023-07-24T17:31:23Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","domain":"*.mydomain","time":"2023-07-24T17:31:29Z","message":"overlaps with local certs: skipped"}
...
... after some time ....
...
{"level":"warn","time":"2023-07-24T17:51:19Z","msg":"stapling OCSP","service":"autocert","error":"unable to write OCSP staple file for [*.mydomain mydomain]: mkdir /.local: read-only file system"}
...

Additional context

Command 'kubectl describe pomerium global':

...
Status:
  Ingress:
    linkerd-viz/linkerd-test:
      Observed At:          2023-07-24T17:37:53Z
      Observed Generation:  1
      Reconciled:           true
...
Events:
...
  Normal   Updated      45m (x16 over 52m)  pomerium-ingress                        linkerd-viz/linkerd-test: config updated

Command 'kubectl describe ingress linkerd-test -n linkerd-viz':

...
Events:
...
  Normal   Updated      49m (x16 over 55m)  pomerium-ingress  config updated

When I access https:// linkerd.mydomain I receive 404 Not Found nginx.

Thanks

wasaga commented 1 year ago

Hello,

couple things here:

  1. I understand you're using wildcard certificates. You only need to reference them in one place - i.e. if you reference them in global CRD certificates section, you do not need to reference them in the Ingress.spec.tls section. This is what caused the cert overlap complaint (it should not really influence your operation, as you're just supplying duplicate certificates, and it would only use one referenced in the global.spec.certificates.
  2. I am confused how NGINX is in the picture? is web service == NGINX?
  3. Please check access log by doing kubectl logs -n pomerium deployment/pomerium - I suppose that your request arrived to the destination and it's the destination service that doesn't like it (probably a Host authority?) - please provide the access log entry for your request. All Pomerium access log entries has x-request-id field that is also returned in the response headers, for correlation.
andreyolv commented 11 months ago

Now logs output this too: {"level":"info","time":"2023-10-25T19:09:27Z","message":"using /etc/ssl/certs/ca-certificates.crt as the system root certificate authority bundle"}

@wasaga

  1. Ok, but with Ingress.spec.tls or not, same problem.
  2. Sorry, I did not understand your question.
  3. {"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-25T21:08:16Z","message":"get"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"info","record-type":"type.googleapis.com/pomerium.config.Config","record-id":"ingress-controller","time":"2023-10-25T21:08:16Z","message":"put"}
{"level":"info","ts":"2023-10-25T21:08:16Z","msg":"new pomerium config applied","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","Ingress":{"name":"marquez-test","namespace":"marquez"},"namespace":"marquez","name":"marquez-test","reconcileID":"xxxxxxxxxxxxx"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config_id":"ingress-controller","version":7,"err_count":0,"time":"2023-10-25T21:08:16Z","message":"set db config info"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config_id":"pomerium-crd","version":1,"err_count":0,"time":"2023-10-25T21:08:16Z","message":"set db config info"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"pomerium-crd","time":"2023-10-25T21:08:16Z","message":"get"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-25T21:08:16Z","message":"get"}
{"level":"info","Algorithm":"ES256","KeyID":"xxxxxxxxxxxxxxxx","Public Key":{"use":"sig","kty":"EC","kid":"xxxxxxxxxxxxxxxxxxx","crv":"P-256","alg":"ES256","x":"xxxxxxxxxxxxxxxxx","y":"xxxxxxxxxxxxxx"},"time":"2023-10-25T21:08:16Z","message":"authorize: signing key"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-25T21:08:16Z","message":"cds: add 1 cluster(s), remove 0 cluster(s)"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-25T21:08:16Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-25T21:08:16Z","message":"cds: added/updated 1 cluster(s), skipped 0 unmodified cluster(s)"}
{"level":"info","time":"2023-10-25T21:08:16Z","message":"service registry reporter stopping"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config":"databroker","checksum":"bb047254eba6628d","time":"2023-10-25T21:08:16Z","message":"config: updated config"}
{"level":"warn","time":"2023-10-25T21:13:11Z","msg":"stapling OCSP","service":"autocert","error":"no OCSP stapling for [*.xxxxxxxxxxx xxxxxxxxxxx]: parsing OCSP response: ocsp: error from server: unauthorized"}
andreyolv commented 11 months ago

I think I discovered the problem.

The secret in spec.certificates in kind Pomerium must be in PEM format according to documentation and I'm passing tls.crt and tls.key in secret.

I'll try to change it here and let you know if it works.

wasaga commented 11 months ago

My question 2 was why do you see responses from nginx?

In your logs I do not see pomerium access log entries for your domain.

Make sure you're actually talking to pomerium and there's no dns mixup.

If pomerium responds there's x-request-id header set in the response and you can find relevant access and authorization log entries https://www.pomerium.com/docs/capabilities/audit-logs

andreyolv commented 11 months ago

I tested pomerium ingress controller locally following https://www.pomerium.com/docs/deploy/k8s/quickstart step by step, and same problem. Seems my certificate is completely ignored: "message":"neither autocert, insecure_server or manually provided certificates were provided, server will be using a self-signed certificate"}

Pod logs:

I1027 12:57:39.823650       1 request.go:690] Waited for 1.036440278s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/source.toolkit.fluxcd.io/v1beta2?timeout=32s
{"level":"info","ts":"2023-10-27T12:57:39Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"localhost:41441"}
{"level":"info","ts":"2023-10-27T12:57:39Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"127.0.0.1:41441"}
{"level":"info","ts":"2023-10-27T12:57:39Z","msg":"Starting EventSource","controller":"bootstrap pod/pomerium-b6f9dc578-8659r","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","source":"kind source: *v1.Pomerium"}
{"level":"info","ts":"2023-10-27T12:57:39Z","msg":"Starting EventSource","controller":"bootstrap pod/pomerium-b6f9dc578-8659r","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2023-10-27T12:57:39Z","msg":"Starting Controller","controller":"bootstrap pod/pomerium-b6f9dc578-8659r","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium"}
{"level":"info","ts":"2023-10-27T12:57:40Z","msg":"Starting workers","controller":"bootstrap pod/pomerium-b6f9dc578-8659r","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","worker count":1}
{"level":"warn","time":"2023-10-27T12:59:08Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"info","envoy_version":"1.25.5+b1095c058415dfb2261e695a0f144311a7dc346b6eb47ecbb0a01b7de2c7299f","version":"v0.0.0","time":"2023-10-27T12:59:08Z","message":"cmd/pomerium"}
{"level":"info","address":"127.0.0.1:45921","time":"2023-10-27T12:59:08Z","message":"grpc: dialing"}
{"level":"info","outbound_port":"45921","databroker_urls":["http://127.0.0.1:5443"],"time":"2023-10-27T12:59:08Z","message":"config: starting databroker config source syncer"}
{"level":"info","service":"all","config":"databroker","checksum":"87b634f1ef4e5355","time":"2023-10-27T12:59:08Z","message":"config: updated config"}
{"level":"info","ts":"2023-10-27T12:59:08Z","msg":"config updated","controller":"bootstrap pod/pomerium-b6f9dc578-8659r","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","Pomerium":{"name":"global"},"namespace":"","name":"global","reconcileID":"ebbcfb42-dd93-4da3-9bea-90c21e15091a"}
{"level":"warn","time":"2023-10-27T12:59:08Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"info","time":"2023-10-27T12:59:08Z","logger":"maintenance","msg":"started background certificate maintenance","service":"autocert","cache":"0xc000e00400"}
{"level":"info","service":"autocert-manager","addr":":8080","time":"2023-10-27T12:59:08Z","message":"starting http redirect server"}
{"level":"info","grpc-port":"43873","http-port":"34961","outbound-port":"45921","metrics-port":"45687","debug-port":"45723","acme-tls-alpn-port":"33399","time":"2023-10-27T12:59:08Z","message":"server started"}
{"level":"info","time":"2023-10-27T12:59:08Z","message":"envoy: starting envoy process"}
{"level":"info","path":"/tmp/pomerium-envoy1907169401/envoy","checksum":"b1095c058415dfb2261e695a0f144311a7dc346b6eb47ecbb0a01b7de2c7299f","time":"2023-10-27T12:59:08Z","message":"running envoy"}
{"level":"info","pid":22,"time":"2023-10-27T12:59:08Z","message":"envoy: start monitoring subprocess"}
{"level":"info","time":"2023-10-27T12:59:08Z","message":"enabled authenticate service"}
{"level":"info","Algorithm":"ES256","KeyID":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","Public Key":{"use":"sig","kty":"EC","kid":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","crv":"P-256","alg":"ES256","x":"9z7uALUyv_T0qqxX6oxk6mJGJAKddecjSrppNPhxaQQ","y":"jBPRZtGanS3ArHtSmcWHDoctlcMycGNibHaZiFPzD2M"},"time":"2023-10-27T12:59:08Z","message":"authorize: signing key"}
{"level":"info","address":"127.0.0.1:45921","time":"2023-10-27T12:59:08Z","message":"grpc: dialing"}
{"level":"info","time":"2023-10-27T12:59:08Z","message":"enabled authorize service"}
{"level":"info","Algorithm":"ES256","KeyID":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","Public Key":{"use":"sig","kty":"EC","kid":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","crv":"P-256","alg":"ES256","x":"9z7uALUyv_T0qqxX6oxk6mJGJAKddecjSrppNPhxaQQ","y":"jBPRZtGanS3ArHtSmcWHDoctlcMycGNibHaZiFPzD2M"},"time":"2023-10-27T12:59:08Z","message":"authorize: signing key"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"initializing epoch 0 (base id=105364320, hot restart version=11.104)"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"statically linked extensions:"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.matching.http.input: envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.dubbo_proxy.protocols: dubbo"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.request_id: envoy.request_id.uuid"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.health_checkers: envoy.health_checkers.redis, envoy.health_checkers.thrift"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.route.early_data_policy: envoy.route.early_data_policy.default"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.path.match: envoy.path.match.uri_template.uri_template_matcher"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.retry_priorities: envoy.retry_priorities.previous_priorities"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.connection_handler: envoy.connection_handler.default"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.load_balancing_policies: envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.random, envoy.load_balancing_policies.round_robin"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  network.connection.client: default, envoy_internal"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.resolvers: envoy.ip"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.regex_engines: envoy.regex_engines.google_re2"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.common.key_value: envoy.key_value.file_based"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.http.header_validators: envoy.http.header_validators.envoy_default"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  quic.http_server_connection: quic.http_server_connection.default"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.dubbo_proxy.serializers: dubbo.hessian2"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.listener_manager_impl: envoy.listener_manager_impl.default"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.thrift_proxy.transports: auto, framed, header, unframed"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.quic.proof_source: envoy.quic.proof_source.filter_chain"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.dubbo_proxy.filters: envoy.filters.dubbo.router"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.upstream_codec"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.upstreams: envoy.filters.connection_pools.tcp.generic"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"HTTP header map info:"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  request header map: 672 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  request trailer map: 120 bytes: "}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"  response trailer map: 144 bytes: grpc-message,grpc-status"}
{"level":"info","time":"2023-10-27T12:59:08Z","message":"enabled databroker service"}
{"level":"info","address":"127.0.0.1:45921","time":"2023-10-27T12:59:08Z","message":"grpc: dialing"}
{"level":"info","address":"127.0.0.1:45921","time":"2023-10-27T12:59:08Z","message":"grpc: dialing"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"runtime: layers:\\n  - name: static_layer_0\\n    static_layer:\\n      re2:\\n        max_program_size:\\n          error_level: 1048576\\n          warn_level: 1024\\n      overload:\\n        global_downstream_max_connections: 50000"}
{"level":"info","service":"envoy","name":"admin","time":"2023-10-27T12:59:08Z","message":"admin address: /tmp/pomerium-envoy-admin.sock"}
{"level":"info","service":"envoy","name":"config","time":"2023-10-27T12:59:08Z","message":"loading tracing configuration"}
{"level":"info","service":"envoy","name":"config","time":"2023-10-27T12:59:08Z","message":"loading 0 static secret(s)"}
{"level":"info","service":"envoy","name":"config","time":"2023-10-27T12:59:08Z","message":"loading 1 cluster(s)"}
{"level":"info","service":"envoy","name":"config","time":"2023-10-27T12:59:08Z","message":"loading 0 listener(s)"}
{"level":"info","service":"envoy","name":"config","time":"2023-10-27T12:59:08Z","message":"loading stats configuration"}
{"level":"info","service":"envoy","name":"runtime","time":"2023-10-27T12:59:08Z","message":"RTDS has finished initialization"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:08Z","message":"cm init: initializing cds"}
{"level":"info","time":"2023-10-27T12:59:08Z","message":"enabled proxy service"}
{"level":"warn","time":"2023-10-27T12:59:08Z","message":"proxy: configuration has no policies"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"starting main dispatch loop"}
{"level":"info","addr":"127.0.0.1:43873","time":"2023-10-27T12:59:08Z","message":"starting control-plane gRPC server"}
{"level":"info","addr":"127.0.0.1:34961","time":"2023-10-27T12:59:08Z","message":"starting control-plane http server"}
{"level":"info","addr":"127.0.0.1:45723","time":"2023-10-27T12:59:08Z","message":"starting control-plane debug server"}
{"level":"info","addr":"127.0.0.1:45687","time":"2023-10-27T12:59:08Z","message":"starting control-plane metrics server"}
{"level":"info","name":"identity_manager","duration":30000,"time":"2023-10-27T12:59:08Z","message":"acquire lease"}
{"level":"info","time":"2023-10-27T12:59:08Z","message":"using in-memory store"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:08Z","message":"cds: add 7 cluster(s), remove 0 cluster(s)"}
{"level":"info","service":"identity_manager","syncer_id":"identity_manager","syncer_type":"","time":"2023-10-27T12:59:08Z","message":"initial sync"}
{"level":"info","type":"","time":"2023-10-27T12:59:08Z","message":"sync latest"}
{"level":"info","service":"identity_manager","syncer_id":"identity_manager","syncer_type":"","time":"2023-10-27T12:59:08Z","message":"listening for updates"}
{"level":"info","service":"identity_manager","sessions":0,"users":0,"time":"2023-10-27T12:59:08Z","message":"initial sync complete"}
{"level":"info","server_version":13410585744877032546,"record_version":0,"time":"2023-10-27T12:59:08Z","message":"sync"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:08Z","message":"cds: added/updated 6 cluster(s), skipped 1 unmodified cluster(s)"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:08Z","message":"cm init: all clusters initialized"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T12:59:08Z","message":"all clusters initialized. initializing init manager"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:08Z","message":"lds: add/update listener \\'http-ingress\\'"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:08Z","message":"lds: add/update listener \\'grpc-ingress\\'"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:08Z","message":"lds: add/update listener \\'metrics-ingress-4587868786652142039\\'"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:08Z","message":"lds: add/update listener \\'outbound-ingress\\'"}
{"level":"info","service":"envoy","name":"config","time":"2023-10-27T12:59:08Z","message":"all dependencies initialized. starting workers"}
{"level":"info","name":"ingress-controller","duration":30000,"time":"2023-10-27T12:59:09Z","message":"acquire lease"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","time":"2023-10-27T12:59:09Z","message":"initial sync"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","time":"2023-10-27T12:59:09Z","message":"sync latest"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","time":"2023-10-27T12:59:09Z","message":"listening for updates"}
{"level":"info","server_version":13410585744877032546,"record_version":0,"time":"2023-10-27T12:59:09Z","message":"sync"}
{"level":"info","time":"2023-10-27T12:59:09Z","message":"using in-memory registry"}
I1027 12:59:10.265817       1 request.go:690] Waited for 1.047578895s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/core.strimzi.io/v1beta2?timeout=32s
{"level":"info","ts":"2023-10-27T12:59:10Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":"localhost:44023"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"127.0.0.1:44023"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting EventSource","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","source":"kind source: *v1.Ingress"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting EventSource","controller":"pomerium-crd","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","source":"kind source: *v1.Pomerium"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting EventSource","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","source":"kind source: *v1.IngressClass"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting EventSource","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting EventSource","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","source":"kind source: *v1.Service"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting EventSource","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","source":"kind source: *v1.Endpoints"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting EventSource","controller":"pomerium-crd","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting Controller","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting Controller","controller":"pomerium-crd","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium"}
{"level":"info","ts":"2023-10-27T12:59:10Z","logger":"initial-sync","msg":"starting..."}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting workers","controller":"pomerium-crd","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","worker count":1}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"Starting workers","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","worker count":1}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"pomerium-crd","time":"2023-10-27T12:59:10Z","message":"get"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-27T12:59:10Z","message":"get"}
{"level":"info","ts":"2023-10-27T12:59:10Z","logger":"initial-sync","msg":"complete"}
{"level":"info","record-type":"type.googleapis.com/pomerium.config.Config","record-id":"pomerium-crd","time":"2023-10-27T12:59:10Z","message":"put"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-27T12:59:10Z","message":"get"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"new pomerium config applied","controller":"pomerium-crd","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","Pomerium":{"name":"global"},"namespace":"","name":"global","reconcileID":"87243220-8849-4c23-8410-5c7d79101220"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config_id":"pomerium-crd","version":1,"err_count":0,"time":"2023-10-27T12:59:10Z","message":"set db config info"}
{"level":"info","Algorithm":"ES256","KeyID":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","Public Key":{"use":"sig","kty":"EC","kid":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","crv":"P-256","alg":"ES256","x":"9z7uALUyv_T0qqxX6oxk6mJGJAKddecjSrppNPhxaQQ","y":"jBPRZtGanS3ArHtSmcWHDoctlcMycGNibHaZiFPzD2M"},"time":"2023-10-27T12:59:10Z","message":"authorize: signing key"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:10Z","message":"lds: add/update listener \\'http-ingress\\'"}
{"level":"info","time":"2023-10-27T12:59:10Z","message":"service registry reporter stopping"}
{"level":"warn","time":"2023-10-27T12:59:10Z","message":"proxy: configuration has no policies"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config":"databroker","checksum":"3b621a2b5eeeb2aa","time":"2023-10-27T12:59:10Z","message":"config: updated config"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"deprecated config","controller":"pomerium-crd","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","Pomerium":{"name":"global"},"namespace":"","name":"global","reconcileID":"87243220-8849-4c23-8410-5c7d79101220","key":"storage","docs":"https://www.pomerium.com/docs/topics/data-storage#persistence","msg":"please specify a persistent storage backend"}
{"level":"info","ts":"2023-10-27T12:59:10Z","msg":"config updated","controller":"pomerium-crd","controllerGroup":"ingress.pomerium.io","controllerKind":"Pomerium","Pomerium":{"name":"global"},"namespace":"","name":"global","reconcileID":"87243220-8849-4c23-8410-5c7d79101220"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-27T12:59:10Z","message":"get"}
{"level":"warn","time":"2023-10-27T12:59:10Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"pomerium-crd","time":"2023-10-27T12:59:10Z","message":"get"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-27T12:59:55Z","message":"get"}
{"level":"warn","time":"2023-10-27T12:59:55Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T12:59:55Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T12:59:55Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"info","record-type":"type.googleapis.com/pomerium.config.Config","record-id":"ingress-controller","time":"2023-10-27T12:59:56Z","message":"put"}
{"level":"warn","time":"2023-10-27T12:59:56Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"info","ts":"2023-10-27T12:59:56Z","msg":"new pomerium config applied","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","Ingress":{"name":"verify","namespace":"teste"},"namespace":"teste","name":"verify","reconcileID":"448fb9f9-8410-4067-9313-6ec3c99f0d77"}
{"level":"warn","time":"2023-10-27T12:59:56Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T12:59:56Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config_id":"ingress-controller","version":2,"err_count":0,"time":"2023-10-27T12:59:56Z","message":"set db config info"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config_id":"pomerium-crd","version":1,"err_count":0,"time":"2023-10-27T12:59:56Z","message":"set db config info"}
{"level":"warn","time":"2023-10-27T12:59:56Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"pomerium-crd","time":"2023-10-27T12:59:56Z","message":"get"}
{"level":"info","Algorithm":"ES256","KeyID":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","Public Key":{"use":"sig","kty":"EC","kid":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","crv":"P-256","alg":"ES256","x":"9z7uALUyv_T0qqxX6oxk6mJGJAKddecjSrppNPhxaQQ","y":"jBPRZtGanS3ArHtSmcWHDoctlcMycGNibHaZiFPzD2M"},"time":"2023-10-27T12:59:56Z","message":"authorize: signing key"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-27T12:59:56Z","message":"get"}
{"level":"warn","time":"2023-10-27T12:59:56Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T12:59:56Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T12:59:56Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:56Z","message":"cds: add 1 cluster(s), remove 0 cluster(s)"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T12:59:56Z","message":"cds: added/updated 1 cluster(s), skipped 0 unmodified cluster(s)"}
{"level":"info","time":"2023-10-27T12:59:56Z","message":"service registry reporter stopping"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config":"databroker","checksum":"a2b8b79f0310cf9b","time":"2023-10-27T12:59:56Z","message":"config: updated config"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-27T13:11:38Z","message":"get"}
{"level":"warn","time":"2023-10-27T13:11:38Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T13:11:38Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T13:11:38Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"info","record-type":"type.googleapis.com/pomerium.config.Config","record-id":"ingress-controller","time":"2023-10-27T13:11:38Z","message":"put"}
{"level":"info","ts":"2023-10-27T13:11:38Z","msg":"new pomerium config applied","controller":"pomerium-ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","Ingress":{"name":"verify","namespace":"teste"},"namespace":"teste","name":"verify","reconcileID":"1ce7bf22-ba05-43dc-a56f-e5f2ad58b778"}
{"level":"warn","time":"2023-10-27T13:11:38Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T13:11:38Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config_id":"ingress-controller","version":3,"err_count":0,"time":"2023-10-27T13:11:38Z","message":"set db config info"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config_id":"pomerium-crd","version":1,"err_count":0,"time":"2023-10-27T13:11:38Z","message":"set db config info"}
{"level":"warn","time":"2023-10-27T13:11:38Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"pomerium-crd","time":"2023-10-27T13:11:38Z","message":"get"}
{"level":"info","Algorithm":"ES256","KeyID":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","Public Key":{"use":"sig","kty":"EC","kid":"ee281ec4fc46c9d43b796aa94d943e533a0b44ae5a1e9c7bd7b955ae8b839bc5","crv":"P-256","alg":"ES256","x":"9z7uALUyv_T0qqxX6oxk6mJGJAKddecjSrppNPhxaQQ","y":"jBPRZtGanS3ArHtSmcWHDoctlcMycGNibHaZiFPzD2M"},"time":"2023-10-27T13:11:38Z","message":"authorize: signing key"}
{"level":"info","service":"envoy","name":"upstream","time":"2023-10-27T13:11:38Z","message":"lds: add/update listener \\'http-ingress\\'"}
{"level":"info","time":"2023-10-27T13:11:38Z","message":"service registry reporter stopping"}
{"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","service":"all","config":"databroker","checksum":"8c96d714c3b6d387","time":"2023-10-27T13:11:38Z","message":"config: updated config"}
{"level":"info","service":"envoy","name":"main","time":"2023-10-27T13:14:08Z","message":"shutting down parent after drain"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"ingress-controller","time":"2023-10-27T13:18:41Z","message":"get"}
{"level":"warn","time":"2023-10-27T13:18:41Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T13:18:41Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T13:18:41Z","message":"config: set_authorization_header is deprecated, use ${pomerium.id_token} or ${pomerium.access_token} in set_request_headers instead"}
{"level":"warn","time":"2023-10-27T13:18:41Z","message":"neither `autocert`, `insecure_server` or manually provided certificates were provided, server will be using a self-signed certificate"}
{"level":"info","type":"type.googleapis.com/pomerium.config.Config","id":"pomerium-crd","time":"2023-10-27T13:18:41Z","message":"get"}

Describe pomerium global:

Name:         global
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  ingress.pomerium.io/v1
Kind:         Pomerium
Metadata:
  Creation Timestamp:  2023-10-27T12:59:08Z
  Generation:          1
  Resource Version:    2271191
  UID:                 568451c2-aeb9-4cb9-b6f5-d6c38116433a
Spec:
  Authenticate:
    URL:  https://authenticate.pomerium.app
  Certificates:
    pomerium/pomerium-wildcard-tls
  Secrets:  pomerium/bootstrap
Status:
  Ingress:
    teste/verify:
      Observed At:          2023-10-27T13:18:41Z
      Observed Generation:  3
      Reconciled:           true
  Settings Status:
    Observed At:          2023-10-27T12:59:10Z
    Observed Generation:  1
    Reconciled:           true
    Warnings:
      storage: please specify a persistent storage backend, please see https://www.pomerium.com/docs/topics/data-storage#persistence
Events:
  Type     Reason      Age                  From                                    Message
  ----     ------      ----                 ----                                    -------
  Normal   Updated     22m                  bootstrap pod/pomerium-b6f9dc578-8659r  config updated
  Warning  Validation  22m                  pomerium-crd                            storage: please specify a persistent storage backend, please see https://www.pomerium.com/docs/topics/data-storage#persistence
  Normal   Updated     22m                  pomerium-crd                            config updated
  Normal   Updated     3m18s (x4 over 22m)  pomerium-ingress                        teste/verify: config updated
wasaga commented 11 months ago

please disregard that error message it is coming from pomerium core on very initial start before the configuration from kubernetes is synced to it.

your problem is not with certs. you do not have traffic coming to pomerium-proxy service. there is not a single access log entry in your log output. please do kubectl describe -n pomerium pomerium-proxy and see whether it has correct external IP address that matches your DNS entries.

please do curl -kv https://your-server/.well-known/pomerium/jwks.json. the output should look like below. 

< HTTP/2 200 
< accept-ranges: bytes
< cache-control: max-age=60
< content-length: 236
< content-type: application/json
< etag: "19e1c1febb9e21dd"
< vary: Accept-Encoding,Origin
< date: Fri, 27 Oct 2023 14:19:24 GMT
< x-envoy-upstream-service-time: 2
< x1: v1
< x13: v3
< x2: v2
< server: envoy
< x-request-id: e4e115cd-e13f-9549-9ccd-a2d639901166
< 

{"keys":[{"use":"sig","kty":"EC","kid":"d20cf4224985ba04567cafb69aa2b9e5912c68a8a9bd316526b5e89b9dc97186","crv":"P-256","alg":"ES256","x":"PRhFADKJj6i6bFq9TMaKEismYNlS1dNaWIII3PUebYg","y":"dDObt0PxTKUE-7yk5AMimW4cEepLWB1XSZF06pG2e0Y"}]}
andreyolv commented 11 months ago

It was a problem with my loadbalancer external IP. Now I can connect, but only I use the annotation ingress.pomerium.io/allow_public_unauthenticated_access: "true" in my ingress, then it works without problems.

However, with the Azure IDP, when accessing the page I am redirected to log into Azure, and at the end of the login it returns me: 500 Internal Server Error Internal Server Error: Internal/urlutil: malformed unix timestamp field

Pomerium deployment logs:

{"level":"info","type":"type.googleapis.com/user.ServiceAccount","query":"","offset":0,"limit":1,"filter":{"$or":[{"id":""},{"$index":""}]},"time":"2023-11-09T13:11:27Z","message":"query"}
{"level":"info","type":"type.googleapis.com/session.Session","query":"","offset":0,"limit":1,"filter":{"$or":[{"id":""},{"$index":""}]},"time":"2023-11-09T13:11:27Z","message":"query"}
{"level":"info","service":"authorize","request-id":"981ddfed-8bf0-44c5-8578-04e48eeed69e","check-request-id":"981ddfed-8bf0-44c5-8578-04e48eeed69e","method":"GET","path":"/","host":"myhost.myhost","ip":"myipXXXX","user":"","email":"","allow":false,"allow-why-false":["user-unauthenticated"],"deny":false,"deny-why-false":[],"time":"2023-11-09T13:11:27Z","message":"authorize check"}
{"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"myhost.myhost","path":"/","user-agent":"Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0","referer":"","forwarded-for":"myipXXXX","request-id":"981ddfed-8bf0-44c5-8578-04e48eeed69e","duration":16.246075,"size":1423,"response-code":302,"response-code-details":"ext_authz_denied","time":"2023-11-09T13:11:27Z","message":"http-request"}

Ingress:

  annotations:
    ingress.pomerium.io/allow_websockets: 'true'
    ingress.pomerium.io/pass_identity_headers: 'true'
    ingress.pomerium.io/allow_any_authenticated_user: 'true'
    ingress.pomerium.io/secure_upstream: 'true'
    ingress.pomerium.io/policy: |
      allow:
        and:
        - claim/groups: 'XXXXXXXXXMYADGROUPXXXXXXX'
andreyolv commented 11 months ago

I discovered the problem, my identityProvider secret containing client_id and client_secret had an enter (\n) at the end of values :(

Now it's working, thank you very much for your support

cmo-pomerium commented 11 months ago

@andreyolv Thanks for following up - I'm closing this issue assuming your problem has been solved!