Closed arcivanov closed 3 years ago
cc @yangminzhu
This should be fixed by https://github.com/istio/istio/pull/25934, but that PR is only in 1.8.
@xulingqing could we cherry-pick the fix to 1.7?
@yangminzhu what is the expectation? It still NACKs with that PR
@yangminzhu what is the expectation? It still NACKs with that PR
The expectation is to accept the config but the JWT verification will always fail due to the use of the fake jwks.
It shouldn't NACK but if it did, should look into the PR to see why it's not working as expected, John, how did you reproduce the NACK?
apiVersion: "security.istio.io/v1beta1"
kind: "RequestAuthentication"
metadata:
name: "default"
namespace: "istio-system"
spec:
jwtRules:
- issuer: "test-issuer-1@istio.io"
jwksUri: "https://httpbin.org/get"
- issuer: "test-issuer-2@istio.io"
jwksUri: "https://raw.githubusercontent.com/istio/istio/master/tests/common/jwt/jwks.json"
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: egressgateway
namespace: "istio-system"
spec:
selector:
matchLabels:
app: istio-egressgateway
rules:
- to: # only allow /allow for company.com
- operation:
paths: ["/allow"]
hosts: ["www.company.com"]
- to: # checks only a call 443 over istio mutual without JWT
- operation:
hosts: ["a-only.com"]
from:
- source:
principals: ["cluster.local/ns/default/sa/a"]
- to: # checks a and c can call 443 over istio mutual with JWT
- operation:
hosts: ["jwt-only.com"]
from:
- source:
requestPrincipals: ["test-issuer-1@istio.io/sub-1"]
- to: # checks only a can call 443 over istio mutual with JWT
- operation:
hosts: ["jwt-and-a-only.com"]
from:
- source:
requestPrincipals: ["test-issuer-1@istio.io/sub-1"]
principals: ["cluster.local/ns/default/sa/a"]
---
# The following policy redirects the request through egress gateway.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: test-egress
namespace: default
spec:
selector:
istio: egressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "www.company.com"
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: ISTIO_MUTUAL
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: route-via-egressgateway
namespace: default
spec:
hosts:
- "www.company.com"
gateways:
- test-egress
- mesh
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
port:
number: 80
weight: 100
- match:
- gateways:
- test-egress
port: 80
route:
- destination:
host: b.default.svc.cluster.local
port:
number: 8090
weight: 100
headers:
request:
add:
x-egress-test: "handled-by-egress-gateway"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: route-via-egressgateway-2
namespace: default
spec:
hosts:
- "a-only.com"
- "jwt-only.com"
- "jwt-and-a-only.com"
gateways:
- test-egress
- mesh
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
port:
number: 443
weight: 100
- match:
- gateways:
- test-egress
port: 443
route:
- destination:
host: b.default.svc.cluster.local
port:
number: 8090
weight: 100
headers:
request:
add:
x-egress-test: "handled-by-egress-gateway"
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
name: "test-egress"
namespace: default
spec:
host: "istio-egressgateway.istio-system.svc.cluster.local"
trafficPolicy:
portLevelSettings:
- port:
number: 443
tls:
mode: ISTIO_MUTUAL
---
Maybe the difference is 404 vs 200 but without a valid JWKS response?
Maybe the difference is 404 vs 200 but without a valid JWKS response?
Yes, the PR only addresses the 404 case. It could be improved to try to address the 200 with invalid JWKS case, istiod could do some basic validation even it might just be a subset of the validation done by Envoy.
not stale
Using k3s v1.21.1 and istio 1.10.1 if the RequestAuthentication fails the istio-ingressgateway never starts. Example ra
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: p01ra
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
jwtRules:
- issuer: "https://k6k.p01.net/auth/realms/p01project"
jwksUri: "https://k6k.p01.net/auth/realms/p01project/protocol/openid-connect/certs"
forwardOriginalToken: true
outputPayloadToHeader: x-jwt-payload
Here k6k.p01.net was not resolved
istio-proxy@istio-ingressgateway-7749f5c9df-nhk45:/$ curl https://k6k.p01.net/auth/realms/p01project/protocol/openid-connect/certs
curl: (6) Could not resolve host: k6k.p01.net
istio-proxy@istio-ingressgateway-7749f5c9df-nhk45:/$
and the istio-ingressgateway never starts and log say
2021-06-17T14:50:11.416751Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 17 successful, 0 rejected; lds updates: 0 successful, 17 rejected
2021-06-17T14:50:13.417443Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 17 successful, 0 rejected; lds updates: 0 successful, 17 rejected
The only solution to start the istio-ingressgateway is to delete the ra
@MirtoBusico I am currently having the same problem, and the resolution that you suggested worked like a charm! But this cannot be accepted as a resolution to this issue. Is there any update on this issue from the istio
team?
Well I've found a solution. See issues 29366 and 32421
To recap: create al privata CA and install istio using istio operator inserting the CA pem Create a certificate signed by this CA for keycloak (here k6k.m.net) The configuration file (istio-operator) contains
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: example-istiocontrolplane
spec:
profile: demo
meshConfig:
accessLogEncoding: TEXT
accessLogFile: "/dev/stdout"
accessLogFormat: ""
outboundTrafficPolicy:
mode: ALLOW_ANY
values:
pilot:
jwksResolverExtraRootCA: |
-----BEGIN CERTIFICATE-----
MIIEKTCCAxGgAwIBAgIUNjT4liTyGcGTewYUb2NZj4oWMGowDQYJKoZIhvcNAQEL
BQAwgaMxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UEBwwEUm9t
ZTEcMBoGA1UECgwTQnVzaWNvIE1pcnRvIFNpbHZpbzETMBEGA1UECwwKTGFib3Jh
dG9yeTEcMBoGA1UEAwwTQnVzaWNvIE1pcnRvIFNpbHZpbzEkMCIGCSqGSIb3DQEJ
ARYVbWlydG9idXNpY29AZ21haWwuY29tMB4XDTIxMDIxOTExNDA1N1oXDTMxMDIx
NzExNDA1N1owgaMxCzAJBgNVBAYTAklUMQ4wDAYDVQQIDAVJdGFseTENMAsGA1UE
BwwEUm9tZTEcMBoGA1UECgwTQnVzaWNvIE1pcnRvIFNpbHZpbzETMBEGA1UECwwK
TGFib3JhdG9yeTEcMBoGA1UEAwwTQnVzaWNvIE1pcnRvIFNpbHZpbzEkMCIGCSqG
SIb3DQEJARYVbWlydG9idXNpY29AZ21haWwuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAtt+v2C9p60rx1Q/yOQsgsis/dBNAo4efFlyN0Ibs9ts4
a0LRQp4EwZpv9+tysVqGZvN4fJ99mdyiJHiFlchMfq4t+OzOHnym7Yi5khHS5/rv
TwvwD+1igMny1FybVOxSlfdZGF5mhgRFD6mQod/hix5QJgegmygEhj0VV/i2rZhH
FW0oMR1smLfALQQZhGJ//TgCjNlpK2D6zlxEXIr+QLyAje+kQyyqkJefY/Vggg9m
gRFV7gu3MKKGE5B+ESfqhZbUqsKz/rsxs2L23Selp3FM+DhKIC8DM06dgAh7DYUQ
IeIe9HT7+RTTs3KM7ArDXrF+BF+D8O/a4D3YBDhHswIDAQABo1MwUTAdBgNVHQ4E
FgQUGHqUwQ6vRxvpB7+MCUEAwmUgbtAwHwYDVR0jBBgwFoAUGHqUwQ6vRxvpB7+M
CUEAwmUgbtAwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAVThR
2NmNuey07twCW4/B8v6zOCeO/n8Z+waRw1XK9XmA+QPUTi+bLKvfx+7RVgaZD6SR
EQHMCshGD7In5PbSBsrp6ocmCvcopd2iqvt2GLvJHuZy7hI+RgaMgQo9hhThHf9e
FoW3C41Mm3ofvUubIKLEFKnCvxOsD+Ayyg69pGwNM+PQK1XvXWWYm8eroPICxriq
8ULUgAc+leGiHbAKSXGLj6U/njyRzkxAmXzkierT1DFDHa9sZst7nCaSycKY7rBj
GU2xRTOpYrQHcsaBZBjTT8/ag2IasCzFVeZ5+bMmTaDus5QT3tIubR8ukTx5jf0S
BiQTL+/xni6Fxkrl3Q==
-----END CERTIFICATE-----
And I installed with:
kubectl create ns istio-system
kubectl apply -f istio-operator
Now I can use the authentication file
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: m-ra
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
jwtRules:
- issuer: "https://k6k.m.net/auth/realms/m-project"
jwksUri: "https://k6k.m.net/auth/realms/m-project/protocol/openid-connect/certs"
forwardOriginalToken: true
outputPayloadToHeader: x-jwt-payload
With this everything works as expected
🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-05-18. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.
Created by the issue and PR lifecycle manager.
In istio 1.7.3, istio-ingressgateway with RequestAuthentication applied segfaults in Envoy and fails to come up if JWKS fails to return a valid payload.
There are three scenarios that all fail identically:
jwtRules.issuer
refers to a URL that is either behind the ingress itself or is otherwise unavailable, causing OIDC.well-known/openid-configuration
lookup to fail and thus no JWKS being available.jwtRules.jwksUri
is specified but is similarly unavailable as in [1].jwtRules.jwksUri
has a typo causing JWKS to not be available.This is related to #25578
[ ] Docs [ ] Installation [X] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [X] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [X] Upgrade
Expected behavior
No segfault, Ingress comes up, eventually obtains JWKS denying access until it is resolved.
Steps to reproduce the bug
RequestAuthentication
with either 1.1.issuer
that has a URL that when inferred points to inaccessible (permanently or temporarily, including due to starting ingress traversal requirement)openid-configuration
1.2.jwksUri
that has a URL that points to inaccessible (permanently or temporarily, including due to starting ingress traversal requirement)Version (include the output of
istioctl version --remote
andkubectl version --short
andhelm version
if you used Helm)How was Istio installed?
N/A
Environment where bug was observed (cloud vendor, OS, etc)
AWS/Ubuntu 20.04 LTS
Additionally, please consider running
istioctl bug-report
and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.