search

Orange-OpenSource / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://orange-opensource.github.io/nifikop/
Apache License 2.0
128 stars 34 forks source link

Untrusted proxy error after logging in with OIDC #23

Closed clettieri closed 4 years ago

clettieri commented 4 years ago

Type of question

General Context/ Troubleshooting

Question

What did you do? Started a secured, 3 node cluster in EKS. OpenID connect is configured.

What did you expect to see? After connecting to cluster from web browser and logging in from OpenID conncet, I expected to see the NiFi UI.

What did you see instead? Under which circumstances? Instead I see:

Untrusted proxy nifi-cluster-1-node.nifi-cluster-headless.nifi.svc.cluster.local, O=cert-manager

Environment

Additional context I am using an ELB that has stickness turned on so I think this issue here is with NiFi authorizers, not authenticating via OpenID connect. I'm using the self-signed certs issued by nifi and then I have a separate internal load balancer with an ACM cert and external-dns configured. This component is working.

I have initial admin e-mail configured as well. I'm also using the suggested NiFi properties mapping config from the nifikop docs.

I've tried researching this untrusted proxy error and it seems there are two common suggestions:

  1. Make sure the host name in authorizers.xml is exact match of cert. (verified this multiple times, even tried patching explicitly what I saw on the cert)
  2. Provide /proxy permission to the initial admin user. (haven't been able to try this)

Since authorizers.xml is generated by nifikop, I don't think its a typo that's causing the issue here. Any thoughts on what to try next?

Thanks in advance!

erdrix commented 4 years ago

Can you share with me your NifiCluster.Spec.rReadOnlyConfig.NifiProperties.OverrideConfigs ?

I think the error may come from the nifi.security.identity.mapping properties, we recently fixed the example pattern: https://github.com/Orange-OpenSource/nifikop/blob/master/config/samples/tls_secured_nificluster.yaml#L29 : 

nifi.security.identity.mapping.pattern.dn=CN=(.*)(?:, (?:O|OU)=.*)?
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE
clettieri commented 4 years ago

I appreciate that @erdrix. I'm using that pattern already it looks like, here it is copied directly from conf/nifi.properties

nifi.security.identity.mapping.pattern.dn=CN=(.*)(?:, (?:O|OU)=.*)?
nifi.security.identity.mapping.transform.dn=NONE
nifi.security.identity.mapping.value.dn=$1
clettieri commented 4 years ago

Some more context, in nifi-user.log on the "untrusted proxy node" - I also am seeing:

020-08-05 12:14:44,868 WARN [NiFi Web Server-17] o.a.n.w.a.c.IllegalStateExceptionMapper java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.. Returning Conflict response.
java.lang.IllegalStateException: Kerberos ticket login not supported by this NiFi.

...

2020-08-05 12:14:44,900 INFO [NiFi Web Server-21] o.a.n.w.a.c.IllegalArgumentExceptionMapper java.lang.IllegalArgumentException: The login request identifier was not found in the request. Unable to continue.}. Returning Bad Request} response.
java.lang.IllegalArgumentException: The login request identifier was not found in the request. Unable to continue.

...

2020-08-05 12:14:44,943 INFO [NiFi Web Server-21] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[anonymous], groups[none] does not have permission to access the requested resource. Unknown user with identity 'anonymous'. Returning Unauthorized response.

Kerberos is still commented out in the conf/login-identity-providers.xml and I have OIDC configured, not sure what could be causing this error message to happen but it did occur when attempting to login via OIDC.

clettieri commented 4 years ago

Checking the logs on the other nodes when logging in:

Node 0 nifi-user.log

2020-08-05 12:35:25,361 INFO [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<my.email@domain.com><CN=nifi-cluster-1-node.nifi-cluster-headless.nifi.svc.cluster.local, O=cert-manager>) GET https://nifi-cluster-0-node.nifi-cluster-headless.nifi.svc.cluster.local:8443/nifi-api/flow/current-user (source ip: 10.6.114.87)
2020-08-05 12:35:25,363 WARN [NiFi Web Server-20] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy nifi-cluster-1-node.nifi-cluster-headless.nifi.svc.cluster.local, O=cert-manage

Is this <my.email@domain.com><CN=nifi-cluster-1-node.nifi-cluster-headless.nifi.svc.cluster.local, O=cert-manager> the identity that needs to be in authorizers.xml? Meaning I need to handle a mapping for when its my OIDC email and the node?

erdrix commented 4 years ago

No, that part is correct, because you come with your user email on a node, and the node wants to proxy your user to propagate the action you are doing. That's why the error say untrusted proxy error.

There are two possible reasons :

  1. the node don't have the associated proxy userpolicies.
  2. the way the node is identified doesn't match with the one defined into users.xml file.

Can you share with me your authorization.xml, users.xml and nifi.properties (only the part about nifi.security.identity) files (take them directly from inside the container).

clettieri commented 4 years ago

Thanks for confirming that's OK.

nifi.properties

nifi.security.identity.mapping.pattern.dn=CN=(.*)(?:, (?:O|OU)=.*)?
nifi.security.identity.mapping.transform.dn=NONE
nifi.security.identity.mapping.value.dn=$1

authorizations.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
    <policies>
        <policy identifier="f99bccd1-a30e-3e4a-98a2-dbc708edc67f" resource="/flow" action="R">
            <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9"/>
        </policy>
        <policy identifier="b8775bd4-704a-34c6-987b-84f2daf7a515" resource="/restricted-components" action="W">
            <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9"/>
        </policy>
        <policy identifier="627410be-1717-35b4-a06f-e9362b89e0b7" resource="/tenants" action="R">
            <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9"/>
        </policy>
        <policy identifier="15e4e0bd-cb28-34fd-8587-f8d15162cba5" resource="/tenants" action="W">
            <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9"/>
        </policy>
        <policy identifier="ff96062a-fa99-36dc-9942-0f6442ae7212" resource="/policies" action="R">
            <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9"/>
        </policy>
        <policy identifier="ad99ea98-3af6-3561-ae27-5bf09e1d969d" resource="/policies" action="W">
            <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9"/>
        </policy>
        <policy identifier="2e1015cb-0fed-3005-8e0d-722311f21a03" resource="/controller" action="R">
            <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9"/>
        </policy>
        <policy identifier="c6322e6c-4cc1-3bcc-91b3-2ed2111674cf" resource="/controller" action="W">
            <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9"/>
        </policy>
        <policy identifier="287edf48-da72-359b-8f61-da5d4c45a270" resource="/proxy" action="W">
            <user identifier="c83814f8-259f-334f-99a5-faa525109a87"/>
            <user identifier="c9bb30b5-404f-3174-9e6f-9ccd23b27f07"/>
            <user identifier="faa75827-35f6-3a33-9f82-869480c1edd4"/>
        </policy>
    </policies>
</authorizations>

users.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
    <groups/>
    <users>
        <user identifier="c83814f8-259f-334f-99a5-faa525109a87" identity="nifi-cluster-2-node.nifi-cluster-headless.nifi.svc.cluster.local"/>
        <user identifier="c9bb30b5-404f-3174-9e6f-9ccd23b27f07" identity="nifi-cluster-1-node.nifi-cluster-headless.nifi.svc.cluster.local"/>
        <user identifier="faa75827-35f6-3a33-9f82-869480c1edd4" identity="nifi-cluster-0-node.nifi-cluster-headless.nifi.svc.cluster.local"/>
        <user identifier="736bfec2-119c-3efb-9465-a38bf0bb05d9" identity="my.email@domain.com"/>
    </users>
</tenants>

Really appreciate the help here @erdrix ! One thing I came across while researching this issue, was that the admin user might need the /proxy W permission as well. Haven't found an easy way to test this out yet.

clettieri commented 4 years ago

UPDATE: A 1 node cluster works - after logging in with OIDC I see the NiFi UI as expected.

I'm guessing the issue might be in the load balancer? Even though I've enabled sticky sessions on my AWS internal LB, after the OIDC log in maybe it's not working as expected.

AWS had a problem creating a load balancer service with ClientIP session affinity. Maybe my workaround here is what was causing the issue. I'll approach the problem from this angle and see what happens.

erdrix commented 4 years ago

I don't think that's the cause :/ A 1-node cluster works, because the nodeyou land on doesn't have to proxy the request to another node, so even though the node username is still bad, no checking is done, so it will work ...

By checking the regexp I think it's still wrong :/ Try this one : CN=([^,]*)(?:, (?:O|OU)=.*)? it should work !

Can you tell me which version of cert-manager you are using ? (In some cases it seems like part O is added and for other ones it's not, and so far we only tested it without ...)

clettieri commented 4 years ago

It worked 🎉

Thank you so much @erdrix

For the record, I am using cert-manager v0.13.