Egress proxy RBAC - Githubissues

conqerAtapple commented 3 years ago

Title: Add support for egress proxy RBAC

Description: If envoy is used as egress proxy, it looks like currently there is not much support for RBAC. From https://github.com/envoyproxy/envoy/issues/10336, the only way to do this might be to use a config server or DNS records. If we need add further rich semantics (RBAC), the recommendation is to write a upstream connection filter or something along those lines. One way to achieve this might be to add a filter extension which extends from ListenerFilter and add RBAC like configuration for it. If so, are there any existing examples of this?

The deployment architecture could be stand alone envoy (not side car). So this might mean a tproxy like setup .

I am happy to add this feature but wanted to clarify the approach and high level design.

https://github.com/envoyproxy/envoy/issues/10336

@alyssawilk

mattklein123 commented 3 years ago

@conqerAtapple can you describe a bit more what type of RBAC rules you are after? This has come up several times and I would like to make sure we are collecting all similar use cases.

conqerAtapple commented 3 years ago

@conqerAtapple can you describe a bit more what type of RBAC rules you are after? This has come up several times and I would like to make sure we are collecting all similar use cases.

The general use case here is to :

Deny by default
Allow only subset of DNS /resolved ip/CIDR
In addition to above, allow only certain Http paths (or header/tokens)
Add principal from TLS context or even a custom identity manager for authorization

Not sure if the above covers everything but I think i got the general idea captured.

thanks @mattklein123

mattklein123 commented 3 years ago

Couldn't pretty much all of the above be done today using existing filters like RBAC, IP tagging, etc.?

The only part I'm unclear about is DNS restrictions. Do you mean you actually want to put restrictions on the returned resolutions from the DNS resolver?

conqerAtapple commented 3 years ago

Couldn't pretty much all of the above be done today using existing filters like RBAC, IP tagging, etc.? I thought RBAC filter is designed for ingress. Are there examples for RBAC filter being used for egress? Does IP tagging work for TCP proxying?

The only part I'm unclear about is DNS restrictions. Do you mean you actually want to put restrictions on the returned resolutions from the DNS resolver? Oh i meant wildcard matching for egress. There might be a way to express "I only want to allow connections to upstream hosts that matches name good" but couldnt find an example.

conqerAtapple commented 3 years ago

Adding an example configuration I tried:

                  virtual_hosts:                                                                     
                    - name: local_service                                                            
                      domains: ["*cnn.*"]                                                             
                      routes:                                                                        
                        - match:                                                                     
                            prefix: "/"                                                              
                          route:                                                                     
                            cluster: dynamic_forward_proxy_cluster                                   
                          typed_per_filter_config:                                                   
                              envoy.filters.http.rbac:                                               
                                "@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBACPerRoute
                                rbac:                                                                
                                    rules:                                                           
                                        action: ALLOW                                                
                                        policies:                                                    
                                            "per-route-rule":                                        
                                                permissions:                                         
                                                    - any: true                                      
                                                principals:                                          
                                                    - any: true                                      
                    - name: local_service2                                                           
                      domains: ["www.google.com"]                                                    
                      routes:                                                                        
                        - match:                                                                     
                            prefix: "/"                                                              
                          route:                                                                     
                            cluster: dynamic_forward_proxy_cluster2                                  
                          typed_per_filter_config:                                                   
                              envoy.filters.http.rbac:                                               
                                "@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBACPerRoute
                                rbac:                                                                
                                    rules:                                                           
                                        action: DENY                                                 
                                        policies:                                                    
                                            "per-route-rule":                                        
                                                permissions:                                         
                                                    - any: true                                      
                                                principals:                                          
                                                    - any: true

Expected results: request for www.google.com should have been denied access. Observed result: Envoy successfully proxies requests for www.google.com .

Is this because i am misconfiguring ? Or is it because of the design of rbac filter which is intended for ingress?

@mattklein123

conqerAtapple commented 3 years ago

update:

I was able to enforce the test policy by adding rbac http filter as:


                  - name: envoy.filters.http.rbac                                                    
                    typed_config:                                                                    
                        "@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC      
                        rules:                                                                       
                            action: DENY                                                             
                            policies:                                                                
                                "per-route-rule":                                                    
                                    permissions:                                                     
                                        - any: true                                                  
                                    principals:                                                      
                                        - any: true

Conclusion: Configuration for route specific rbac policy needs a http filter also to drive the policy.

conqerAtapple commented 3 years ago

Keeping this issue open to collect any related requirements @mattklein123

mattklein123 commented 3 years ago

In general "egress" vs. "ingress" is an abstract concept that Envoy doesn't really care about (this is not entirely true as it leaks into a few places in the API, but not many). For all of this type of config it doesn't really matter if Envoy is being used as a forward proxy or a reverse proxy.

Can you clarify what other features you need now that you have this part sorted out?

conqerAtapple commented 3 years ago

In general "egress" vs. "ingress" is an abstract concept that Envoy doesn't really care about (this is not entirely true as it leaks into a few places in the API, but not many). For all of this type of config it doesn't really matter if Envoy is being used as a forward proxy or a reverse proxy.

I was a bit confused. Thanks a lot for the clarification.

Can you clarify what other features you need now that you have this part sorted out?

We are still collecting all the asks. Will update or close this issue based on that.

Appreciate your help @mattklein123 .

conqerAtapple commented 3 years ago

An interesting use case which does not seem to work for us is - rbac on destination IP addresses where IP was resolved by DNS. For example: For CONNECT request curl -v -x 192.168.192.3:80 www.example.com, if we had rbac policy DENY for ip CIDR that matches www.example.com, the policy would not be enforced. This issue might not be just applicable to connect case, but any case where destination ip address is not known at rbac filter processing time.

Edit:

Maybe a way to achieve this is for invoking rbac when a connect event or DNS resolve event is completed.

Looking at the code, the rbac filter does not seem to know the upstream host information when executing policies. Would it make sense to pass cluster information to the policy engine?

@mattklein123

conqerAtapple commented 3 years ago

Proposal:

Add API to StreamInfo to add/update the resolved upstream host addresses
Have dynamic forward proxy(dynamic_forward_proxy/proxy_filter.cc) use the above API to update the resolved addresses.
Add new policy to RBAC filter - upstream which could consist of :
- upstream ip and port: match by CIDR range
- upstream host: regex match

Please let me know.

ping @mattklein123

mattklein123 commented 3 years ago

Yes I think this makes sense to me at a high level. cc @yangminzhu

conqerAtapple commented 3 years ago

Question: For the generic case (not dynamic proxy) where the DNS is resolved by the cluster, how do I get the ClusterManager for a request in StreamInfo? This is so that we can filter the resolved IP addresses for a cluster for RBAC.

@mattklein123 @yangminzhu

alyssawilk commented 3 years ago

So I could imagine doing this by having an RBAC filter after the DFP filter - right now DFP waits on DNS resolution before proceeding. That said I think we were planning on rewiring that, and even if we didn't it's not going to play with happy eyeballs support. cc @RyanTheOptimist to join the discussion.

@conqerAtapple can you talk a bit more about the use case blocking specific destination endpoints by IP rather than hostname?

mattklein123 commented 3 years ago

Yeah I agree with @alyssawilk that I had assumed that RBAC would run after DFP, and could grab info from stream info on the upstream host target that has been selected.

conqerAtapple commented 3 years ago

For DFP based route, the above proposal which includes using the resoled logical DNS address for RBAC policy makes sense I think. For the generic case, where the DNS is resolved asynchronously, we might have to use upstream filters as you suggested.

The use case we are trying to solve is where the client(s) are connecting to upstreams which need to be filtered by some CIDR policy. An example is where clients could be connecting to a set of unknown / dynamic services on the internet . We might want to apply policies on these dynamic connection requests.

@alyssawilk

RyanTheOptimist commented 3 years ago

If I'm understanding correctly, it sounds like we would like to be able to block access to to upstream hosts which resolve to a particular ip or set of IP. What should happen if a host resolved to two IPs, one which is forbidden and one which is not? Is the idea that the request should use the non-blocked IP and succeed? Fail because of the blocked IP? Something else?

conqerAtapple commented 3 years ago

Good question. Maybe we should be able to specify policy where we can pick between all-ips(AND) vs one-of-ips (OR) which satisfy the condition ?

yangminzhu commented 3 years ago

So I think this is not about the Egress proxy RBAC but instead to support the dynamically resolved upstream IP in the RBAC API.

One thing is you need to be careful when using the resolved upstream IP because they could change anytime and make your RBAC policy not working as expected, i.e. unexpected reject or policy bypass (security issue). I'm still not sure why won't you use the host name in the RBAC policy? the host seems a better alternative than the resolved IP addresses.

Good question. Maybe we should be able to specify policy where we can pick between all-ips(AND) vs one-of-ips (OR) which satisfy the condition ?

I think the problem is the load balancing does not have the information in the RBAC policy and could pick one of the IPs that will always be rejected by the RBAC filter.

conqerAtapple commented 3 years ago

So I think this is not about the Egress proxy RBAC but instead to support the dynamically resolved upstream IP in the RBAC API.

I guess thats is right. Although the problem manifests especially for egress case since we currently dont have a way to express upstream information in the policy.

One thing is you need to be careful when using the resolved upstream IP because they could change anytime and make your RBAC policy not working as expected, i.e. unexpected reject or policy bypass (security issue). I'm still not sure why won't you use the host name in the RBAC policy? the host seems a better alternative than the resolved IP addresses.

One reason being that we would like to be closer to the ground truth and not have a dependency on the DNS server.

Good question. Maybe we should be able to specify policy where we can pick between all-ips(AND) vs one-of-ips (OR) which satisfy the condition ?

I think the problem is the load balancing does not have the information in the RBAC policy and could pick one of the IPs that will always be rejected by the RBAC filter.

That is correct. In its current state, RBAC filter does not have any information on upstream hosts. Maybe this could be changed via referencing the LB priority set from RBAC filter? Or have RBAC filter observe DNS resolution ?

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

conqerAtapple commented 3 years ago

not stale. Working on implementation of a matcher extension https://github.com/envoyproxy/envoy/pull/17645

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] commented 2 years ago

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

envoyproxy / envoy

Egress proxy RBAC #17410