LoadBalancer issues & Questions

[mishushakov]

Hi, guys Just wanted to report some things I noticed again, when using your Managed Kubernetes + LB.

First of all, when I'm trying to resolve the domain, which points to LB, from inside of a cluster, it gets a request timeout:

It's not critical for me, since my services talk to each other by their DNS names. Maybe you could look at it and tell us what's the reason why it happens (or maybe I'm just to dumb to figure it out).

I'm also running a small mail server inside of Kubernetes cluster, which is responsible for receiving mails and submitting automatic responses (sort of auto-responder, but with AI).

It all runs pretty well, but I recently looked into logs and saw that:

As you see, someone's trying to brute-force the mail-server, but the built-in protection mechanisms are disabled, since LB's doesn't pass client's information to the pod. The Kubernetes-native way to allow it is to set service.spec.externalTrafficPolicy, but DO seems to ignore it.

Here is my current configuration:

So, luckily you have support for the PROXY protocol, so I decided to give it a spin. First thing I noticed, that traefik ingress controller just broke, but I found a solution here: https://github.com/digitalocean/digitalocean-cloud-controller-manager/issues/144#issuecomment-474655595

The question here is, which CIDR should I use with the DO LB? Or am I good to go with the insecure rule?

Other thing I noticed is that my mail server stopped receiving mails (and spam smtp logins as well, 🤘).

I then browsed the docs of docker-mailserver (which is what I use) and found the answer: https://github.com/tomav/docker-mailserver/wiki/Using-in-Kubernetes#proxy-port-to-service-via-proxy-protocol

So, after applying the sample configuration and reloading the pod, I still didn't see anything happen.

Maybe I'm getting something wrong and probably I should just spend some more time on configuring postfix, but, even if the solution would have worked, I wouldn't be able to talk to SMTP/IMAP from my application without PROXY and I would then need to connect via outside domain, which my application, that is running inside of cluster can not resolve!!! ☹️

Or should I just try to do some magic with ingress, so the traffic will flow like this?!?

Client -> LoadBalancer (with PROXY) -> Node -> Traefik Ingress controller -> Mail Server

Currently it's like that:

Client -> LoadBalancer -> Node -> Mail Server

Do you think the traefik will be able to forward the client information to the Mail Server? Because, then it looks like a possible solution, maybe...

Do you also plan support for automatic node-scaling, based on metrics-server? This menu just makes me laugh:

Aren't your offering a managed service? So, why should I bother with managing the amount of required nodes for myself? Isn't DO smart enough to do if for me? 😄

Do you also have any interest in adding server-less platform capabilities into DO offering with toolkits like Knative? The Google Cloud Run thing just blows my mind. Let me know, if you need any help there.

Thank you!!!

[timoreimann]

Hey there @mishushakov 👋

Let me try to answer your questions as good as I can.

First of all, when I'm trying to resolve the domain, which points to LB, from inside of a cluster, it gets a request timeout:

[...]

I'm guessing that DNS resolution works fine but request routing does not. If so, then this is most likely due to an existing Kubernetes restriction as initially discovered in https://github.com/digitalocean/DOKS/issues/8. See this comment of mine for what the reason is (in short: from inside a cluster, you cannot access an external LB IP address that's already registered as a LoadBalancer-typed Service) along with a pointer to the corresponding upstream issue. A workaround is described as well, which may or may not work for. Other than that, the real fix is to submit a PR upstream, which is something we at DO intend to do.

The Kubernetes-native way to allow it is to set service.spec.externalTrafficPolicy, but DO seems to ignore it

Could you elaborate how a local external traffic policy (I'm assuming that is what you mean) can help you here? As far as the externalTrafficPolicy: Local feature itself is concerned, this is working and supported in DOKS.

The question here is, which CIDR should I use with the DO LB [to white-list traffic]? Or am I good to go with the insecure rule?

Someone else asked the same question on the issue you referenced. I left a response further down, the tl;dr: is: you shouldn't rely on any exposed LB IP addresses to be stable. What you can do instead is create an additional firewall on top of the one that DOKS creates for you (more restrictive rules override less restrictive ones) and block all traffic except the one towards the LB. (DO firewalls support referencing LB IDs for this purpose.)

I think this approach is better than white-listing individual IPs since the DO firewall doesn't even forward unwanted requests through to Traefik where it'd need to be filtered within the application. I think the one use case where this wouldn't work is when you wanted to white-list several IP addresses. (See also #144.) If that's something you'd prefer over blocking at the firewall level, I'd be curious to know so that we can track need for it appropriately.

Other thing I noticed is that my mail server stopped receiving mails.

I then browsed the docs of docker-mailserver (which is what I use) and found the answer: https://github.com/tomav/docker-mailserver/wiki/Using-in-Kubernetes#proxy-port-to-service-via-proxy-protocol

So, after applying the sample configuration and reloading the pod, I still didn't see anything happen.

This is hard to tell without knowing more about your specific setup. If docker-mailserver is supposed to work with PROXY protocol, then doing so with DO LBs should work as well.

Or should I just try to do some magic with ingress, so the traffic will flow like this?!?

Client -> LoadBalancer (with PROXY) -> Node -> Traefik Ingress controller -> Mail Server

Currently it's like that:

Client -> LoadBalancer -> Node -> Mail Server

Do you think the traefik will be able to forward the client information to the Mail Server? Because, then it looks like a possible solution, maybe...

I understand you intend to speak SMTP/IMAP to your mail server? Ingress controllers do not really make sense for that as they tend to serve HTTP/S primarily. I know that some controllers support TCP too (including Traefik 2.0), but then that shouldn't be different from from using the DO LB directly in TCP mode.

Based on the limited information I have, I think that you should be able to get this to work when using DO LBs set to send TCP traffic together with PROXY protocol. If you need additional help, I suggest to reach out to any of the support channels provided by DO.

Do you also plan support for automatic node-scaling, based on metrics-server?

[...]

We do have plans to support auto-scaling. There's a feature request for that on our new DOKS Github presence. Feel free to 👍the issue to express interest. Also, if you happen to have feedback around that feature we'd love to hear about it on that issue. Thanks!

Do you also have any interest in adding server-less platform capabilities into DO offering with toolkits like Knative?

Adding serverless-like capabilities is something we've thought about, though no concrete plans exist at this point. Feel free to file a feature request on that mentioned DOKS repository.

Hope this helps!

[mishushakov]

Hey hey, i'm back with some updates.

the real fix is to submit a PR upstream, which is something we at DO intend to do

Great, let's do it! That will solve 99% of my problems

Could you elaborate how a local external traffic policy (I'm assuming that is what you mean) can help you here?

Never mind, i just googled and found some similar issues and people (mostly on GKE and EKS) reported, that setting this in their LoadBalancer helped...

create an additional firewall

Yep, that's what i did. This way, there is actually no way the traffic can flow inside of the cluster, without LB

Ingress controllers do not really make sense for that as they tend to serve HTTP/S primarily.

The initial thinking was, that i want to accomplish both goals without compromise: my application, that is inside of cluster has to access the mail-server without PROXY + the mail server should still get the IPs to function properly.

That's why i thought, that i can just make Ingress, that will route the traffic to my ingress controller, which will "resolve" PROXY protocol and forward the request to/from the mail-server.

Unfortunately, as you noticed, i discovered, that the Ingress resources aren't designed for tcp traffic, but there's an open issue in Kubernetes repo (https://github.com/kubernetes/kubernetes/issues/23291), so maybe i could leave my case there, so it might bring it forward.

I think that you should be able to get this to work when using DO LBs set to send TCP traffic together with PROXY protocol

I learned it the hard way over the weekend. What i did is i have set up traefik with the flags as described in the https://github.com/digitalocean/digitalocean-cloud-controller-manager/issues/144#issuecomment-474655595

But it didn't worked in my case. After enabling the proxy protocol in my LB config, my node just went down. I didn't actually noticed it, since i was too busy with playing with Traefik arguments to see whether it works or not.

After trying too hard, i noticed, that the issue might be in communication between the node and the LoadBalancer, because when i curled the LB IP it would say 503 Service Unavailable, which means, that either: a) my node is really down b) LoadBalancer is failing to make health checks.

I then came to the conclusion, that the LoadBalancer may also use PROXY protocol to talk to the health check endpoint, so i have added the PROXY arguments to it as well and voilà!

- --entryPoints=Name:https Address::443 TLS Compress:true
- --entryPoints=Name:http Address::80 Redirect.EntryPoint:https Compress:true
- --entryPoints=Name:hc Address::8000 Compress:true
+ --entryPoints=Name:https Address::443 TLS Compress:true ProxyProtocol.Default:true ProxyProtocol.Insecure:true
+ --entryPoints=Name:http Address::80 Redirect.EntryPoint:https Compress:true  ProxyProtocol.Default:true ProxyProtocol.Insecure:true
+ --entryPoints=Name:hc Address::8000 Compress:true ProxyProtocol.Default:true ProxyProtocol.Insecure:true

My Workloads were accessible again and my Mail-Server was now able to retrieve the IPs (and pretend to block them, which doesn't make any sense in a containerised application)

I have since adjusted the configuration of my mail-server and the IPs were then unable to do authentication requests without TLS (smtpd_tls_auth_only = yes in postfix's main.cf).

Now, i'm a little worried actually. Since all of the SPAM requests flow through the LoadBalancer to my Node, am i the one who is billed for the traffic? Ok, it might sound like a joke, if you look at how much traffic goes through my droplets:

But, actually, when more and more spammers start to harass my LB, at some point it will get really nasty and i'd have to raise the costs for my services, just so i could purchase the transfer? Could you provide some more details about dealing with such cases (or point me at someone, who can)?

I'm really interested how you deal with it. Do the LoadBalancers make any SPAM checks? Does they have firewalls? I have found those IPs in some ban-lists and it's clear, that they should not make requests to my Nodes.

Another issue i noticed, is that when i edit the forwarding rules in the Dashboard and after i press "Save" it doesn't actually update them. I waited for 10 minutes for the settings to apply, but then nothing happened. I then have removed the port completely, added it again and it just worked right away!

Do you also plan to support more container runtimes, besides docker? I'm fine with it, but there may be cases, where you might not actually require it. I don't think it is as hard to implement.

Thank you so much @timoreimann for your very in-depth response. That really helped. Have a nice day! Greetings to the DO team out there.

[timoreimann]

Do the LoadBalancers make any SPAM checks? Does they have firewalls? I have found those IPs in some ban-lists and it's clear, that they should not make requests to my Nodes.

I don't think the LB does any spam checks: it's a TCP and HTTP LB, so it wouldn't know about the SMTP protocol, and even less about the semantics of spam. In my opinion, that would be the job of something running on top of the LB (or rather through it).

It is currently not possible to protect LBs by firewalls. It's a feature we receive occasionally though, so let me add another data point internally that may influence the product roadmap in this regard.

Another issue i noticed, is that when i edit the forwarding rules in the Dashboard and after i press "Save" it doesn't actually update them. I waited for 10 minutes for the settings to apply, but then nothing happened. I then have removed the port completely, added it again and it just worked right away!

By dashboard, I suppose you referred to the DigitalOcean cloud console? In general, you should not make changes to LBs if they are managed by Kubernetes (that is, if they were created through a Service of type LoadBalancer). Otherwise, the next reconciliation run inside cloud-controller-manager would revert any manual customizations you have made to the state that is deferred from the Service specification. Can I ask why you intended to modify forwarding rules directly?

Do you also plan to support more container runtimes, besides docker? I'm fine with it, but there may be cases, where you might not actually require it. I don't think it is as hard to implement.

It's something we have considered but not received many requests for. If you are interested in that feature, could you please file a feature request in github.com/digitalocean/DOKS and outline why it'd be helpful for you to have?

Thanks for your feedback, I really appreciate it!

[mishushakov]

Hello again,

By dashboard, I suppose you referred to the DigitalOcean cloud console? In general, you should not make changes to LBs if they are managed by Kubernetes

Then why i can?

Otherwise, the next reconciliation run inside cloud-controller-manager would revert any manual customizations you have made to the state that is deferred from the Service specification

How do i know it? I don't get any warnings about that...

Can I ask why you intended to modify forwarding rules directly?

Because i switched to NodePort to minimise the failure footprint of the Load Balancers

[timoreimann]

Hi @mishushakov

By dashboard, I suppose you referred to the DigitalOcean cloud console? In general, you should not make changes to LBs if they are managed by Kubernetes

Then why i can?

We have not restricted how users can interact with resources that are managed by Kubernetes. But you're raising a good point (that we have also considered), we possibly should.

Otherwise, the next reconciliation run inside cloud-controller-manager would revert any manual customizations you have made to the state that is deferred from the Service specification

How do i know it? I don't get any warnings about that...

There is a section about this part on our known DOKS issues section. Admittedly, that's not immediately discoverable and definitely not a technical safeguard. I'll add another warning to CCM to help raise awareness.

Apologies if my comment came across as "you should know this"; rather, I meant to explain how the system works and how this kind of problem can be avoided in a future. It's a common one and we're working towards making the system more robust. (For instance, CCM v0.1.16 contained #245 which allows LB renames without breaking things, and we intend to update our DOKS images to that version of CCM soon.) Until things are entirely foolproof, the best we can do is raise awareness.

[timoreimann]

I'll be closing this one as I think all questions have been answered, and I see you have also filed digitalocean/DOKS/#11 as a follow-up item. Feel free to reach out to our support for less feature-oriented questions.

Thanks again for your input and feedback!