search

jitsi / ice4j

A Java implementation of the ICE protocol
Apache License 2.0
437 stars 232 forks source link

Using ice4j behind NLB in a VPC at AWS (Kubernetes) #204

Open citosid opened 4 years ago

citosid commented 4 years ago

First of all, thank you so much for this awesome work! it has been really nice working with this tool.

We have a problem though. Our infrastructure runs behind an Application Load Balancer inside a VPC in a Kubernetes cluster. The VPC has an external IP.

The requests are something like this:

pod:10.0.0.2
ec2Instance: 172.16.4.145
eni: 52.204.71.1 # Example IP
alb: 
  - 35.170.123.9
  - 34.203.105.22

User -> alb (35.170.123.9) -> ec2Instance -> pod

The discovery from the pod to find the external IP works, as can be seen from the logs:

INFO: Using org.ice4j.ice.harvest.StunMappingCandidateHarvester, face=/172.16.4.145, mask=/52.204.71.1
Jun 03, 2020 6:52:18 AM org.ice4j.ice.harvest.MappingCandidateHarvesters initialize

However, as mentioned before, the IPs the user is hitting, are the IP's from the ALB. And those are different from the one the pod is using to go to internet.

We cannot "hard-code" the IPs of the ALB in the configuration because those are elastic. AWS will change them as the load in our system increases or decreases.

And the IP of the ENI does not allow access from the outside, is just from within the VPC out.

Is there any way to use the DNS name to do the discovery?

Thank you for your time and hard work!

bgrozev commented 4 years ago

First of all, thank you so much for this awesome work! it has been really nice working with this tool.

Thank you for the kind words!

Is there any way to use the DNS name to do the discovery?

No, but it should be easy to add a new MappingCandidateHarvester similar to StunMappingCandidateHarvester. How often would you need to rediscover?

citosid commented 4 years ago

Thanks for your quick answer @bgrozev !

The TTL for the DNS in the ALB is 60 seconds. This is to quickly react to changes in the traffic.

https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html#request-routing

bgrozev commented 4 years ago

You could have a timer running rediscovery every minute

citosid commented 4 years ago

I'm not really a java programmer. I can see the Stun harvester is done here:

https://github.com/jitsi/ice4j/blob/master/src/main/java/org/ice4j/ice/harvest/StunMappingCandidateHarvester.java#L70

Could you point me in the right direction?

bgrozev commented 4 years ago

All mapping harvesters are initialized here: https://github.com/jitsi/ice4j/blob/master/src/main/java/org/ice4j/ice/harvest/MappingCandidateHarvesters.java#L118

citosid commented 4 years ago

Thanks @bgrozev . Will try to implement it and make a PR. Should I send it to you?

bgrozev commented 4 years ago

Just open it here and we'll take a look

citosid commented 4 years ago

@bgrozev , we're still working on this and researching our options. Question, can I add multiple IP addresses here:

NAT_HARVESTER_PUBLIC_ADDRESS
citosid commented 4 years ago

@bgrozev, I don't think we are going to need this. At least not for the time being. We found a different way to do it.

You can see it here.

We are still figuring how to make this more resilient to changes in the NLB. But maybe using (in our specific case) octo for Jitsi will help us to restart the video bridges without loosing the whole conference.

However, we will still be willing to write our own harvester in the future. But for the time being, this seems to be good enough four our use case.

I will not close the issue, but if you feel is better to, feel free to do it.

Thank you so much for your hard work!

citosid commented 4 years ago

@bgrozev , so after a few months more testing and working to do our release, we are in the staging phase. And it seems we might need to do this in the end.

Has it been any work on your side? We are looking into how to fix this (either by adding this code here or doing it externally)

bgrozev commented 4 years ago

@citosid there's been no work on this from our side.

citosid commented 4 years ago

Ok. Will keep looking into it and will let you know.

Thank you so much!!!

jbg commented 4 years ago

@citosid What about if we made it possible to specify a DNS name for NAT_HARVESTER_PUBLIC_ADDRESS (e.g. k8s-default-jvb-abcdef-abcdef.elb.us-east-1.amazonaws.com), and when starting ICE, the DNS name is resolved? This way you'd always have the current IP(s) for the load balancer.