VPN IP address isn't showing up

kswagler-rh commented 1 year ago

Fleet version: Fleet 4.44.0

💥 Actual behavior

If a host connects to the internet via a VPN, Fleet doesn't show the IP address for this host.

🧑‍💻 Steps to reproduce

Enroll a host to Fleet that connects to the internet via a VPN
Observe the empty "Private IP" and "Public IP" field on the host's Host details page.

🕯️ More info (optional)

Today, the Fleet filters out private IP addresses in the public IP field. We added this filtering to address the following issue: https://github.com/fleetdm/fleet/issues/8924

The VPN address is one of these private IP's that gets filtered out.

When Fleet filters out a public IP this is what the Host details page looks like this:

The "Learn more" links to this page in the docs: https://fleetdm.com/docs/deploy/public-ip

🛠️ To fix

[ ] Backend: Remove the filtering for the public IP field so that the IP address the host uses to connect to Fleet is always shown in the Public IP field in the UI/API.
- In the VPN IP address case, this will reveal the VPN IP address.
[x] Frontend: For the Public IP address field on the Host details page and Hosts page, always show a tooltip w/ the dotted line under the "Public IP address" header: The IP address the host uses to connect to Fleet.
- On the Host details and Host page, remove the old tooltip on hover over the value:
[x] Remove the purple callout box from this doc page: https://fleetdm.com/docs/deploy/public-ip

lucasmrod commented 1 year ago

Hi @kswagler-rh!

Just for reference, #8924 is the issue where we introduced the behavior to not allow "Private IPs" in the "Public IP address".

lukeheath commented 1 year ago

I'm assigning this to @zayhanlon for triage in the CX group. This may require some thought, as we explicitly disallowed private IPs in the public IP field in #8924 as a bug fix.

zhumo commented 1 year ago

Another customer feedback:

I think having an IP address with no extra distinction placed upon it and having Fleet do its best to record that. I don't care about public vs. public. I.e. go back to thinking of it in terms of primary_ip

At our organization, it is quite common for there to be two proxies between nodes and Fleet. This does make it quite hard to rely on HTTP headers for valid info. This is a popular and decent summary of those issues: https://www.brainonfire.net/blog/2022/03/04/understanding-using-xff/

zayhanlon commented 1 year ago

@xpkoala Can you tackle reproducing this so we can prioritize? Thanks!

xpkoala commented 1 year ago

@zayhanlon This does appear to still be the case.

zayhanlon commented 1 year ago

Thanks Reed!

@sharon-fdm for prioritization on Monday

sharon-fdm commented 1 year ago

@xp could you please provide reproduction steps?

RachelElysia commented 1 year ago

@sharon-fdm

@xpkoala I think @sharon-fdm meant to nudge you for specific reproduction steps instead of @xp.

@xpkoala tagged frontend for this bug, however, the value being returned from the API for IP addresses is a backend issue and there is no logic modifying these values on the frontend. Therefore, I am removing the ~frontend tag and adding ~backend.

From reading through the comments, it looks like the CX managers should have a conversation where/how a VPN IP address will present itself in the host data and then assign this ticket to the backend. It'll also be super helpful for QA to outline reproduction steps for dev/testing purposes.

lukeheath commented 1 year ago

This bug has aged out. Moving back to drafting.

noahtalerman commented 7 months ago

Hey @dherder this looks like a bug in Fleet. When you get the chance, can you please update the issue description to use the bug report template: https://github.com/fleetdm/fleet/issues/new?assignees=&labels=bug%2C%3Areproduce%2C%3Aincoming&projects=&template=bug-report.md&title=

After we update the description, can you please assign @sharon-fdm, adding the :release, incoming, and #g-endpoint ops so that it ends up on the endpoint ops board.

@kswagler-rh thanks for calling this to our attention again! Sorry you're still running into this.

dherder commented 7 months ago

@kswagler-rh im testing in Fleet 4.43.0 and cannot reproduce. My VPN IP is being reported in Fleet as the Public IP.

lukeheath commented 7 months ago

@kswagler-rh I'm assigning this to you to make sure it doesn't slip through the cracks.

@dherder is unable to reproduce on the latest. Would you please see if you can still reproduce on 4.44.0? If so, please follow Noah's steps (or @dherder) to get this into the Endpoint Ops board so we can fix. This is now our oldest bug in the system, so we want to triage quickly.

Thanks!

kswagler-rh commented 7 months ago

Thanks folks. We are a few versions behind, I'll try and get them updated today. Thanks!

kswagler-rh commented 6 months ago

Hi all,

We are now running on 4.44.0 for a few days and I do not see changes, we are still not seeing the Public IP field populate with private IPs. I also noticed that the Public IP docs indicate that the field would not show private IP space IPs.

dherder commented 6 months ago

@kswagler-rh here's the steps that I'm taking to try and repo:

Tested on a macOS system running 14.2.1 (23C71)

Signup for a free VPN service like protonVPN (which is just wireguard I think)
Confirm that my Public IP reported in Fleet is my WAN (ISP) address
Get an active VPN connection
Refetch host vitals
Confirm Public IP as viewed in Fleet changes from the ISP provided WAN address to the VPN routed address.

Not sure if this could be dependent on the VPN provider? Were you testing on a macOS, Linux, or Windows host?

kswagler-rh commented 6 months ago

ah I see the confusion. We are talking about the corporate VPN which hands out 10.0.0.0/8 IPs on the same "segment" as the Fleet server. If you connect to a VPN or on the same network as the Fleet server you would see something like this still

@zayhanlon This does appear to still be the case.

lukeheath commented 6 months ago

@kswagler-rh Would you please re-post the image from inside GitHub? If it's posted inside ZenHub it doesn't copy over. Thanks!

lukeheath commented 6 months ago

@xpkoala Would you please try reproducing using the information @kswagler-rh provides? Thanks!

noahtalerman commented 6 months ago

Hey @xpkoala if the bug is unclear, please feel free to schedule some time w/ myself and @kswagler-rh.

We can probably get to the bottom of this in 20 mins together on a call.

lukeheath commented 6 months ago

@sharon-fdm Will you please make sure this bug is actioned? It's currently our oldest open bug, so we'd like to prioritize getting it resolved. I'm adding this to the release board so it gets more eyes.

sharon-fdm commented 6 months ago

@lukeheath , it's on our board now. will try to swap some backend task with it this sprint.

lucasmrod commented 6 months ago

We added this filtering a while ago to fix https://github.com/fleetdm/fleet/issues/9857: https://github.com/fleetdm/fleet/blob/2c383a060fab66105361f7ed88bc9969a02ec5e1/server/service/osquery_utils/queries.go#L376-L389

We may need to define what the "Public IP" field in Fleet is. Is it the IP of the device when it connects to Fleet? Or is it the IP of the device when surfing the internet?

noahtalerman commented 6 months ago

UPDATE: We landed on a solution and updated the "To fix" section in the issue. For more info on the call see this Google doc here (internal).

@sharon-fdm FYI I pulled this bug onto the drafting board and off release board.

Lucas, Rachel, and I are meeting this afternoon to discuss the solution.

noahtalerman commented 6 months ago

Hey @kswagler-rh heads up, I updated the issue description and moved your original issue description here:

Goal

To have the ability to select if private IPs are shown in the public_ip field in the host details.

Context

It will be helpful to be able to quickly and in one place determine the IP that is reaching the Fleet server. An example is when devices are connected to a VPN, and so have a NAT'd Private IP, but are reaching the Fleet server from a private VPN IP.

=========== OP above =========

Mo: I chatted with OP and learned that the the VPN-provided 10.0.0.0/8 IP address is not showing up at all.

Expected Behavior

As a Fleet user who is connecting to the internet via VPN, I would expect my device to show the VPN IP address (special address) in the host details as the public address

Actual Behavior

It does not appear.

noahtalerman commented 6 months ago

From the "To fix" section:

Frontend: For the Public IP field on the Host details page and Hosts page, always show a tooltip to that explains that this is the IP the host uses to connect to Fleet. Points the user to the docs page: https://fleetdm.com/docs/deploy/public-ip

TODO Rachael: Wireframes for the new tooltip.

@rachaelshaw I passed this bug that needs some UI design to you.

When you get the chance, can you please take a pass at the tooltip placement/UI? During design review, we can riff on the language.

noahtalerman commented 6 months ago

Hey @lucasmrod, @RachelElysia, and @rachaelshaw I updated the "To fix" section in the issue description w/ the expected frontend and backend behavior.

I also think we can remove the Public IPs doc page entirely to cut down on doc content. This is included in the "To fix" section.

Moving this over to the release board.

noahtalerman commented 6 months ago

Backend: Remove the filtering for the public IP field so that the IP address the host uses to connect to Fleet is always shown in the Public IP field in the UI/API. In the VPN IP address case, this will reveal the VPN IP address.

Frontend: For the Public IP address field on the Host details page and Hosts page, always show a tooltip w/ the dotted line under the "Public IP address" header: The IP address the host uses to connect to Fleet.

Hey @terjekv heads up, we're updating the behavior of the Public IP address field.

What do you think?

When you filed this issue, it seemed like we weren't being clear as to how Fleet determines the public IP. Goal of this fix is to make this clear: The IP address the host uses to connect to Fleet.

terjekv commented 6 months ago

Looks great! Thanks!

lucasmrod commented 6 months ago

I also think we can remove the Public IPs doc page entirely to cut down on doc content. This is included in the "To fix" section.

I believe it's still useful for administrators to know the HTTP headers Fleet uses to determine the hosts' IP, right? That said, as part of this change we should remove the WARNING message in that page.

rfairburn commented 6 months ago

I believe it's still useful for administrators to know the HTTP headers Fleet uses to determine the hosts' IP, right? That said, as part of this change we should remove the WARNING message in that page.

This is correct. While AWS ALB will have one of these headers set, if the administrator is using something else to do handle ingress/load balancing such as nginx, the proxy pass or equivalent will need to set one of this headers in the configuration.

While I could guess that these are what is used (they are the common headers), I would feel better seeing it in the documentation.

noahtalerman commented 6 months ago

I believe it's still useful for administrators to know the HTTP headers Fleet uses to determine the hosts' IP, right? That said, as part of this change we should remove the WARNING message in that page.

@lucasmrod makes sense to me. I updated the issue description to reflect this. Please feel free to update it if I missed anything.

lucasmrod commented 6 months ago

https://github.com/fleetdm/fleet/pull/17352 (commenting to connect)

RachelElysia commented 6 months ago

@lucasmrod I'll have a frontend PR up today

fleet-release commented 5 months ago

I couldn't think of a haiku this time. (See fleetdm.com logs for more information.)

fleetdm / fleet