search

objective-see / LuLu

LuLu is the free macOS firewall
GNU General Public License v3.0
9.42k stars 436 forks source link

Lulu corrupting connection and disconnecting persistent connections #530

Open labrnth opened 1 year ago

labrnth commented 1 year ago

Good morning,

macOS version: 13.3 Lulu version: 2.4.2

I recently re-installed my Mac recently as I was getting frequent disconnects from SSH sessions and IRC. I thought it was a side effect from upgrading macOS over the years without a fresh install. I did a clean build of macOS and selectively restored data to clean out years of junk.

Even after this clean re-install of macOS I noticed that at some point it started having the same network symptoms as before where I would frequently get disconnected from my SSH sessions and IRC. I had recently re-installed Lulu, so I decided to disable it as I knew it installs a content filter which directly hooks into the network stack. Sure enough, after disabling it I was not having the same disconnects as before.

So something Lulu is doing to the network stack is causing network corruption under certain conditions. A couple of observations I've made:

  1. I have an endpoint security solution installed (SentinelOne) which also installs a content filter. This along with Lulu are the only two applications that install a content filter.
  2. The disconnects appeared to happen when there was more CPU activity (load) as I would see it much more frequently when I heard my fans spin up

For now, I've had to remove Lulu from my Mac until this is resolved, as it causes too much chaos for me during the work day. Please let me know if there is something I can provide (logs?) to help diagnose this issue. I can see in the list of GitHub issues that there are others experiencing network issues with applications, disconnects or flat out losing Internet access. So there is obviously some underlying issue that needs to be addressed.

When Lulu was working it was an amazing security solution and I look forward to re-introducing it some day soon.

Cheers, -Chris

objective-see commented 1 year ago

Aloha Chris, Looking into this more, but in the past there have been issues with Apple's core Network Extension framework (which LuLu is built a top), when multiple filters are installed (chained).

Do you see the same issue if the other filter (SentinelOne) is uninstalled or disabled?

Mahalo!

labrnth commented 1 year ago

I'll test this with the SentinelOne content filter disabled and get back to you. Whats interesting is that I've tried the same setup with Little Snitch (and SentinelOne), which also uses a Network Extension, and I'm not seeing this same behaviour. So the network service chaining is happening here as well, but without the side effect (or direct effect).

FYI, I actually spent some time growing up on Maui and lived in Haiku for a few years. Sure miss it!

Mahalo!

labrnth commented 10 months ago

@objective-see Aloha Patrick, wondering if any progress has been made with network service chaining and Lulu. This appears to be what the issue is/was. I haven't tested recently, but I can only imagine that others are having the same issue as many enterprise/business users will have an EDR/AV solution on their macOS device as well.

On a separate note, my heart goes out to you and the people of Maui (a place I once called home) regarding the devastation of the fires. I will be donating to your Go Fund Me to help with relief. God bless!

ashj403 commented 10 months ago

LuLu keeps preventing me from joining any network when I start my Mac. This has been a constant issue. Finally after deleting LuLu my mac is back to normal, please fix this

xnyhps commented 4 months ago

I think I’ve been having the same issue. I have Microsoft Defender installed as a second content filter and I see SSH connections getting corrupted quite often. Even connections to a local running VM end up corrupted sometimes!

I looked at it in Wireshark and experimented with netcat to another machine. Wireshark was showing everything properly (no weird retransmissions or checksum errors), but some incoming packets that were visible in Wireshark were not delivered to nc. At first I thought it was some networking issue, but then I realized that a software problem on my Mac was way more likely. I disabled Lulu and it stopped happening (so far).

skull-squadron commented 3 months ago

Having periodic problems with ssh and now recently (v2.6.3), Lulu constantly reinstalls itself and forgets its rules. Then, today, it randomly disconnected all of my network connections. Uninstalling. Sad.

skull-squadron commented 3 months ago

I have Microsoft Defender installed

MD. Be afraid, especially if it's MDE plan 2 where MSFT rolls out untested definitions to millions/billions of devices that end up deleting all of a user's start menu shortcuts (I shit you not).

wastez commented 2 weeks ago

I also have such problems. Sometimes some services are not working, sometimes only special internet sites, sometimes other stuff. If i disable lulu all working as it should. I'm also using multiple filters (proxifier) but also disabling the other filters make no difference.

Thats really a problem if i need to disable lulu multiple times during the day. I love this software but this is a problem which let me think about an alternative.

PaulDance commented 2 weeks ago

This does indeed look to be very similar to #523. I can easily reproduce the issue as well:

I haven't tested if things also keep on having intermittent failures later on, but at the very least, I can also observe the basic behavior.

I'm currently writing a toy network extension and the same issue applies for it, so this shouldn't be something purely specific to LuLu. More specifically, this seems to happen for any two extensions started in order:

Indeed, LuLu bases itself on macOS' NetworkExtension API that must be used from a "system extension", which is a special mode of execution offering higher privileges but requiring an activation sequence that performs a separate installation and execution. The current status of such extensions can be observed using the systemextensionctl list command. When stopping LuLu, it requests the system to uninstall its system extension, which stops the secondary process, but does not remove its installation files. One can observe that they are still under /Library/SystemExtensions/<UUID>/ and that the previously-mentioned command lists the extension as terminated waiting to uninstall on reboot. This remainder seems to still count as something to consider from the point of view of the system when starting a new extension, thus also triggering the issue even though nothing from the uninstalled extension runs anymore.

After rebooting however, all extensions in pending uninstallation are indeed uninstalled and starting a new one will work just fine. This should therefore serve as a workaround: instead of restarting only LuLu, restart the whole system, then start LuLu if not configured to start at login.

@objective-see do you know if this can be avoided in any way? I went through most of the API's documentation and cannot see any parameter that could affect this. Even by degrading my extension to only be NEFilterManagerGradeInspector because I don't particularly need to be Firewall, the issue remains. It really seems to be unavoidable, but still.