Lulu corrupting connection and disconnecting persistent connections

[labrnth]

Good morning,

macOS version: 13.3 Lulu version: 2.4.2

I recently re-installed my Mac recently as I was getting frequent disconnects from SSH sessions and IRC. I thought it was a side effect from upgrading macOS over the years without a fresh install. I did a clean build of macOS and selectively restored data to clean out years of junk.

Even after this clean re-install of macOS I noticed that at some point it started having the same network symptoms as before where I would frequently get disconnected from my SSH sessions and IRC. I had recently re-installed Lulu, so I decided to disable it as I knew it installs a content filter which directly hooks into the network stack. Sure enough, after disabling it I was not having the same disconnects as before.

So something Lulu is doing to the network stack is causing network corruption under certain conditions. A couple of observations I've made:

1. I have an endpoint security solution installed (SentinelOne) which also installs a content filter. This along with Lulu are the only two applications that install a content filter.
2. The disconnects appeared to happen when there was more CPU activity (load) as I would see it much more frequently when I heard my fans spin up

For now, I've had to remove Lulu from my Mac until this is resolved, as it causes too much chaos for me during the work day. Please let me know if there is something I can provide (logs?) to help diagnose this issue. I can see in the list of GitHub issues that there are others experiencing network issues with applications, disconnects or flat out losing Internet access. So there is obviously some underlying issue that needs to be addressed.

When Lulu was working it was an amazing security solution and I look forward to re-introducing it some day soon.

Cheers, -Chris

[objective-see]

Aloha Chris, Looking into this more, but in the past there have been issues with Apple's core Network Extension framework (which LuLu is built a top), when multiple filters are installed (chained).

Do you see the same issue if the other filter (SentinelOne) is uninstalled or disabled?

Mahalo!

[labrnth]

I'll test this with the SentinelOne content filter disabled and get back to you. Whats interesting is that I've tried the same setup with Little Snitch (and SentinelOne), which also uses a Network Extension, and I'm not seeing this same behaviour. So the network service chaining is happening here as well, but without the side effect (or direct effect).

FYI, I actually spent some time growing up on Maui and lived in Haiku for a few years. Sure miss it!

Mahalo!

[labrnth]

@objective-see Aloha Patrick, wondering if any progress has been made with network service chaining and Lulu. This appears to be what the issue is/was. I haven't tested recently, but I can only imagine that others are having the same issue as many enterprise/business users will have an EDR/AV solution on their macOS device as well.

On a separate note, my heart goes out to you and the people of Maui (a place I once called home) regarding the devastation of the fires. I will be donating to your Go Fund Me to help with relief. God bless!

[ashj403]

LuLu keeps preventing me from joining any network when I start my Mac. This has been a constant issue. Finally after deleting LuLu my mac is back to normal, please fix this

[xnyhps]

I think I’ve been having the same issue. I have Microsoft Defender installed as a second content filter and I see SSH connections getting corrupted quite often. Even connections to a local running VM end up corrupted sometimes!

I looked at it in Wireshark and experimented with netcat to another machine. Wireshark was showing everything properly (no weird retransmissions or checksum errors), but some incoming packets that were visible in Wireshark were not delivered to nc. At first I thought it was some networking issue, but then I realized that a software problem on my Mac was way more likely. I disabled Lulu and it stopped happening (so far).

[skull-squadron]

Having periodic problems with ssh and now recently (v2.6.3), Lulu constantly reinstalls itself and forgets its rules. Then, today, it randomly disconnected all of my network connections. Uninstalling. Sad.

[skull-squadron]

I have Microsoft Defender installed

MD. Be afraid, especially if it's MDE plan 2 where MSFT rolls out untested definitions to millions/billions of devices that end up deleting all of a user's start menu shortcuts (I shit you not).

[wastez]

I also have such problems. Sometimes some services are not working, sometimes only special internet sites, sometimes other stuff. If i disable lulu all working as it should. I'm also using multiple filters (proxifier) but also disabling the other filters make no difference.

Thats really a problem if i need to disable lulu multiple times during the day. I love this software but this is a problem which let me think about an alternative.

[PaulDance]

This does indeed look to be very similar to #523. I can easily reproduce the issue as well:

• Have LuLu not installed and reboot the machine.
• Install and start LuLu.
• Stop it using its integrated Quit.
• Open a TCP connection slowly transmitting data.
• Start LuLu again.
• Observe the connection get closed on a broken pipe.

I haven't tested if things also keep on having intermittent failures later on, but at the very least, I can also observe the basic behavior.

I'm currently writing a toy network extension and the same issue applies for it, so this shouldn't be something purely specific to LuLu. More specifically, this seems to happen for any two extensions started in order:

• LuLu + mine;
• mine + LuLu;
• LuLu stopped + LuLu started again;
• mine stopped + mine started again;
• Little Snitch uninstalled + Little Snitch installed and started again.

Indeed, LuLu bases itself on macOS' NetworkExtension API that must be used from a "system extension", which is a special mode of execution offering higher privileges but requiring an activation sequence that performs a separate installation and execution. The current status of such extensions can be observed using the systemextensionctl list command. When stopping LuLu, it requests the system to uninstall its system extension, which stops the secondary process, but does not remove its installation files. One can observe that they are still under /Library/SystemExtensions/<UUID>/ and that the previously-mentioned command lists the extension as terminated waiting to uninstall on reboot. This remainder seems to still count as something to consider from the point of view of the system when starting a new extension, thus also triggering the issue even though nothing from the uninstalled extension runs anymore.

After rebooting however, all extensions in pending uninstallation are indeed uninstalled and starting a new one will work just fine. This should therefore serve as a workaround: instead of restarting only LuLu, restart the whole system, then start LuLu if not configured to start at login.

@objective-see do you know if this can be avoided in any way? I went through most of the API's documentation and cannot see any parameter that could affect this. Even by degrading my extension to only be NEFilterManagerGradeInspector because I don't particularly need to be Firewall, the issue remains. It really seems to be unavoidable, but still.

[wastez]

Something new here? I also removed my profile and create a new ruleset but this changes nothing.

If @PaulDance is right then nothing will solve this problem until a other way is implemented. How is little snitch working on the new systems? Also via extension?

I love Lulu but as it is in the moment it is nearly unusable if the problem appears several times a day.

[PaulDance]

How is little snitch working on the new systems? Also via extension?

Yes, it's similar in this aspect and therefore suffers from the same problem during activation. Later issues you mention might be different, however, since I was only able to reproduce the first case.

I love Lulu but as it is in the moment it is nearly unusable if the problem appears several times a day.

Little Snitch can be used for a bit of time free of charge: maybe you can try that? How frequently does the issue occur with LuLu? Also, have you tried rebooting your machine?

[wastez]

@PaulDance This happens about 2 to 10 times during a working day but it happens every day. It’s enough to disable and enable the extension in settings and after it is working again for a while. I tried almost all, reboot was the first. Total uninstall. Uninstall and install using a new profile.

Already though about to move to little snitch.

[skull-squadron]

LS works great. 0 issues (when used without LuLu). I've been using it since 2011 (~13 years) on multiple generations and architectures of machines.