search

mullvad / mullvadvpn-app

The Mullvad VPN client app for desktop and mobile
https://mullvad.net/
GNU General Public License v3.0
5.08k stars 338 forks source link

[Mac] Bookmark and iCloud sync not working with VPN on #2401

Closed mikesk8 closed 1 year ago

mikesk8 commented 3 years ago

Hello,

I have noticed that a sync between Safari bookmarks on Mac (10.14.6) and iPhone (14.3) stopped to work some time ago. I did a deep dive and it turned out that switching the VPN (both latest Mullvad apps) off on both devices makes the bookmarks to sync again.

Have you heard about this before?

Thanks, m

faern commented 3 years ago

Thank you for the report. Yes we recently started getting reports about this and have started investigating a little bit. Let's keep this issue as a tracker for when we have it resolved.

faern commented 3 years ago

These Apple services probably use some protocol/ip range that we are blocking for security reasons. If we find out what IPs/ports it's using we can consider if unblocking them would be safe or not.

Does this syncing stop working if the app is running on only one of the devices, or does our VPN have to run on both for the syncing to break? If only running it on one of them stops the syncing, which one? Only the mac or only the iPhone?

steve10883 commented 3 years ago

I spent considerable time diagnosing this way back at the beginning of January, I had hoped this information would have been passed onto you.

  1. It's the Mullvad macOS app. The iOS app does not block syncing in any way.
  2. It's definitely the Mullvad macOS app. The Wireguard macOS app does not block syncing in any way.
faern commented 3 years ago

Thanks for the extra info! Yeah I suspected so. Because the macOS app does pretty strict firewalling. I suppose it's using some multicast protocol that we block. We have an internal task to look at iPhone syncing.

steve10883 commented 3 years ago

Any news on this please? I'd like to go back to using the Mullvad app, but can't considering how important it is to sync this data.

steve10883 commented 3 years ago

@faern Looking at the changelog for version 2021.3 it doesn't appear that this has been addressed. Has somebody been assigned to this yet? Have they looked at it?

Thanks.

faern commented 3 years ago

We have sadly not had the time to investigate this further yet.

ghost commented 3 years ago

Interesting observation:

I had the exact same problem with Cloudflare's VPN (WARP). But with the new update, they have introduced a new 'local proxy mode'.

Using the local proxy fixes the problem and bookmarks now sync perfectly with iOS devices. Connecting using the standard mode still has the issue.

possiblerobot commented 3 years ago

Any word on this issue? Mullvad is still blocking bookmark syncing for Safari on MacOS (not sure about iOS).

Thanks!

steve10883 commented 3 years ago

@faern How is this going? I have read the latest beta notes and it looks like it still hasn't been fixed. https://www.reddit.com/r/mullvadvpn/comments/q7eaak/mullvad_20215beta1/

If that's the case it's a poor showing. This issue has now been ongoing for 11 months and your paying customers deserve better.

cd-a commented 3 years ago

This is the issue that prevents me to switch to mullvad, as it's essential to my workflow.

steve10883 commented 2 years ago

Another release and still not fixed. Unbelievable.

https://github.com/mullvad/mullvadvpn-app/releases/tag/2021.6

paulrudy commented 2 years ago

I just discovered this thread when searching for a solution to the same problem. I guess I'll have to switch VPN providers. I was hoping there would be a straightforward solution.

steve10883 commented 2 years ago

I just discovered this thread when searching for a solution to the same problem. I guess I'll have to switch VPN providers. I was hoping there would be a straightforward solution.

The official Wireguard app works. But don’t expect Mullvad to ever fix their own app.

paulrudy commented 2 years ago

Good to know Wireguard works, thanks

mikesk8 commented 2 years ago

It is a pity Mac users are not so important :( I have been waiting for this feature for a very long time!

possiblerobot commented 2 years ago

There seems to be an issue with the syncing of tab groups as well. It could be related to the bookmark syncing issue. As soon as Mullvad is disconnected, bookmarks and tab groups behave normally again.

faern commented 2 years ago

If anyone can traffic dump this bookmark synchronization with tcpdump or Wireshark or similar and help us figure out what kind of traffic it is that's needed to allow this, we can probably fix it way faster.

It's probably using some kind of local *casting-somethingsomething address/port combination that is currently blocked in the firewall. If we figure out what and then determine that allowing it is not against our security policies, then we can just unblock it in the firewall.

paulrudy commented 2 years ago

I've done a Wireshark capture just now. Is there something I should look or filter for?

pinkisemils commented 2 years ago

I did one too, and, when excluding 17.0.0.0/8 (apple's subnet), I saw no traffic. Probably an error on my end. Is there a deterministic way of making the synchronization take place? Does any iCloud syncing work when the app is connected?

possiblerobot commented 2 years ago

Other things do sync, like notes, iCloud Drive, photos, etc. Even AirDrop works. If it helps, I believe the process that's doing the sync job for Safari is SafariBookmarksSyncAgent.

As for a deterministic test, making any change to the bookmarks should trigger a sync. You can add/remove bookmarks, drag a bookmark to change its order in the list, or you can move a bookmark in or out of a folder. These actions will produce a nearly instant change on other devices when bookmark syncing is working.

paulrudy commented 2 years ago

@faern Please advise what to filter for in a Wireshark dump, I'm happy to share what I find but unfamiliar with Wireshark.

faern commented 2 years ago

@paulrudy Anything going to some local multicast address. Might even be IPv6 for all I know. I'm not sure what to filter for. But if you try to exit all other programs so that the computer is not so chatty and repeat the experiment a few times. Maybe you'll see some packets going out or coming in around the time when the sync happens that are similar every time the sync happens?

paulrudy commented 2 years ago

@faern I ran three tests, one after the next, where I started Wireshark and immediately either created or deleted a Safari bookmark. In each test there's a lot of chat between my local ip and 17.248.188.xxx (the last 3 digits were different in each test). Looking those ip's up show that they belong to Apple. Is that useful or do I need to dig deeper?

faern commented 2 years ago

Yes, the entire 17.x.x.x net is Apple's. But that has nothing to do with the LAN, that's on the internet. Our VPN app does not prevent communication with that IP range. So if their bookmark sync is not performed locally, but rather via their internet servers, then I don't see how we would be blocking it.

paulrudy commented 2 years ago

I don't know if you saw this earlier comment, but I've also confirmed that connecting to Mullvad tunnels via the WireGuard official app does not interfere with iCloud Safari bookmark sync. So it's something about the Mullvad app. Disabling "black ads" and "block trackers" does not seem to make a difference.

In case it's useful, the IVPN app also breaks iCloud Safari bookmark sync

possiblerobot commented 2 years ago

This might help. When trying to sync bookmarks over Mullvad, the console shows this message:

502:com.apple.SafariBookmarksSyncAgent.XPC.BookmarkSyncNetworkConnectivity:2A144B:[
    {name: NetworkQualityPolicy, policyWeight: 8.400, response: {Decision: Must Not Proceed, Score: 0.00, Rationale: [{[wiredQuality]: Required:20.00, Observed:0.00},{[wifiQuality]: Required:50.00, Observed:0.00},{[networkPathAvailability]: Required:1.00, Observed:1.00},]}}
 ], FinalDecision: Must Not Proceed}

When the VPN is disconnected, you get this:

502:com.apple.SafariBookmarksSyncAgent.XPC.BookmarkSyncNetworkConnectivity:FB9C66:[
    {name: DeviceActivityPolicy, policyWeight: 2.000, response: {Decision: Can Proceed, Score: 0.65}}
 ] sumScores:38.210000, denominator:38.910000, FinalDecision: Can Proceed FinalScore: 0.982010}

Could this issue have something to do with this NetworkQualityPolicy / DeviceActivityPolicy stuff?

pinkisemils commented 2 years ago

Seems like this will require the app to use Apple's VPN API instead of just using unixy APIs to create a tunnel device to circumvent these issues. Or maybe there's a better way to inform the system that the routes we've added are legitimate and do work.

paulrudy commented 2 years ago

@pinkisemils Does Wireguard's app use Apple's VPN API? Because, as mentioned, these problems don't occur when connecting through the Wireguard app (to mullvad servers)

dionvl commented 2 years ago

Can confirm that the Mullvad app breaks iCloud sync in several places (Bookmarks, iMessages). Official wireguard app from the Mac App Store works fine.

jgogstad commented 2 years ago

Just confirmed that it's the macOS side that's broken:

  1. iPhone with Mullvad 2021.4 running
  2. macOS with 2021.6

Bookmarks syncs once I disable mullvad on the macOS side.

For what it's worth bookmarks should be synced with https according to https://support.apple.com/en-gb/HT202944. I've done a couple of captures while seeing a bookmark being synced to my phone, and I only see traffic on TCP 443.

paulrudy commented 2 years ago

Is there anything else we can provide that will help solve this bug? Just reminding the devs also that the Wireguard app does not share this bug, it's specific to the macOS version of the Mullvad app.

paulrudy commented 2 years ago

I haven't checked in a few weeks, but as of today:

mietzen commented 2 years ago

iCloud (File) sync is broken for me as well, if I add a file on my iCloud drive on the iPhone it won't appear until I turned off the Mullvad App on my Mac.

I'm on macOS 12.2.1 with Mullvad 2022.1-beta.

cd-a commented 2 years ago

@n-stone Works fine for me with Desktop 2021.6 and iOS 2022.1 Looks like a very bad regression then? Anyone else have the issue?

mietzen commented 2 years ago

I tested it again, it seems to be something sleep related on the macOS site. If the mac is awake und "fresh" connected via the mullvad App everything is fine. But if the mac is connected via mullvad goes in some (deeper??? 5 Minutes is not enough) sleep state and wakes up connects again via mullvad App the iCloud Drive sync won't work. Tricky to test / reproduce.

steve10883 commented 2 years ago

I haven't checked in a few weeks, but as of today:

  • Safari tab groups sync on macOS with VPN connected via Mullvad app or Wireguard app.
  • Safari bookmarks do not sync on macOS with Mullvad VPN connected via Mullvad app or Wireguard app. Previously this only occurred when using the Mullvad app, but not the Wireguard app. Now it occurs with both apps.
  • Safari bookmarks do sync when not connected to any VPN.

I am not seeing this behaviour at all. Using the Wireguard app 1.0.15, Safari 15.3 and macOS 12.2.1. Bookmarks and Reading List sync without any issue.

mietzen commented 2 years ago

I tested it again, it seems to be something sleep related on the macOS site. If the mac is awake und "fresh" connected via the mullvad App everything is fine. But if the mac is connected via mullvad goes in some (deeper??? 5 Minutes is not enough) sleep state and wakes up connects again via mullvad App the iCloud Drive sync won't work. Tricky to test / reproduce.

This is what it looks like: image

After I woke up the mac I straight up created a new folder with a File in it. iCloud Drive tries to upload it and is stuck at around 90%. It will stay like this until I disconnect Mullvad. The new File will not appear in the iCloud drive on my iPhone until the upload is finished. I waited around 10 Minutes before I disconnected Mullvad, Mullvad was fully connected and am.i.mullvad was working and showing the correct server.

mcmurry-1 commented 2 years ago

I'm jumping on here to say that I also have this issue.

On macOS, an active VPN connection with the Mullvad app breaks basic system features, like bookmark syncing. I also often see the iCloud Drive issue mentioned above by @n-stone.

The bookmarks start syncing a few seconds after disconnecting Mullvad, by initiating a connection to gateway.icloud.com. With an active Mullvad connection, this domain just stays silent, the bookmark changes aren't seen.

I think it's very underwhelming that broken system functionality persists for more than 1 year after this was first mentioned here.

macOS 12.2.1 Mullvad 2022.1

steve10883 commented 2 years ago

I tested it again, it seems to be something sleep related on the macOS site. If the mac is awake und "fresh" connected via the mullvad App everything is fine. But if the mac is connected via mullvad goes in some (deeper??? 5 Minutes is not enough) sleep state and wakes up connects again via mullvad App the iCloud Drive sync won't work. Tricky to test / reproduce.

This is what it looks like: image

After I woke up the mac I straight up created a new folder with a File in it. iCloud Drive tries to upload it and is stuck at around 90%. It will stay like this until I disconnect Mullvad. The new File will not appear in the iCloud drive on my iPhone until the upload is finished. I waited around 10 Minutes before I disconnected Mullvad, Mullvad was fully connected and am.i.mullvad was working and showing the correct server.

Some sleep related issues have been fixed in the latest release https://mullvad.net/en/blog/2022/3/1/macos-now-reconnects-instantly-after-being-sleep-mode/

Unfortunately still no fix for bookmark and reading list syncing. 14 months and counting.

pronebird commented 2 years ago

iCloud Drive seems to work fine for me when on VPN. macOS 11.6.4 / Mullvad 2021.6

paulrudy commented 2 years ago

FYI, with Mullvad 2022.1, Safari tab groups sync and Safari bookmark sync continue to be broken.

mietzen commented 2 years ago

@steve10883 I was on the beta before and now updated to 2022.1, error persists. @pronebird The upload error only happens when the mac was put to sleep with mullvad on and directly connects to mullvad after sleep. If I turn of Mullvad put it sleep and connect mullvad again after sleeping everything is fine.

faern commented 2 years ago

Thanks for all the extra information provided here. The iCloud sync issue is probably easier for us to test. So weird that it will sync to 90% and then give up? I would assume it would not sync at all or finish.

I have created a new internal issue for this in particular. Maybe looking at iCloud drive will shine some light on what's up here.

notDavid commented 2 years ago

Fyi, i just noticed kill bird in Terminal (this restarts the back-end process behind iCloud) solves the issue for me, so it's a quick workaround to initiate an iCloud sync without rebooting or whatever. Perhaps this works for other too...

Mullvad 2022.1 macOS 12.2.1

mcmurry-1 commented 2 years ago

Fyi, i just noticed kill bird in Terminal (this restarts the back-end process behind iCloud) solves the issue for me, so it's a quick workaround to initiate an iCloud sync without rebooting or whatever. Perhaps this works for other too...

Mullvad 2022.1 macOS 12.2.1

I'm assuming you are talking about iCloud Drive syncing here, because the bookmark sync issue is not affected by this process at all and still unsolved.

notDavid commented 2 years ago

because the bookmark sync issue is not affected by this process at all and still unsolved

@mcmurry-1 Ah, yes indeed.

Out of curiosity, you could perhaps try: kickstart -k gui/$(id -u)/com.apple.SafariBookmarksSyncAgent I think this should restart the bookmark sync service.

paulrudy commented 2 years ago

Out of curiosity, you could perhaps try: kickstart -k gui/$(id -u)/com.apple.SafariBookmarksSyncAgent I think this should restart the bookmark sync service.

@notDavid I tried to test this, but got zsh: command not found: kickstart

notDavid commented 2 years ago

@paulrudy Sorry, correction:

launchctl kickstart -k gui/$(id -u)/com.apple.SafariBookmarksSyncAgent

paulrudy commented 2 years ago

@notDavid thanks. Unfortunately, that command didn't help. After entering it, I continued to get log items like this, and bookmarks don't sync:

501:com.apple.SafariBookmarksSyncAgent.XPC.CloudTabGroupZoneSubscriptionRegistration:D1E2AB:[
    {name: NetworkQualityPolicy, policyWeight: 8.400, response: {Decision: Must Not Proceed, Score: 0.00, Rationale: [{[wiredQuality]: Required:20.00, Observed:0.00},{[wifiQuality]: Required:50.00, Observed:0.00},{[networkPathAvailability]: Required:1.00, Observed:1.00},]}}
 ], FinalDecision: Must Not Proceed}

Once mullvad was disconnected, bookmarks synced fine.