Open milopersic opened 2 years ago
luis4a0
commented
2 years ago Hi @tazle, everything you describe suggests me that the firewall is messing up with the connection. Do you have the firewall enabled? In that case, adding some rules to allow Multipass can help, as suggested earlier. In some cases, disabling and then enabling internet sharing and firewall helps. Even rebooting the computer may help. Let's try first with these easy solutions to bring the networking up again.
tazle
commented
2 years ago I do have the firewall enabled, and I am not able to disable it. I'll try rebooting, and adding the rules listed earlier here, but I don't immediately see how they would explain not having bootpd running.
tazle
commented
2 years ago It seems that either restart or momentarily enabling and then disabling incoming connections for /Applications/Multipass.app helped. Not sure, which.
Now bootpd gets started when bridge100 is created by qemu, and bootpd.plist includes the subnet set up by com.apple.NetworkSharing.
luis4a0
commented
2 years ago Good to know. Not sure either, we cannot unfortunately fully understand what MacOS is doing. But glad you solved your issue.
EchoAbstract
commented
2 years ago I've started to run into this too and suspect it's a bug in Apple's firewall due to the randomness of when it triggers, that it doesn't happen for some folks at all, and, that running the firewall in debug mode fixes the issue. (That and I've had other networking issues crop up at the same time with unrelated protocols)
To that point, if you're hitting this you can work around it by turning the firewall off in the GUI and enabling it in debug mode like: /usr/libexec/ApplicationFirewall/socketfilterfw -d. (see https://github.com/canonical/multipass/issues/2387#issuecomment-1056765443 for where I got this).
I can confirm that as soon as I disabled the firewall and enabled it in debugging mode that the problem cleared up.
~Regardless of the cause it'd probably be useful for someone on the project to submit a bug with apple~, I was going to suggest we file an issue with apple, but you can't do that without a developer account anymore (looks like you need an app to do it)
cedws
commented
2 years ago It works! Thanks!
mstoffel-sag
commented
2 years ago Got the issue again. After a reboot and no os update. Renders Multipass completely useless on Mac. Can anybody point me to how I at least can recover some files for the VMs?
maufranchini
commented
2 years ago Got the issue again. After a reboot and no os update. Renders Multipass completely useless on Mac. Can anybody point me to how I at least can recover some files for the VMs?
@mstoffel-sag Have you tried to disable the networking ? What they mentioned previously about deleting duplicates in /var/db/dhcpd_leases might work, at least to get your VM up and running and you can recover some files.
bjonnh-work
commented
2 years ago Got the issue again. After a reboot and no os update. Renders Multipass completely useless on Mac. Can anybody point me to how I at least can recover some files for the VMs?
For me putting the firewall in debugging mode (see above https://github.com/canonical/multipass/issues/2387#issuecomment-1143180291) "solves" it, and I can use the VMs again.
mstoffel-sag
commented
2 years ago Thanks, but can't do that. Company prohibits switching off the firewall.
mstoffel-sag
commented
2 years ago Got the issue again. After a reboot and no os update. Renders Multipass completely useless on Mac. Can anybody point me to how I at least can recover some files for the VMs?
@mstoffel-sag Have you tried to disable the networking ? What they mentioned previously about deleting duplicates in /var/db/dhcpd_leases might work, at least to get your VM up and running and you can recover some files.
I deleted all leases without any remedy
EchoAbstract
commented
2 years ago I think the only reliable fix is to run the firewall in debug mode. NOTE: This isn't switching the firewall off, it's still active and protecting the system. Maybe @mstoffel-sag you could check with your IT department to see if there's a way for them to help you run it in debug mode.
bjonnh-work
commented
2 years ago @mstoffel-sag Putting the firewall in debug mode doesn't disable it.
mstoffel-sag
commented
2 years ago Sure but:
sudo /usr/libexec/ApplicationFirewall/socketfilterfw -d Firewall settings cannot be modified from command line on managed Mac computers It's no reliable solution right now. I cannot always ask for support when it stops working. I will wait until apple fixes this.
mstoffel-sag
commented
2 years ago Its weird. Without any change, but another reboot, its working again. Just random
danielmoncada
commented
2 years ago Experienced the same issues as many above..
*Note that reinstalling never worked for me after the initial week of it working before.
System:
anpep
commented
2 years ago I'm having the exact same issue (macOS 13.0, M1 Pro) and I fixed it yesterday using the socketfilterfw -d trick (starting the firewall daemon in debug mode). However, today it no longer works and socketfilterfw crashes with exit code 137 every time I run it. I have found no success trying to get a crash dump but any help is appreciated.
It's totally a bug in Apple's firewall.
mstoffel-sag
commented
2 years ago New Mac OS update 12.5.1. New luck. Multipass working again. Will have a look how long.....
flyingnobita
commented
2 years ago New Mac OS update 12.5.1. New luck. Multipass working again. Will have a look how long.....
This seems to work for me!
mbay-ODW
commented
2 years ago Same issue here. It sometimes works for a while and stops working at some point. However I'm also not able to deactivate the firewall, similar to @mstoffel-sag. Any way to at least configure that the firewall should be started via -d?
@townsend2010: Any feedback on fixing that? Is there already some root cause found?
townsend2010
commented
2 years ago @mbay-ODW,
Any feedback on fixing that? Is there already some root cause found?
This a total black box to us and we don't have a relationship with Apple for this. We have about as much insight as anyone else affected by this unfortunately. Apple consistently breaks this, seemingly fixes it in an update, then breaks it again in a subsequent update. The virtual networking API we use, firewall, etc. are all under Apple's control. We are as frustrated as every user who posts this problem :frowning_face:
mbay-ODW
commented
2 years ago Thanks for your extremely fast feedback, really appreciate that.
In the meanwhile if somebody has an idea where the starting parameters of the socketfilterfw daemon can be configured, I would really love to hear that.
rjoelnorgren
commented
2 years ago Is there any mention of this issue in the official documentation?
I burned considerable time evaluating multipass as a tool when this was still a nascent issue. Given 9 months has passed without progress, I think prospective users should be given a disclaimer that the product is not compatible with Mac OS firewall.
aaronbuckner
commented
2 years ago I agree with @rjoelnorgren, it's very confusing from a user's perspective.
I think there is also a similar issue if you have a VPN enabled on your Mac. These 2 issues have caused me tons of wasted time and as much as I want to use this product, I still cannot.
townsend2010
commented
2 years ago Hey @rjoelnorgren, @aaronbuckner, et. al.,
We use an API provided by Apple that is supposed to make networking of virtual machines easier. Apple keeps breaking this for reasons we cannot possibly know. Other than the API, it's all a black box to us including the underlying vmnet code, macOS firewall, the bootp process, etc. We have no insight into what Apple is doing. It seems Apple silicon based Mac's are much more affected by this, again for reasons we cannot understand.
That said, we are trying to figure out if there are other solutions we could integrate into Multipass to alleviate this madness.
rjoelnorgren
commented
2 years ago @townsend2010, I appreciate the context. I understand that this is a tricky issue for the team to work through. Taking everything into consideration, I think a disclaimer is still well justified. Whether or not it's the fault of the product, the fact remains it has critical gaps in its functionality and reliability when working with Apple's new hardware.
I would understand the hesitancy here if the issue was well understood with a clear plan in place to address it. But from your own description, the issue is opaque to the team with no path forward presently. It has remained an issue for 9 months.
Users should be able to make an informed - decision without having to spend days embedding the tool into their workflows to only find it does not work for them.
aaronbuckner
commented
2 years ago Agreed. People who get burned in this way using Multipass are likely to never consider it again.
I don’t want to use Parallels, because I don’t want all that complexity and cruft, but I don’t have these types of problems, either. So it is possible to make it work.
(It almost sounds like Canonical hasn’t provided adequate testing hardware to the devs, which would be shameful if true.)
On Sep 9, 2022, at 22:03, rjoelnorgren @.***> wrote:
@townsend2010 https://github.com/townsend2010, I appreciate the context. I understand that this is a tricky issue for the team to work through. Taking everything into consideration, I think a disclaimer is still well justified. Whether or not it's the fault of the product, the fact remains it has critical gaps in its functionality and reliability when working with Apple's new hardware.
I would understand the hesitancy here if the issue was well understood with a clear plan in place to address it. But from your own description, the issue is opaque to the team with no path forward presently. It has remained an issue for 9 months.
Users should be able to make an informed - decision without having to spend days embedding the tool into their workflows to only find it does not work for them.
— Reply to this email directly, view it on GitHub https://github.com/canonical/multipass/issues/2387#issuecomment-1242089110, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANINY5ANSPD5XGYHWYKEPZDV5NGUVANCNFSM5LIEH6UA. You are receiving this because you were mentioned.
townsend2010
commented
2 years ago I appreciate your feedback. We will think about how best to message this, but not present it as if everyone will experience this issue. This doesn't affect everyone. To answer @aaronbuckner's theory that we don't have adequate hardware, we do. We have M1 Macbook's along with an M1 Mini used for CI/CD and integration testing and all machines have never shown this issue. And we've heard from other folks using Multipass on M1 Mac's without issue.
I will say I had my Intel Macbook exhibit this behavior one time and I tried and tried to get any info I could as to why this occurs. As soon as I put the macOS firewall into debug mode, it never exhibited the problem again which is kind of the workaround for now.
Regarding Parallels not exhibiting this problem, that is because it doesn't use vmnet, it uses some user level networking they developed, but is closed sourced of course, so we can't use it.
We figured Apple would just fix this and it no longer be an issue since it seems it breaks with one macOS release, is fixed in the next macOS release, then breaks again on the release after that. I will add that we are very frustrated by this too as we have spent considerable resources on something this supposed to be stable.
That said, we are evaluating our options here and will try to come to a resolution that works for everyone. Thank you for your patience.
aaronbuckner
commented
2 years ago Thanks Chris! I am not trying to push a conspiracy theory. Just a frustrating issue that might be at least mitigated with documentation.
On Sep 9, 2022, at 23:50, Chris Townsend @.***> wrote:
I appreciate your feedback. We will think about how best to message this, but not present it as if everyone will experience this issue. This doesn't affect everyone. To answer @aaronbuckner https://github.com/aaronbuckner's theory that we don't have adequate hardware, we do. We have M1 Macbook's along with an M1 Mini used for CI and integration testing and all machines have never shown this issue. And we've heard from other folks using Multipass on M1 Mac's without issue.
I will say I had my Intel Macbook exhibit this behavior one time and I tried and tried to get any info I could as to why this occurs. As soon as I put the macOS firewall into debug mode, it never exhibited the problem again which is kind of the workaround for now.
Regarding Parallels not exhibiting this problem, that is because it doesn't use vmnet, it uses some user level networking they developed, but is closed sourced of course, so we can't use it.
We figured Apple would just fix this and it no longer be an issue since it seems it breaks with one macOS release, is fixed in the next macOS release, then breaks again on the release after that. We are evaluating our options here and will try to come to a resolution that works for everyone. Thank you for your patience.
— Reply to this email directly, view it on GitHub https://github.com/canonical/multipass/issues/2387#issuecomment-1242218111, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANINY5H2XTT7LFYLWX6H3T3V5NTENANCNFSM5LIEH6UA. You are receiving this because you were mentioned.
townsend2010
commented
2 years ago Just soliciting feedback about the documenting part of this, but would something under https://multipass.run/docs/troubleshoot-networking#heading--troubleshoot-networking-on-macos about this be sufficient or is that too buried to be discoverable by users?
aaronbuckner
commented
2 years ago I wouldn’t have found it.
Here’s how I would describe my issue. For me, it’s so major it should be right up front, but maybe I’m just a minority user with this problem.
I install Multipass and it works, while my VPN and Firewall are operating. If I restart my Multipass instance, it does not ever finish booting. If I disable both my firewall and VPN and then restart the instance, it does start and works fine, even if I then turn on my VPN or firewall.
However, it’ll eventually stop responding again and hang, but I’m not sure the timeframe. I assume the Mac went to sleep, woke, and the Multipass instance’s networking then began failing.
That’s where I left it the last time I tried. It seemed consistent with my previous trials. I used to use it all the time on my older Mac but then got an M1, and it never worked again.
I did upgrade the OS at the time I got the M1.
On Sep 10, 2022, at 01:13, Chris Townsend @.***> wrote:
Just soliciting feedback about the documenting part of this, but would something under https://multipass.run/docs/troubleshoot-networking#heading--troubleshoot-networking-on-macos https://multipass.run/docs/troubleshoot-networking#heading--troubleshoot-networking-on-macos about this be sufficient or is that too buried to be discoverable by users?
— Reply to this email directly, view it on GitHub https://github.com/canonical/multipass/issues/2387#issuecomment-1242313979, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANINY5ES5UZVAHJ7H72WBIDV5N44FANCNFSM5LIEH6UA. You are receiving this because you were mentioned.
townsend2010
commented
2 years ago We rely on an ssh connection between the host and the instance(s). Although I know it may appear the instance does not boot, I think it boots fine, but access between the host and the instance cannot be established, so the client can only assume that something is wrong with the instance which can be a number of things.
It really boils down to the macOS firewall and VPN software not accounting correctly (or at all) the bridge100 interface that macOS creates when using vmnet shared networking.
Other than our documentation, I'm not sure where else these types of issues should go. I don't feel the multipass.run landing page is the correct place to call out platform specific issues that may or may not affect users. Maybe making a special case on the multipass.run/docs main page? Release notes on Github? The README.md that is displayed on github.com/canonical/multipass? Just throwing out ideas...
rjoelnorgren
commented
2 years ago @townsend2010, I think that's a good start but it may be worth calling out elsewhere. Just because the user is likely to first encounter the problem outside of the context of networking even if the cause is networking related. I.e. the issue is going to present itself as a failure to start a vm altogether if I recall correctly.
Appreciate you following up on this. I do hope you guys can figure it out longer term. I really did appreciate multipass when it was working :).
townsend2010
commented
2 years ago Ok, thanks for the feedback. That is a good point about it not looking network related and manifesting itself as a VM boot failure.
simonmichael
commented
2 years ago Perhaps multipass could be smarter about providing useful feedback to the user when this or something else is going wrong, perhaps pointing to this issue ? That would at least make it more discoverable and make testing easier.
milopersic
commented
2 years ago For what it's worth, I've been using Multipass with the firewall turned off for the last nine months every day and it's flawless software otherwise. I run my dev environments on it exclusively. I'm in a unique situation though with my machine, and have access to all settings. Hopefully Apple will open up a bit and work with Canonical on the firewall issues. I have to think the overlap of mac fans and ubuntu fans is pretty big.
lukens
commented
2 years ago Perhaps multipass could be smarter about providing useful feedback to the user when this or something else is going wrong, perhaps pointing to this issue ? That would at least make it more discoverable and make testing easier.
Yes. Having discovered this is probably the issue I am facing, after a long search, I feel it would be good if it could point in this direction.
Maybe after timing out it could check if it’s running on a Mac with the firewall enable, and post a message saying this might be the issue (although, the timeout is so long that generally I’d ctrl-ced long before then).
I can’t confirm it is the firewall, as I’m unfortunately also in the position of using a Mac with a company profile that prevents me disabling the firewall, but from reading this thread it seems the likely cause.
space55
commented
2 years ago Hi, all
After coming here from #2740, I can confirm that disabling my firewall immediately solved the issue. I tried adding an exception for multipassd, but that didn't help.
I've seen the Thunderbolt Bridge option in Apple's system preferences for networking before. Would using that help, over using a new bridge network? Just spitballing, not sure, and I don't have the context to be more helpful.
tabascoterrier
commented
2 years ago This issue has been troubling me on and off for a while. I upgraded to macOS 12.6 the other day, and multipass seems to be working normally so far with the firewall on🤞
I'm curious to know if others have seen the same improvement, or if I'm just having a lucky streak.
rduan
commented
2 years ago I am using macos 12.6 and can not launch multipass instance with time out.
absoftware
commented
2 years ago Update to macOS 12.6 helped once until I executed multipass restart. Now I get still
$ multipass start
start failed: The following errors occurred:
myvm: timed out waiting for responseMacBook Pro Apple Silicon (arm64), managed macOS so I cannot disable firewall. I cannot work as no alternatives or workarounds for now. Trick with file /var/db/dhcpd_leases helped only once as well (or it was done together with update of macOS). Please help.
townsend2010
commented
2 years ago @absoftware,
Since it's a managed Mac, does the company who owns it have a support relationship with Apple? If so, it may be worth a try to get Apple tech support to help figure out why the firewall blocks this.
We are still working on this on our end...
aaronbuckner
commented
2 years ago Does Canonical not have a developer/support relationship with Apple?
On Sep 20, 2022, at 08:19, Chris Townsend @.***> wrote:
@absoftware https://github.com/absoftware,
Since it's a managed Mac, does the company who owns it have a support relationship with Apple? If so, it may be worth a try to get Apple tech support to help figure out why the firewall blocks this.
We are still working on this on our end...
— Reply to this email directly, view it on GitHub https://github.com/canonical/multipass/issues/2387#issuecomment-1252273786, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANINY5EYAUVCJNZYC6TRJJTV7GTW5ANCNFSM5LIEH6UA. You are receiving this because you were mentioned.
townsend2010
commented
2 years ago Hi @aaronbuckner,
We do, but the problem is we have yet to reproduce the problem, so just as a thought, if users/organizations can also put some pressure on Apple to figure this out, that would be better. I'm not suggesting anyone has to do this, but it could help.
EDIT: I should add that I did reproduce the problem one time on an Intel Mac, but as mentioned before, putting the firewall into debug mode "fixed" it and even running it in non-debug mode after that has not reproduced the issue.
milopersic
commented
2 years ago Thanks @townsend2010 Chris, just a question - What can users bring before Apple (assuming we have no developer/support relationship with them) that Canonical cannot, or what does Canonical need to better leverage the Apple relationship? It sounds like you're saying that a lack of in-house error logs or the inability to reproduce the issue on demand with various Macs doesn't give you enough "meat" to take to Apple as a demonstration of the problem. For all those experiencing this issue, is there something else we can do to help Canonical in this regard? Just spitballing here.
EinarJohansson
commented
2 years ago I am also using macos 12.6 and can not launch multipass instance without time out.
tabascoterrier
commented
2 years ago This issue has been troubling me on and off for a while. I upgraded to macOS 12.6 the other day, and multipass seems to be working normally so far with the firewall on🤞
It was working well for a good few days, but now the issue is back for me too.
beauwilliams
commented
2 years ago I have the same problem. The only option is to disable the firewall. I've had this problem for 6 months now.
thebadpete
commented
2 years ago I have the same issue. Unfortunately I am doing multipass on a managed MacBook Pro M1 Pro and I cannot modify firewall settings. I guess I am out of luck with multipass unless I run it on a different Mac ...
handlebauer
commented
2 years ago I'm encountering the same symptoms but disabling firewall doesn't do the trick, either
Update: If you are using an unmanaged Mac and run across this issue, the following commands may fix this issue:
Original post
Multipass crashes unexpectedly, and all instances (including primary) will not restart. I am using multipass 1.8.1+mac, multipassd 1.8.1+mac. (multipass --version), on macOS Monterey 12.1, M1 Max, 32 GB RAM. I installed multipass for macOS directly from Canonical's website.
This has happened twice. The first time I uninstalled multipass, reinstalled, and re-built my instances thinking it was a fluke. Multipass performed fine for roughly a week, then crashed again and I lost some work. I am not able to determine what the cause is, or how to reproduce it. System restarts do not help. I'm hoping someone can spot the issue in the logs.
Logs here: