search

Dunedan / mbp-2016-linux

State of Linux on the MacBook Pro 2016 & 2017
2.1k stars 108 forks source link

Thunderbolt issues #87

Open mfauvain opened 5 years ago

mfauvain commented 5 years ago

Hi, macbook 14,1 was working fine with thunderbolt (display and some usb devices connected to it) up until today. Log shows ACPI Error: AE_NO_MEMORY, SSDT 0xFFFF9655E4ED7000 Table is duplicated (20180105/tbdata-562) and a lot more of the kind ACPI Error: Method parse/execution failed \_PR.CPU0.GCAP, AE_ALREADY_EXISTS (20180105/psparse-550). Macbook boots fine, thunderbolt display works as well however none of the connected devices seem to be recognized anymore.

It happened once a month ago and managed to get rid of it with kernel param pci=hpbussize=10,hpmemsize=2M but to no avail this time.

Anyone with same issue? any help or direction greatly appreciated.

EDIT : after some more tests I am wondering if those ACPI errors have anything to do with this issue. Basically, thunderbolt hot-plug still works, but USB/Ethernet/Firewire ports on the thunderbolt display (or even displays as plugging a second monitor works as well) are unavailable. Seems like a thunderbolt3 security issue. If someone could confirm that this it the case? Is it related to ACPI errors?

EDIT 2 : adding ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1" to /etc/udev/rules.d and reboot seems to solve the issue. Sorry for the confusion. Hope this helps someone with same issue and thx for the great site!

Dunedan commented 5 years ago

Great you figured out what the issue was. :+1:

Your solution suggests disabling any kind of authorization for devices connected via Thunderbolt. That's a major security gap, as it allows any malicious device connected via Thunderbolt to take over your MacBook Pro. A better solution would be a tool like bolt for managing authorization of Thunderbolt-connected devices.

mfauvain commented 5 years ago

Seems the issue is not solved in the end.. Connected devices (to thunderbolt display) aren't recognized again. And neither the udev rule nor the bolt tool nor the kernel param seem to be able to wake/authorize them. Display works, but nothing more (not even sound from display). Very frustrating as it seems to randomly work and stop working. boltctl shows connected and authorized thunderbold display.. Happy to do debugging if somenone has an idea where to look..

@Dunedan : might be worth it renaming the issue to 'thunderbolt issue' or similar. In the end I don't believe it has to do with ACPI error I saw in logs.

quasd commented 5 years ago

I had similar problems with my macbook 14,1. In my case it was solved by resetting nvram / smc / updating mac os to later firmware. Have you tried any of these? ( some of these might reset your boot drive back to mac, so you might have to run bootctl update (if that is what you are using))

mfauvain commented 5 years ago

I did reset the SMC and updated mac OS the other day when I had the issue for the second time, but did not seem to solve it. I later installed bolt, which initialy didn't seem to help either and finally set the udev rule and that worked after reboot. Or maybe it was a combination of all? Can't be sure to be honest. Will try to reset SMC and NVRAM (this one I never did) and will report back later. Thx.

Dunedan commented 5 years ago

@mfauvain: To rule out hardware problems: Does Thunderbolt work with macOS?

mfauvain commented 5 years ago

Yes, it works, booting macOS shows all devices connected to thunderbolt display (as well as the eponym display of course). But booting Linux right after doesn't 'wake up' the devices, even though the display works.

mfauvain commented 5 years ago

Tried SMC and NVRAM reset, no luck. When booting OSX, everything works as it should. In linux, boltctl info [uuid] shows Apple, Inc. Thunderbolt Display ├─ type: peripheral ├─ name: Thunderbolt Display ├─ vendor: Apple, Inc. ├─ uuid: [uuid] ├─ dbus path: /org/freedesktop/bolt/devices/[uuid] ├─ status: authorized │ ├─ authflags: none │ ├─ parent: [parent uuid] │ ├─ syspath: /sys/devices/pci0000:00/0000:00:1c.4/0000:04:00.0/0000:05:00.0/0000:06:00.0/domain0/0-0/0-3 │ ├─ authorized: Mon 03 Dec 2018 21:51:09 UTC │ └─ connected: Mon 03 Dec 2018 21:51:09 UTC └─ stored: no

I looked in dconf-editor but under the specified dbus path above there is no /org/freedesktop/bolt/devices/[uuid] folder.. I tried unplugging the thunderbolt display for a while, nothing better.

mfauvain commented 5 years ago

Still trying to sort this out. With another laptop (macbook air linux 4.15) connected to same thunderbolt display lsusb shows : Bus 003 Device 007: ID 05ac:9227 Apple, Inc. Thunderbolt Display Bus 003 Device 006: ID 05ac:1112 Apple, Inc. FaceTime HD Camera (Display) Bus 003 Device 005: ID 05ac:1107 Apple, Inc. Thunderbolt Display Audio Bus 003 Device 004: ID 05ac:0307 Apple, Inc. Bus 003 Device 003: ID 05ac:0267 Apple, Inc. Bus 003 Device 002: ID 05ac:9127 Apple, Inc. Hub in Thunderbolt Display

14,1 with linux 4.16.13 lsusb doesn't show the line Thunderbolt Display (and obviously neither the other devices) HOWEVER I am writing this using thunderbolt display connected and working. Lost. Also, connected firewire drive lights up but doesn't show with lsblk. All other USB connected devices do not light up (and do not show up)

@Dunedan let me know if this not the right forum as I do have the feeling it is a security issue with thunderbolt and not really a kernel issue. Could be useful to others though to know that linux on this macbook has thunderbolt shaky.

Will keep on investigating.

EDIT : upon thunderbolt plugging, logs are showing 07:52:29 kernel: thunderbolt 0000:06:00.0: 4: uid: 0x100010055ad90 07:52:28 kernel: thunderbolt 0000:06:00.0: 4: reading drom (length: 0x97) 07:52:28 kernel: thunderbolt 0000:06:00.0: unknown1: 0x0 unknown4: 0x0 07:52:28 kernel: thunderbolt 0000:06:00.0: Upstream Port Number: 2 Depth: 1 Route String: 0x4 Enabled: 1, PlugEventsDelay: 254ms 07:52:28 kernel: thunderbolt 0000:06:00.0: Config: 07:52:28 kernel: thunderbolt 0000:06:00.0: Max Port Number: 13 07:52:28 kernel: thunderbolt 0000:06:00.0: Switch: 8086:1513 (Revision: 2, TB Version: 1) 07:52:28 kernel: thunderbolt 0000:06:00.0: current switch config:

but then all ports are disabled kernel: thunderbolt 0000:06:00.0: 4:d: disabled by eeprom

full log here log.txt

I don't know how it can be enabled (or disabled). If anyone has a clue happy to hear.

mfauvain commented 5 years ago

back at it, trying to boot USB drive with linux install and I have same issue. Holding option key on boot shows USB key, start booting and then stop (after a lot of apple properties lines) basically saying the USB is no present anymore... I've read the v interesting article from Lukas Wunner https://lwn.net/Articles/707616/ from nov 2016 and it seems Alpine Ridge 4C 'was' not well supported. I don't know how to apply the patches he mentions neither do I know if those have been integrated in kernel 4.16. To summarize my intents to get thunderbolt port switch from dponly mode with linux 4.16:

Dunedan commented 5 years ago

So the problems are not only limited to Thunderbolt, but affect USB as well? That doesn't sound good. Have you tried different Linux distributions to rule out problems with the one you're using?

mfauvain commented 5 years ago

No, sorry if it was not clear To clarify : 14,1 linux -> usbc/tb3 port -> any usb device or usb hub : works 14,1 linux -> usbc/tb3 port -> tb3 to tb2 adapter -> tb display : only display works. Ethernet, usb, firewire, facetime, audio (from display) don't work (but used to) 14,1 linux -> usbc/tb3 port -> tb3 to tb2 adapter -> tb2 to ethernet adapter : don't work (but used to) 14,1 osx -> everything work

from my readings it seems tb3 port on the 14,1 is stuck in security mode dponly and refuses anything that is not display. note that nothing has changed on the linux side before the issue appeared. I tried to boot usb arch install but same issue. will try to boot usb with different distro and revert.

mfauvain commented 5 years ago

tried booting ubuntu 18.04, same issue. is it possible that macbook firmware has been altered?

Dunedan commented 5 years ago

Ah, so everything directly attached to the Thunderbolt 3 ports works, but as soon as you put a Thunderbolt-3-to-2-adapter in the middle it stops working?

What display are you using? Is it an Apple Thunderbolt Display or a different one?

mfauvain commented 5 years ago

Yes, plugging a usb c (non thunderbolt) dock works. Yes it is an apple tb display. Plugging the tb3 to tb2 adapter to apple thunderbolt display have only the display working, none of the thunderbolt display ports neither the integrated camera/audio are recognized. I would have thought it has to do with linux thunderbolt driver no setting the right security profile and keeping it to dponly (as per https://www.kernel.org/doc/html/v4.14/admin-guide/thunderbolt.html. However I can't explain why booting a different distro or even a usb install (arch or ubuntu) give same issue ie ports on apple thunderbolt display ar not recognized. then maybe it is the tb3 to tb2 adapter? but why is it working when booting osx? I tried using the adapter with a tb2-ethernet adapter (so 14,1 -> tb3 to tb2 adapter -> tb2 to ethernet adapter) and the device appears in dmesg : [ 7545.522802] thunderbolt 0000:06:00.0: current switch config: [ 7545.522814] thunderbolt 0000:06:00.0: Switch: 8086:1549 (Revision: 0, TB Version: 1) [ 7545.522819] thunderbolt 0000:06:00.0: Max Port Number: 2 [ 7545.522823] thunderbolt 0000:06:00.0: Config: [ 7545.522831] thunderbolt 0000:06:00.0: Upstream Port Number: 1 Depth: 1 Route String: 0x3 Enabled: 1, PlugEventsDelay: 254ms [ 7545.522836] thunderbolt 0000:06:00.0: unknown1: 0x0 unknown4: 0x0 [ 7545.541007] thunderbolt 0000:06:00.0: 3: reading drom (length: 0x7b)

and boltctl shows ● Apple Thunderbolt to Gigabit Ethernet Adapter ├─ type: peripheral ├─ name: Thunderbolt to Gigabit Ethernet Adapter ├─ vendor: Apple, Inc. ├─ uuid: a021e600-0200-0100-ffff-ffffffffffff ├─ status: authorized │ ├─ domain: d0010000-0080-7f08-235f-4312aa32a021 │ └─ authflags: none ├─ authorized: Sat 08 Dec 2018 16:00:01 UTC ├─ connected: Sat 08 Dec 2018 16:00:01 UTC └─ stored: Fri 07 Dec 2018 19:14:34 UTC ├─ policy: auto └─ key: no

no error messages, but the ethernet port doesn't show up. mistery.

I dont have a tb3 device to plug directly unfortunately.

Dunedan commented 5 years ago

I'm not sure if that adapter already known to work. There is a bug report in the Kernel Bugzilla about it: https://bugzilla.kernel.org/show_bug.cgi?id=189731 Mind that this bug report is using it with a Dell notebook and not with a MacBook Pro. As discussed in the bug report on Macs the EFI might setup the necessary tunnels for the adapter to work. The Thunderbolt implementation in the Linux kernel is still encountering lots of changes, so what I can imagine is that a newer kernel version handles Thunderbolt differently, while an old one might simply fall back to the EFI-set up tunnel. Did you update your kernel version between the adapter working and it not working anymore?

mfauvain commented 5 years ago

I did not update the kernel since day one. Question that bothers me though is : on day one I used a usb key to install arch on this machine and on that day, thunderbolt display + all connected devices were working (actually the usb key was plugged into thunderbolt display usb port) I even used the external keyboard plugged to that same display to do all the install as internal keyboard was not recognized before installing macbook12-spi-driver. How come today booting the same usb key (no changes at all on that key) has only display working, not the keyboard, not even the usb port : holding option key at startup shows the key, booting starts and then fails right after kernel tries to read the usb key to follow on booting process.. would the tb3 to tb2 adapter be blocking this now??? can anything be 'written' to that adpter's chip?

Dunedan commented 5 years ago

can anything be 'written' to that adpter's chip?

The adapter contains firmware as well, which might get updated. I'm not sure how the process for that looks like though. I'd guess macOS does it at some point.

Did you update macOS at any point?

mfauvain commented 5 years ago

I did update macos after having the issue and posting it here. But I honestly doubt adapter firmware was updated then, and anyway thunderbolt display ports had stopped working before and macos update did not solve the issue.

Dunedan commented 5 years ago

Something must have changed on software-side. You already confirmed that you didn't do a kernel update nor a macOS update between the Thunderbolt connections working and not. Any other changes you did? Not necessarily updating updating software, but installing new packages, changing configuration, etc? E.g. to set a firmware password is known to affect certain things as well.

kayateia commented 5 years ago

I've had a very similar experience with a 14,1 running Arch. When I first plugged it into my TB3 dock, the monitor came on, the mouse and keyboard hooked in through a USB3 hub worked fine. After that it's never worked again. DP monitor pass-through only. I tried the kernel parameters mentioned above and not had any luck. The only thing that changed in there, really, is that I did an Arch upgrade; the install was previously from June-July, and I updated it to the repos as of yesterday. I tried downgrading the kernel back to the package that was used previously and it didn't fix anything.

The symptoms I'm seeing now are basically that the TB3 dock is visible in things like bolt or tbtadm. I can explore the device tree in /sys and find it there, Linux knows what its name is, etc. TB3 security is listed as "none", and the device is authorized by the kernel. It just never goes the next step; none of the devices in the dock are recognized as existing, including a USB controller, audio output, SD card reader, etc... it just stops enumerating at the dock.

Everything works 100% in macOS with no drivers or configuration. I also tried booting into macOS first, then into Linux, in case something gets initialized, but no luck.

If I can gather any useful data for debugging, please let me know and I'll post it.

mfauvain commented 5 years ago

On my side I didn't do any update, arch on the laptop was installed using an archive repo and I don't update. I did install bolt and tbtadm but that was after having the issue. For ref, after arch initial install everything was working nicely during about 2months when I experienced the issue for first time, then tried the kernel param and it worked right away, I then removed the kernel params and I would say 2 weeks later I had the issue again, I then tried the kernelk param again, installed bolt and tbtadm but no luck since then. Of note, when laptop did succeed to recognize devices after second issue, I remember seeing a weird iphone network in network list on login screen (using gnome). I guess it is stupid but could it be that iphone connection would lock tb3 ports? (i did use iphone tethering from time to time before second issue occured).

I reported the issue there as well https://github.com/intel/thunderbolt-software-user-space/issues/65

kayateia commented 5 years ago

So, today I noticed after an Arch update that my Thunderbolt dock was randomly working again. I saw a lot of dmesg errors about allocating PCI space, so I am starting to think that maybe I was onto something when I studied stories of people using eGPUs in BootCamp. I tried making an updated DSDT earlier with a large PCI space for Thunderbolt, as the eGPU people suggested, but I don't think it really fixed anything, and I'm not using it currently. So this is weird, but it's good that I did have a good result today. It means Apple's firmware updates haven't broken it unfixably.

I pulled a lot of logs and info dumps and will probably be looking over them later this week, to see if I can figure out what changed. The Arch update didn't involve a new kernel and didn't even involve a reboot, because I'm stuck on 4.16.13 with Bluetooth. So I am a bit mystified. But I will try to report back if I can find anything in it. I can also post the logs somewhere if people are interested.

kayateia commented 5 years ago

Several days later, this is working very consistently now. Home dock and work dock (TB3 in both cases) work reliably.

On hindsight, I'm pretty sure this happened during the pacman upgrade I did that day, because my external monitor flickered on and off a few times, and it seems that the system sees the display via passthrough vs full TB3 dock as a different display. The only thing of remote relevance that changed is that the linux-firmware package was upgraded. The version I have now is:

linux-firmware-20190118.a8b75ca-1

(was linux-firmware-20181218.0f22c85-1)

You might give a try to updating that and see if it fixes the problem.

mfauvain commented 5 years ago

@kayateia, thank you, are you still on 4.16 kernel? Will try the firmware, however, in the end I switched to a different setup where I don't need the thunderbolt with 14,1 macbook but using old 10,2 (that works perfect with thunderbolt) and synergy between the two. v practical.

Dunedan commented 5 years ago

Looking at the latest changes to upstream linux-firmware I don't see anything which would suggest that it's the new firmware package that fixed it. Mysterious. :thinking:

kayateia commented 5 years ago

@mfauvain Yeah, I'm still on 4.16.13. I haven't been able to get 4.20 to boot, and I couldn't get Bluetooth to work on 4.17 - 4.19.

@Dunedan Totally in agreement here, I looked through there too and it was mostly just RPi updates.

For the sake of science I'm going to post up the update log from that update, maybe there's something I'm not seeing in it because of a lack of familiarity with all the device handling bits in newer Linux.

https://gist.github.com/kayateia/59faf74d488ef6e33fee2da3293636f6

christophgysin commented 5 years ago

@kayateia Did you try to boot kernel 4.20 with the patches at https://github.com/christophgysin/linux/tree/4.20-mbp ?

Dunedan commented 3 years ago

@mfauvain @kayateia : Any updates regarding this issue? Is this now working for you?

mfauvain commented 3 years ago

I haven't been using thunderbolt on the machine since then. USBc dock still has some random issue reported in another thread.

BryanMorfe commented 3 years ago

Hi @Dunedan @mfauvain @kayateia, I have been having a similar issue since I installed Ubuntu a while back. I have a MacBook Pro 13,3 and two of my TB3 ports simply refuse to work fully. Here are the details:

First, here's my setup to add context:

Background

I first installed 18.04 a while back, and I think all 4 ports were working (though I can't be completely sure since I never used all four at the same time). On a normal day, I noticed that my Ubuntu installation simply REFUSED to boot; I'd get the grub command line, and both the Docking Station and NVMe SSD would show up and work with no problem (I use my keyboard and mouse in this command line), but as soon as I set it up for booting, it noticed that all TB3 ports stopped working -- my mouse and keyboard would shut off, and the SSD (which contains the installation) didn't work. Of course, this meant that I couldn't boot. I tried reinstalling 18.04, nothing. I decided to make the jump to 20.04, and nothing again. After playing around trying different things and googling around to see if someone had a similar issue, the issue was unresolved. At that point I had given up and decided to go back to macOS, where everything worked with no problem.
On another regular day, I said, you know what? Let's try it again, and to my surprise IT WORKED! For some reason, it worked... My Docking Station, and my NVMe were working out of the box. At that point, at my own risk of never having a working installation again, I tried playing around with it and trying different things, and that's when I discovered something oddly interesting... On arbitrary boots, my two left TB3 ports worked, and my 2 right TB3 ports wouldn't work... on other boots it was the opposite... I had no idea what was going on, but at least I knew how to get back into the Ubuntu installation. If the ports on the left didn't let me boot, then I'd switch to the ports on the right.
At this point I could use my entire setup, and as long as I didn't need to use the other two non-functioning ports, it'd be okay, so I just forgot about it.

Now

Well, surprise, I left the issue unresolved and now it's coming back to bite me. I recently bought a TB3 PCIe enclosure for an FPGA and I need all 4 ports working for my full system. I have been trying to find a solution for the past 3 days. I've updated my kernel, have been updating macOS, updating... updating, trying to authorize, removing devices, trying different grub boot options... and nothing has worked so far. I can confirm that the FPGA gets enumerated both in macOS and in Ubuntu when I use one of the two working TB3 ports. I recently stumbled upon this post and tried every suggestion here, and nothing seems to work.

What Works and What Doesn't

The TB3 ports seem to be providing power. The TB3 enclosure has x16 PCIe Gen 3 port, so it has two TB3 ports for the necessary bandwidth. When I connect it, boltctl recognizes the device, and it appears to be authorized, however, a kernel log (using dmesg) doesn't show as if any device is connected (as opposed to when I connect it to a working port). I don't get any message from the thunderbolt module nor the pcieport module.

May be worth noting that I, too, see many ACPI errors in my logs, and it may have something to do with sleeping ports. I notice that when my mac is sleeping, I can connect my phone for charging, but if I wanted to do something more, I'd have to wake it up.

At this point I have no idea how or even if there's a fix. I will try to recompile my current kernel 5.4.0-58-generic this weekend and see if I can play around with some setting that may fix the problem. I'll take a look at the thunderbolt module code as well and see if I find something there.

If anybody has the solution, please let me know. If not, I'll report if I find something useful in my adventure with the kernel this weekend.

Thanks!

mfauvain commented 3 years ago

I had very similar experience unfortunately and simply gave up using thunderbolt on my 14,1. Having said this I run also an older macbook (I believe 10,1) connected to 2 thunderbolt displays (with ethernet connected on the display, plus some usb devices on the displays as well) and it has been working perfectly since then. I am running Arch, kernel 5.8.9 still, as moving to 5.9.x gave error with the sound output on the displays. I did nothing special to make the 2 thunderbolts ports work. And my 'work around' setup, which is actually much better for me is using my 14,1 with synergy as main computer on which I type etc.. sound is shared via pulseaudio (works perfect), clipboard via CopyQ.. etc.

One thing though, that might help. With this same 14,1 I use a USBc dock (not thunderbolt). And it randomly gives me trouble recognizing all ports on the dock apart from the Ethernet port... weird. And what seems to be working when I have this issue is :

worth trying for the thunderbolt issue I guess.

beat commented 3 years ago

I also had this problem, and found a solution, that I'm documenting here in case it can help others:

The solution here is bolt (https://gitlab.freedesktop.org/bolt/bolt). Install it via apt if not already installed:

sudo apt-get install bolt

you may need to reboot with the thunderbolt ethernet adapter after installation (mine was already installed).

It comes with an utility boltctl

sudo boltctl list

gives you the list of peripherals already seen, with an UUID, e.g.:

○ Apple Thunderbolt to Gigabit Ethernet Adapter #2
   ├─ type:          peripheral
   ├─ name:          Thunderbolt to Gigabit Ethernet Adapter
   ├─ vendor:        Apple, Inc.
   ├─ uuid:          c0b12300-0200-0100-ffff-ffffffffffff
   ├─ status:        connected
   │  ├─ domain:     00000000-0000-0018-0332-e6e03c12d249
   │  └─ authflags:  none
   ├─ connected:     dim 17 oct 2021 10:13:44
   └─ stored:        dim 12 sep 2021 09:34:10
      ├─ policy:     iommu
      └─ key:        no

You can see the policy "iommu" with key "no" preventing the automatic re-connection on plugin or wakeup.

Then there are 3 commands that can be used to temporarily or permanently authorize the device(s), using the UUID that appeared above:

To authorize temporarily a device:

boltctl authorize c0b12300-0200-0100-ffff-ffffffffffff

But to authorize permanently, first it needs to be removed from the auto-stored list, then to be added again with the auto policy:

boltctl forget c0b12300-0200-0100-ffff-ffffffffffff

boltctl enroll --policy auto c0b12300-0200-0100-ffff-ffffffffffff