ValveSoftware / SteamOS

SteamOS community tracker
1.61k stars 70 forks source link

Undocking the Deck from the Dock causes network to be DoS from the ethernet port #1187

Closed RodoMa92 closed 10 months ago

RodoMa92 commented 1 year ago

Your system information

Please describe your issue in as much detail as possible:

I'm not sure if anyone from the team is still investigating this issue, the relevant link is this one here. Quite recently a user with the proper tools managed to capture and debug the cause, link here. It seems that the Dock begins to spew ethernet pause frames over and over after a number of random disconnection event from the Deck if the dock is plugged in into the outlet. How a network adapter would do something like that when there is no upstream USB comms is beyond my understanding, however quite a lot of people would want this fixed.

Can you please check with your internal team for the dock if they can reproduce it and properly fix it? At least one people reproduced it using Windows, so it doesn't seem driver related.

Steps for reproducing this issue:

  1. Connect the Dock to the Deck
  2. Disconnect it and replug it for a number of times
  3. Observe the network LED go mad and pull down your entire network with it.
unclejack commented 1 year ago

This has also been reported by others with different hardware: https://lucumr.pocoo.org/2020/7/6/usb-c-network-hubs/ https://mjtsai.com/blog/2022/05/11/usb-c-hubs-breaking-ethernet-networks/

If the Steam Deck's Dock firmware has the means to disable the ethernet port when not connected to a host or to disable the transmission of pause frames completely, they might be able to fix it. Otherwise the issue persists until the device gets recycled.

RodoMa92 commented 1 year ago

Yep, you are right on point, this is the exact same issue reported on these posts.

How can something like this be a standard occurrence with these devices is beyond me, but I seriously hope that they can just shut off the ethernet adapter on USB c removal, although I'm not sure if they have the ability to to that (if they have a 10 Gbit uplink between the deck and the dock using a MUX they should be able to just cut either USB power/data on a disconnect event and therefore fix it without additional changes at the adapter level, but without a block diagram of this thing it's hard to say).

I hope that someone who works at Valve will respond with more details on this.

RodoMa92 commented 1 year ago

Luckily I've managed to found a couple of images of the internal of the official dock, and just by taking a look at the corresponding datasheets, it seems that it's definively possible to power off individual ports at least on the integrated 4 port hub of the VL817 USB hub. I guess it's just question on how much do Valve cares about this issue.

Still haven't took a look into the main USB-C To Video-Out + USB mux (KTM5030), but I guess you should also be able to just disable the whole hub from that side as well, maybe even lowering power consumption further (or even leaving that software tunable for people that wants their controller charged when the deck is sleeping).

All of the three main components (USB-C -> Video + USB 3.1, USB 3.1 to 4 port USB 3.1 and the ethernet adapter AX88179A (which is connected internally to one port of the hub, confirmed by lsusb tree command, see below)) have a corresponding SPI/BIOS looking chip close to them, so they should be definitively firmware tunable. To what amount remains to be seen, but the above should be doable in my opinion.

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M
    ID 1d6b:0003 Linux Foundation 3.0 root hub
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        ID 2109:0817 VIA Labs, Inc. 
        |__ Port 4: Dev 3, If 0, Class=Communications, Driver=cdc_ncm, 5000M
            ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
        |__ Port 4: Dev 3, If 1, Class=CDC Data, Driver=cdc_ncm, 5000M
            ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet
RodoMa92 commented 1 year ago

I wouldn't have guessed. The firmware that dropped one day ago in the repos seem to actually do the proposed above thing to power off the ethernet adapter while leaving the other three ports untouched and still powered to charge peripherals.

Assuming is not a fluke, this seems to be finally fixed.

I'll keep an eye out if it comes back, if not, I'll close this.

Thanks, Valve.

RodoMa92 commented 1 year ago

After doing multiple tests with disconnecting the dock from the deck, put it to sleep, plugging and unplugging the ethernet cable with or without the dock attached I can not get any LEDs from the adapter unless the Deck is powered on. To me, this seems to behave as I proposed above now.

Closing this, thanks again.

RodoMa92 commented 1 year ago

OK, sadly it's not completely fixed. Assuming you did implement what I suggested above in firmware .121, it happens that the dock sometimes interpret an unplug event as a disconnect -> reconnect event and does miss the fact that the Deck is not connected anymore to the dock, and the ethernet adapter seems to remain active again (and potentially cause havoc again). A suggestion I might want to give you is to wait for a delay of 250ms beetween each event, if there is no way to actually check for an active communication with the Deck.

RodoMa92 commented 1 year ago

Today this has happened again. .121 firmware. Please, Valve, properly fix this stuff.

RodoMa92 commented 1 year ago

On .123 it seems to behave better for now. Haven't used it much up until now tho. I'll keep it monitored and report back if something breaks again.

RodoMa92 commented 11 months ago

I have managed to fix my keyboard now, so now I'm able to test it again. From the last week of testing it seems to behave as suggested above, so it might be finally fixed. I'll still keep testing for a while before closing this, tho. If other people affected by this can also test their devices with .123, it might be faster to finally close this chapter on the Dock.

Thanks, Valve. Better late than never at least.

syberphunk commented 10 months ago

Oh, so this is what was pulling down my d-link unmanaged switch and causing it to completely freeze the network up.

RodoMa92 commented 10 months ago

Yeah, after a month of testing this I can't reproduce this anymore so I feel more confident in closing this.

When Valve will drop the .123 firmware update it should be gone for good.

kneciu1337 commented 10 months ago

Was trying to install .123 but I’m getting permission denied on line 16 even through sudo, any ideas?

RodoMa92 commented 10 months ago

Was trying to install .123 but I’m getting permission denied on line 16 even through sudo, any ideas?

Right click on the hub_update binary, click properties and go in the permission tab, you probably lost the executable flag. Just check it on that tab and press OK.

kneciu1337 commented 10 months ago

Was trying to install .123 but I’m getting permission denied on line 16 even through sudo, any ideas?

Right click on the hub_update binary, click properties and go in the permission tab, you probably lost the executable flag. Just check it on that tab and press OK.

Thank a lot for the tip, this did the trick and installed without issues.

My1 commented 4 months ago

have the issue too, I need to check what my dock firmware is but I had it happen today again, almost all devices were not able to see the network and get their DHCP, micraculously the PS5 didnt seem to care but it was weird.

RodoMa92 commented 4 months ago

have the issue too, I need to check what my dock firmware is but I had it happen today again, almost all devices were not able to see the network and get their DHCP, micraculously the PS5 didnt seem to care but it was weird.

You probably are under .123. Just update to it or to an higher build if you can and it will go away.

My1 commented 4 months ago

okay I see .121 but I dont find any update functions, any chance .123 isnt in stable yet?

RodoMa92 commented 4 months ago

okay I see .121 but I dont find any update functions, any chance .123 isnt in stable yet?

Sadly I have no clue, since I left SteamOS like a year ago. You can do that manually if you feel confident on doing it, there is a general procedure above. Another way would be to switch temporarily to main, update the dock and then go back to stable.