Wireless device stopped working in NixOS (but works in Ubuntu)

Zer0- commented 7 years ago

Issue description

I have a pci-e wireless aetheros card with a QCA9882 chipset. I used to use it to serve up a wireless AP with hostapd. It would only work in 2.4Ghz but suddenly it stopped working completely under NixOS. I don't know what changed. ifconfig -a and iwconfig both don't list any wireless devices (see below).

However when I booted up a Ubuntu liveusb I was able to use nmcli to connect to and access point and ping Google (with ethernet physically unplugged). So I'm pretty sure this is not a hardware issue.

ifconfig -a output (NixOS)

enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:24:1d:8f:d3:63  txqueuelen 1000  (Ethernet)
        RX packets 6322204  bytes 442019445 (421.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10946545  bytes 15291632450 (14.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s6f0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 00:04:23:df:13:44  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp4s6f1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 00:04:23:df:13:45  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 5248  bytes 503833 (492.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5248  bytes 503833 (492.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vboxnet0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.56.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 0a:00:27:00:00:00  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

iwconfig output (NixOS)

lo        no wireless extensions.

enp4s6f1  no wireless extensions.

vboxnet0  no wireless extensions.

enp3s0    no wireless extensions.

enp4s6f0  no wireless extensions.

`dmesg | grep ath10k` output (NixOS)

[   37.187424] ath10k_pci 0000:02:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   37.472653] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:02:00.0.bin failed with error -2
[   37.472670] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/cal-pci-0000:02:00.0.bin failed with error -2
[   37.581317] ath10k_pci 0000:02:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043222ff sub 0000:0000
[   37.581319] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 0 tracing 0 dfs 0 testmode 0
[   37.581458] ath10k_pci 0000:02:00.0: firmware ver 10.2.4.70.63-2 api 5 features no-p2p,raw-mode,mfp crc32 4ff9e14d
[   37.621594] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
[   37.657912] ath10k_pci 0000:02:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[   38.786332] ath10k_pci 0000:02:00.0: firmware crashed! (uuid n/a)
[   38.786379] ath10k_pci 0000:02:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043222ff sub 0000:0000
[   38.786381] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 0 tracing 0 dfs 0 testmode 0
[   38.786535] ath10k_pci 0000:02:00.0: firmware ver 10.2.4.70.63-2 api 5 features no-p2p,raw-mode,mfp crc32 4ff9e14d
[   38.786539] ath10k_pci 0000:02:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[   38.786541] ath10k_pci 0000:02:00.0: htt-ver 0.0 wmi-op 5 htt-op 2 cal otp max-sta 128 raw 0 hwcrypto 1
[   38.788553] ath10k_pci 0000:02:00.0: firmware register dump:
[   38.788586] ath10k_pci 0000:02:00.0: [00]: 0x4100016C 0x000015B3 0x009BFF84 0x00955B31
[   38.788620] ath10k_pci 0000:02:00.0: [04]: 0x009BFF84 0x00060130 0x00000013 0x00000000
[   38.788653] ath10k_pci 0000:02:00.0: [08]: 0x009C8EA4 0x00007170 0x00000001 0x00435300
[   38.788687] ath10k_pci 0000:02:00.0: [12]: 0x00000009 0x00000000 0x00958360 0x0095836B
[   38.788720] ath10k_pci 0000:02:00.0: [16]: 0x00958080 0x0094078E 0x00000000 0x00000000
[   38.788753] ath10k_pci 0000:02:00.0: [20]: 0x409BFF84 0x0040AB44 0x004101E0 0x0000000C
[   38.788786] ath10k_pci 0000:02:00.0: [24]: 0x809BFE1C 0x0040ABA4 0x00000006 0xC09BFF84
[   38.788819] ath10k_pci 0000:02:00.0: [28]: 0x8098E907 0x0040ABD4 0x009C8E7C 0x0000016B
[   38.788852] ath10k_pci 0000:02:00.0: [32]: 0x8098FD3B 0x0040ABF4 0x004209EC 0x00420B0C
[   38.788885] ath10k_pci 0000:02:00.0: [36]: 0x8098A7DA 0x0040AC34 0x00420AA0 0x004062DC
[   38.788919] ath10k_pci 0000:02:00.0: [40]: 0x80988785 0x0040AC94 0x00410760 0x00400000
[   38.788952] ath10k_pci 0000:02:00.0: [44]: 0x8098AEC8 0x0040AD44 0x00410760 0x00000001
[   38.788989] ath10k_pci 0000:02:00.0: [48]: 0x809B67C4 0x0040AD84 0x00410760 0x00411344
[   38.789022] ath10k_pci 0000:02:00.0: [52]: 0x809B293C 0x0040ADA4 0x00400000 0x00416FA4
[   38.789055] ath10k_pci 0000:02:00.0: [56]: 0x809C0C3D 0x0040ADE4 0x0040AE08 0x00412118
[   44.144104] ath10k_pci 0000:02:00.0: wmi unified ready event not received
[   44.202279] ath10k_pci 0000:02:00.0: could not init core (-110)
[   44.202309] ath10k_pci 0000:02:00.0: could not probe fw (-110)
[   44.211111] ath10k_pci 0000:02:00.0: cannot restart a device that hasn't been started

ifconfig output (Ubuntu)

enp3s0    Link encap:Ethernet  HWaddr 00:24:1d:8f:d3:63  
          inet addr:192.168.4.3  Bcast:192.168.4.255  Mask:255.255.255.0
          inet6 addr: fe80::1902:178a:dcba:1986/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1775 errors:0 dropped:0 overruns:0 frame:0
          TX packets:360 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2469612 (2.4 MB)  TX bytes:32487 (32.4 KB)

enp4s6f0  Link encap:Ethernet  HWaddr 00:04:23:df:13:44  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

enp4s6f1  Link encap:Ethernet  HWaddr 00:04:23:df:13:45  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:208 errors:0 dropped:0 overruns:0 frame:0
          TX packets:208 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:16012 (16.0 KB)  TX bytes:16012 (16.0 KB)

wlp2s0    Link encap:Ethernet  HWaddr 00:0e:8e:5b:1f:b6  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

iwconfig output (Ubuntu)

enp4s6f0  no wireless extensions.

enp4s6f1  no wireless extensions.

lo        no wireless extensions.

enp3s0    no wireless extensions.

wlp2s0    IEEE 802.11  ESSID:off/any  
          Mode:Managed  Access Point: Not-Associated   Tx-Power=0 dBm   
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Encryption key:off
          Power Management:on

`dmesg | grep ath10k` output (Ubuntu)

[   33.610905] ath10k_pci 0000:02:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   33.801925] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:02:00.0.bin failed with error -2
[   33.801949] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/cal-pci-0000:02:00.0.bin failed with error -2
[   33.841143] ath10k_pci 0000:02:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043222ff sub 0000:0000
[   33.841145] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 1 tracing 1 dfs 0 testmode 0
[   33.841340] ath10k_pci 0000:02:00.0: firmware ver 10.2.4.70.9-2 api 5 features no-p2p,raw-mode crc32 b8d50af5
[   33.881565] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
[   33.901264] ath10k_pci 0000:02:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[   35.027593] ath10k_pci 0000:02:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 cal otp max-sta 128 raw 0 hwcrypto 1
[   35.151165] ath10k_pci 0000:02:00.0 wlp2s0: renamed from wlan0

My hardware-configuration.nix

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, linuxManualConfig, ... }:

{
  imports =
    [ <nixpkgs/nixos/modules/installer/scan/not-detected.nix>
    ];

  boot.kernelPackages = pkgs.linuxPackages_4_9;
  boot.initrd.availableKernelModules = [ "ahci" "ohci_pci" "ehci_pci" "pata_atiixp" "firewire_ohci" "usb_storage" "floppy" "sd_mod" "sr_mod" ];
  boot.kernelModules = [ "kvm-amd" ];
  boot.extraModulePackages = [ ];
  boot.extraModprobeConfig = ''
    options ath10k_core skip_otp=y
    options cfg80211 ieee80211_regdom=CA
  '';
  hardware.firmware = [ pkgs.firmwareLinuxNonfree ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/b92bad80-3705-4405-bc47-e20f72db7f78";
      fsType = "btrfs";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/cb32307a-99f2-4a30-ac47-b456ddd75b6c";
      fsType = "ext2";
    };

  swapDevices =
    [ { device = "/dev/disk/by-uuid/6be80b01-2817-4a9e-8c39-566b39273267"; }
    ];

  nix.maxJobs = lib.mkDefault 6;
  powerManagement.cpuFreqGovernor = "ondemand";
}

I can provide more information, but do not know what to include. I'd like to mention that I have tried to boot a NixOS entry (from grub) from a date I know for sure my wifi worked and this did not work.

EDIT - Additional information

I booted up a live usb of nixos-minimal-17.03.1463.81fceb2554-x86_64-linux.iso which correctly showed my device as wlp2s0.

I also booted up nixos-minimal-17.09.2034.78eed74497-x86_64-linux.iso which did not work.

I remain hopeful that this is a regression and there could be a fix.

Steps to reproduce

If you have the same hardware try using wifi with NixOS and Ubunutu. That device will not be usable in NixOS.

Technical details

System: (NixOS: nixos-version, Ubuntu/Fedora: lsb_release -a, ...)

NixOS: 17.09.2034.78eed74497 (Hummingbird)
Ubuntu: ubuntu-16.04.3-desktop
md5 of Ubuntu iso used:
0d9fe8e1ea408a5895cbbe3431989295

Nix version: (run nix-env --version)
```
nix-env (Nix) 1.11.15
```
Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion)
```
17.09.2034.78eed74497
```
Sandboxing enabled: (run grep build-use-sandbox /etc/nix/nix.conf)
```
build-use-sandbox = false
```

orivej commented 7 years ago

Clearly, Ubuntu uses firmware 10.2.4.70.9 and it works, while NixOS uses firmware 10.2.4.70.63 and it crashes. Could you try reverting 8ba0b7bc3b7a0c359aced533295862e3b92ac9e8? (You can regain WiFi on NixOS with nixos-rebuild --rollback.)

Zer0- commented 7 years ago

Commit 8ba0b7b is not on the branch release-17.09 so I could not revert it. I did however cherry pick it to that branch and now I have a wifi interface. My dmesg now:

[   37.102096] ath10k_pci 0000:02:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   37.381527] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:02:00.0.bin failed with error -2
[   37.381546] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/cal-pci-0000:02:00.0.bin failed with error -2
[   37.449812] ath10k_pci 0000:02:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043222ff sub 0000:0000
[   37.449814] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 0 tracing 0 dfs 0 testmode 0
[   37.449957] ath10k_pci 0000:02:00.0: firmware ver 10.2.4.70.54 api 5 features no-p2p,raw-mode,mfp crc32 9d340dd9
[   37.490065] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
[   37.502760] ath10k_pci 0000:02:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[   38.637293] ath10k_pci 0000:02:00.0: htt-ver 2.1 wmi-op 5 htt-op 2 cal otp max-sta 128 raw 0 hwcrypto 1
[   38.755061] ath10k_pci 0000:02:00.0 wlp2s0: renamed from wlan0

So that commit is actually a fix. Should I open a PR to cherry pick it to 17.09?

orivej commented 7 years ago

@bendlas @fpletz Let's cherry pick 8ba0b7bc3b7a0c359aced533295862e3b92ac9e8 into the release?

orivej commented 7 years ago

It's interesting that the firmware version 10.2.4.70.54 is now smaller than before, maybe the upstream has realized that the newer version crashes…

bendlas commented 6 years ago

Let's cherry pick 8ba0b7b into the release?

Yeah keeping linux drivers up-to-date for stable makes sense, since they tend to break with kernel releases. Regarding the changed update process: I tried to mimic what had been in place so, for equal driver versions, there shouldn't be any difference in what we deliver. IMO we can backport it to stable, so that @wkennington and @fpletz are freed from the burden of maintaining that merged repository.

fpletz commented 6 years ago

Backported. Thank you!

nico202 commented 5 years ago

Mine stopped working (again, from november) with the same error. I'm on 19.03pre168781.929cc78363e (Koi)

[  557.853539] ath10k_pci 0000:02:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[  558.064795] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:02:00.0.bin failed with error -2                                                                        
[  558.064804] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/cal-pci-0000:02:00.0.bin failed with error -2                                                                            
[  558.064983] ath10k_pci 0000:02:00.0: qca9377 hw1.0 target 0x05020000 chip_id 0x003820ff sub 17aa:4035                                                                                         
[  558.064985] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 0 tracing 0 dfs 0 testmode 0
[  558.065518] ath10k_pci 0000:02:00.0: firmware ver WLAN.TF.2.1-00021-QCARMSWP-1 api 6 features wowlan,ignore-otp crc32 42e41877

srghma commented 5 years ago

Same error (wifi device disappeared) after switching on nixos-18.09.2337.5d972dd2c94 and Linux kernel 4.14.111

15554859898265644142197593319574

bendlas commented 5 years ago

@srghma Can you reproduce this on 19.03? We have updated to the most recent firmware-unfree release there

srghma commented 5 years ago

@bendlas tnx, I confirm that updating to 19.03 and linux kernel to 4.19.34 helped

NixOS / nixpkgs