Upstream Sound

[ghost]

Hello, I'm trying to work on upstreaming sound support with some others. The main step is to find the minimal patch on top of upstream kernel in order to have the bdw-rt5677 sound card up show up in the output of $ aplay -l.

@aeroevan, @ehegnes, @reynhout, and I have discussed this some on the #linux-samus channel on Freenode IRC. Please feel free to join and ask questions, share thoughts, etc. I'm posting this issue to summarize what we've found so far, and in hopes others are interested as well.

linux-samus has been great as a stop-gap up until now, but besides audio, everything else on the Chromebook Pixel is working on the upstream linux kernel I believe.

One caveat: first time for me doing non-trivial things with the kernel source, so please take the below with a grain of salt.

Background

Currently, one can find all Chrome OS kernel code in their forked version of the upstream Linux kernel here: link. Unfortunately, even just focusing on the sound subsystem, the two have diverged quite a lot.

The first step was to ask the maintainers of the Chrome OS code if they could upstream the code. Unfortunately it appears they can't devote the time.

You can see the scripts directory to see how the linux-samus repo gets around this. Basically, build.sh is run which calls patch.sh, which clones the upstream kernel and the Chrome OS kernel and then calls create_patch.sh, which, for each of the 41 files related to sound (this is a good reference for later), makes a mega patch of the diffs of the files. It also adds haswell-Makefile.patch to the mix. patch.sh then calls apply_patch.sh which applies the newly created patch along with monkey.patch. Presumably @raphael does some extra work to edit/compile/test/repeat until things are working.

You might also want to refer to @tsowell's repo here, who was the first (?) to make sound work on linux for CB Pixel, and is basically raphael's approach except hard-coded for Linux 3.19. Unfortunately the patch used is one mega-sized-patch. But there are some good notes there.

Lastly, I've also found it helpful to reference @jakeday's repo here towards this end as well.

The Chrome OS kernel tree

The Chrome OS kernel has some interesting branches.

• chromeos-3.14: The branch used by this repo to get the diffs. Note that it has a whole bunch of backports. This is the one from which most investigation thus far has been on.
• chromeos-4.4: This branch would hopefully be more similar to Linux 4.4 and we could work from there! Unfortunately, it doesn't appear to have the bdw-rt5677.c driver.
• chromeos-3.19: The only other 3.xx branch that is higher than 3.14.
• Any branch matching $(git branch -vv --all | grep samus). Presumably these are branches tracking releases for the CB Pixel. I haven't spent time looking at how much they differ from chromeos-3.14. They warrant more looking-into.

As aeroevan noted, the Chrome OS tree appears to have a whole bunch of backports, so are somewhat Frankenstein in nature.

1st Relevant Patch

First we need to get the driver added. From the chromium tree, look at this commit:

71e498c497becf7db17d6d8ef8283e820aad78de
CHROMIUM: ASoC: bdw-rt5677: Add ASoC machine driver for Broadwell platforms with RT5677 codec
Online: https://chromium-review.googlesource.com/221304

aeroevan made the first step a while ago by cherry-picking the first commit described above.

Unfortunately, on compile+boot (make sure to also enable the new kernel config item), just this first one by itself results in no recognition by ALSA, and the observed complaint in dmesg:

bdw-rt5677 bdw-rt5677: ASoC: CPU DAI Capture Pin not registered

Note that this error message is printed out when soc_bind_dai_link() fails at any of three points. At this point you may want to search the issues for this message 'not registered' to see various reports of seeing this and what caused it. It appears to most commonly be due to a misplaced IntcSST2.bin firmware -- see more on that below.

The Capture Pin refers to the microphone input. Only interested in sound output, I modified the bdw_rt5677_dais static array to remove both the 'Capture' DAI and the 'Wake on Voice' DAI. Not surprisingly, this just went on to complain that the Codec snd-soc-dummy-dai fails to register as well.

2nd Relevant Patch

So it It appears aeroevan, reynhout, and I all indepently saw the following commit as a natural one to see what it affected next.

9018f70c26ac3b0a8e8df5f2cfe86392db8d5b53
CHROMIUM: ASoC: rt5677: Add ACPI device probing
Online: https://chromium-review.googlesource.com/207622

Personally, it seemed to be what I wanted: get the codec (rt5677) to correctly link with the new bdw-rt5677 driver. What was especially encouraging was that building the linux-samus without this commit causes aplay -l to no longer show the device and causes dmesg to say:

bdw-rt5677 bdw-rt5677: ASoC: CODEC i2c-RT5677CE:00 not registered

However, applying this patch on top of the 1st Relevant Patch results in a kernel oops while inside rt5677_i2c_probe() that we haven't investigated enough to figure out why it is happening yet.

You can see the dmesg from when this oops happen here (just search for "Oops") and look at the stack trace below. To contrast, you can see a regular linux-samus dmesg here. So it appears the Oops is a null pointer deref; I did try to look online at how to use the Oops Code to find which specific assembly instruction in the kernel is causing the deref but IIRC it was slightly inscrutable. Probably worth trying again.

At this point it is good to note the Intel sound firmware: IntcSST2.bin. Most likely your distro firmware package already ships it: You would find it in somewhere like /usr/lib/firmware/intel/ or /lib/firmware/intel/. If you don't find it, you can download it manually from the repo in the firmware/ directory. Note that the 1st Relevant Patch from above modifies the Intel SST code to link/associate (?) the bdw-rt5677 driver. Recall above that people having similar 'not registered' errors had misplaced IntcSST2.bin somewhere. So one potential thing going wrong is that somehow, despite the change, we aren't correctly associating IntcSST2.bin with the right items. You can see the dmesg output here that happens when booting linxus-samus without IntcSST2.bin in place. The 'not registered' result does seem relevant, though possibly a red herring.

You might notice that both of these patches were written by someone named Ben Zhang. Some of us have attempted emailing this person at various points but have yet to hear a response (not that he has any obligation to respond of course).

Firmware red herring

• Look at the commit af48f1d08a5474184da9aaf8b77f4a2848b1875e in the upstream kernel which introduces VAD into the rt5677 codec, and then further look at the commit 1865cf21076d8325424b6ba3d972221ebcc4e4a7 in the chromium kernel tree.
• Next observe that booting into a properly-set-up linux-samus, without the rt5677_elf_vad firmware nor the en_us.mmap file, we can still get a successful aplay -l.
• We can infer from all this that en_us.mmap is used by rt5677_elf_vad to do english-US word detection for the 'OK GOOGLE' wake-on-voice feature that the CB Pixel has.
• It follows that we can cross out a red herring: You might notice that the mainline linux rt5677.c codec requests two firmware: RT5677_FIRMWARE1 and RT5677_FIRMWARE2, whereas in the chromium tree there is instead a call for request_firmware(&fwp, "rt5677_elf_vad", dev);.
• But once again, none of the three of these firmware are actually needed for ALSA to recognize the device, so this is irrelevant.

  'Not Registered'

• Following the code, my best guess is that the part in soc_bind_dai_link() that's erroring out (see link to soc-core.c file above) is the call to snd_soc_find_dai(). My impression is that snd_soc_find_dai() returns NULL because the attempt to register the device in the first place (add it to the component list) never succeeds. It appears the only function that adds to the component list is the snd_soc_register_component(). More code-reading is needed here, e.g. where is this function called?
• In general, we need to look more into the execution path that determines that the mapping is OK vs not registered.

  rt5677.c vs. rt5677-spi.c

• It's not clear what the role of rt5677-spi.c is. And how does this role differ from rt5677.c? Need to read the code again.

  Other potential TODOs to keep in mind

• Look at the list of the 41 files that are used by the linux-samus build scripts. Examine each one.
• Do printk-debugging to find what is the actual path of the code leading up to complaining that things are getting 'not registered'. Correspondingly add print statements e.g. to the driver-loading code.
• Try an overshoot and take-away approach, rather than a build-from-scratch.
• Ask on #kernelnewbies on oftc.net IRC
• Try different upstream kernel versions. 3.14? 3.19? 4.4? Latest?
• Email http://vger.kernel.org/vger-lists.html#alsa-devel and http://vger.kernel.org/vger-lists.html#linux-sound
• Email kernelnewbies mailing list

  IMPORTANT: Kernel Config

  These ones are crucial for sound it seems

CONFIG_DW_DMAC=y

CONFIG_SND_SOC
CONFIG_SND_SOC_I2C_AND_SPI
CONFIG_SND_SOC_INTEL_SST
CONFIG_SND_SOC_INTEL_SST_ACPI

CONFIG_SND_SOC_INTEL_BROADWELL_MACH (selects CONFIG_SND_SOC_INTEL_HASWELL and CONFIG_SND_SOC_RT286)
CONFIG_SND_SOC_INTEL_HASWELL_MACH

CONFIG_SND_SOC_INTEL_BDW_RT5677_MACH (selects CONFIG_SND_SOC_RT5677 and CONFIG_SND_SOC_RT5677_SPI)

That last one is one you get access to from the '1st Important Patch'.

To Help With Debugging

CONFIG_SND_DEBUG
CONFIG_SND_DEBUG_VERBOSE
CONFIG_SND_VERBOSE_PRINTK

Advice

I recommend the following initial setup

1. Once you've chosen a upstream-version to try things on, use make localmodconfig for that version to get a minimal kernel config (to lower compile times)
2. Manually edit it using your favorite make-config tool to setup the sound-config listed above.
3. Clone linux-samus, and use this new config (using make oldconfig or make olddefconfig) to compile linux-samus, boot in, and make sure everything looks sane, bdw-rt5677 shows up in aplay -l, etc.
4. Use this new config for the routine below

   One Possible Routine

5. Clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git(to work on)
6. Clone https://chromium.googlesource.com/chromiumos/third_party/kernel (for reference)
7. Clone git@github.com:raphael/linux-samus.git (for reference)
8. Checkout the branches you want from each one
9. Edit linux source when ready
10. Compile and boot
11. $ aplay -l
12. $ dmesg
13. $ ls -l /proc/asound
14. $ cat /proc/asound/{cards,devices,modules}
15. $ ls -l /dev/snd
16. goto step 5

[reynhout]

@genkimarshall This is really great. It captures everything I've gathered as well, plus some new ideas to pursue.

I've been focused adding audio support to the 4.4.15 mainline kernel, as a starting point for moving on to 4.5+.

My first attempt was to follow raphael's pattern of bringing the 3.14 sound files forward, then applying his patches. This failed to build on 4.4.15. I spent some time updating calls, but since it didn't match my real goals, I abandoned it before too long. It might be informative to find the point in 4.4.x where the divergence becomes significant. Somewhere between raphael's 4.4.2 (or 4.4.6?) and 4.4.15.

Later attempts began with the 4.4.15 mainline tree and selectively brought over files from raphael's list. Only a few appeared necessary, and after minor cleanup and integration changes, the driver would build successfully, but I hit the same pin registration errors you mentioned.

Support for the RT5677 seems to begin with the chromeos-3.14 branch, and it looks like chromeos-3.18 adds hotword detection. The only more recent branch is chromeos-4.4, which does include the RT5677 codec, but not the machine driver we need. Perhaps Google intends to keep SAMUS on the 3.14 kernel?

I've also tried to find a similar machine driver from 3.14 and checking its diffs against the 4.4 tree. In general, these diffs are fairly small, mostly structural. bdw-rt5677.c appears to be a one-off written by Ben Zhang, whereas the others are fairly standardized and written by people at Intel, so there were no great candidates for comparison

This is somewhat encouraging though -- it implies that the Linux audio subsystem isn't greatly different, and that the Ben Zhang driver should be able to be pulled forward without huge updates.

It does look like we've exhausted all hope of finding a simple shortcut. I'll need to do some more reading to see if I can figure out how these drivers are supposed to work.

[coolstar]

I can confirm here rt5677-spi.c plays no role whatsoever on the Pixel 2. I wasn't able to outright remove the module without Linux's dependency tree breaking and not loading any of the modules. However, I was able to nop out all the functions (make them just return 0 and do nothing) in rt5677-spi.c and still have working audio:

https://ghostbin.com/paste/aspdc

[reynhout]

@coolstar If I'm reading the code correctly, the rt5677-spi.c driver is only used for the wake-on-voice "OK Google" detection. So it is not needed outside of ChromeOS.

[coolstar]

@reynhout yes that's Intel Smart Sound Wake on Voice: http://www.intel.com/content/www/us/en/architecture-and-technology/smart-sound-technology.html

It's only used for "OK Google" in Chrome OS or "Cortana" in Windows

[coolstar]

Also something to note. IntcSST2.bin (the binary blob) is the same blob in both the Windows and Linux Intel Smart Sound Audio drivers. I did find some drivers for the ALC5677 on Windows (with the 10EC5677 hardware id instead of RT5677CE) however those drivers didn't work since they were for bay trail only...

As far as Linux goes I took a look at the upstream rt5677 and I have an idea of what needs to be changed to update the audio driver to kernel 4.7.

[sprc]

It looks like someone submitted the sound driver upstream yesterday:

https://lkml.org/lkml/2016/8/14/207

[raphael]

Thanks for the pointer! I applied the patches to 4.7 and sound is working just fine. I'm also not having any sleep issues like I did on 4.5. I released v4.7-1 that uses the 4.7 kernel patched with these upstream patches.

[aeroevan]

Looks like his git tree is here: https://github.com/johnkeeping/linux/tree/jk/chromebook-audio

[coolstar]

I'm running the deb builds for Linux 4.7-3 on samus, however audio is still non-functional. Also I see a bunch of errors in dmesg :/ https://ghostbin.com/paste/22s8h

[dsem]

Looks like this is slated for inclusion in 4.9: https://lkml.org/lkml/2016/10/5/314

[rroll1]

Out of the box 4.9-rc1 doesn't appear to have supported sound. Not sure if there's anything that needs to be done after booting into that kernel?

Of note: Wifi also doesn't appear to be working in rc1, but I'm not sure if that's because I had to disable CONFIG_MODVERSIONS to get the kernel to boot.

[aeroevan]

It works for me, are you sure your config has the module (CONFIG_SND_SOC_INTEL_BDW_RT5677_MACH) selected?

[rroll1]

@aeroevan: Yup that did it. The AUR package for Linux Mainline is still using the 4.8 config, so it didn't have any of the new modules in it. Added that one and it works flawlessly now.

Thanks!

[dsem]

Hmm.. Having trouble with 4.9 from Fedora's fedora-rawhide-kernel-nodebug repo. The module appears to be included and loaded:

[user:~] $ lsmod | grep 5677
snd_soc_sst_bdw_rt5677_mach    16384  4
snd_soc_rt5677        229376  5 snd_soc_sst_bdw_rt5677_mach
snd_soc_rt5677_spi     16384  1 snd_soc_rt5677
snd_soc_sst_haswell_pcm    90112  3 snd_soc_sst_bdw_rt5677_mach
snd_soc_rl6231         16384  1 snd_soc_rt5677
snd_soc_core          217088  3 snd_soc_rt5677,snd_soc_sst_haswell_pcm,snd_soc_sst_bdw_rt5677_mach
snd_pcm               118784  8 snd_hda_intel,snd_hda_codec,snd_pcm_dmaengine,snd_hda_core,snd_soc_rt5677,snd_hda_codec_hdmi,snd_soc_sst_haswell_pcm,snd_soc_core

but the gnome sound settings don't show any devices (and there's no sound).

Here are the logs:

[user:~] $ journalctl -b --no-pager | grep 5677
Nov 21 08:16:36 pixel kernel: bdw-rt5677 bdw-rt5677: snd-soc-dummy-dai <-> System Pin mapping ok
Nov 21 08:16:36 pixel kernel: rt5677 i2c-RT5677CE:00: Can't add headphone jack gpio
Nov 21 08:16:36 pixel kernel: rt5677 i2c-RT5677CE:00: Can't add mic jack gpio
Nov 21 08:16:36 pixel kernel: bdw-rt5677 bdw-rt5677: rt5677-aif1 <-> snd-soc-dummy-dai mapping ok
Nov 21 08:16:36 pixel kernel: input: bdw-rt5677 Headphone Jack as /devices/pci0000:00/INT3438:00/bdw-rt5677/sound/card0/input10
Nov 21 08:16:36 pixel kernel: input: bdw-rt5677 Mic Jack as /devices/pci0000:00/INT3438:00/bdw-rt5677/sound/card0/input11
Nov 21 08:16:46 pixel /usr/libexec/gdm-x-session[1427]: (II) config/udev: Adding input device bdw-rt5677 Headphone Jack (/dev/input/event10)
Nov 21 08:16:46 pixel /usr/libexec/gdm-x-session[1427]: (II) config/udev: Adding input device bdw-rt5677 Mic Jack (/dev/input/event11)

Any idea what piece could be missing?

[dsem]

Well, the culprit was PulseAudio. I'm not sure if this was my doing or what, but I had two module config files in /etc/pulse/: default.pa and default.pa.orig, last modified April 2016. They were the same except for default.pa had two extra lines:

load-module module-alsa-source device=hw:0,1
load-module module-alsa-source device=hw:0,2

Removing these lines and rebooting caused the audio to work again.

[rroll1]

@dsem I actually just moved to Fedora myself, and I'm having trouble getting sound enabled on the 4.9 rawhide kernel.

It looks like you had your default.pa file replaced, but I just had the default (mostly fresh install). I even deleted my custom settings in .config/pulse

The odd thing is that Pulse is actually reporting sound -- like I'm seeing a bouncing line in Pavucontrol that corresponds to the music that's playing, but no sound is coming out of my speakers or headphones.

I'm kind of at a loss as to what to do at this point. The computer says it's working, but my ears seem to disagree. Nothing is muted, but in alsamixer I do see that the speakers and headphones, while not muted, don't have any volume to them, and I can't increase the volume there.

Would you (or anyone else) happen to have some insight? I know this repo tends to be Arch-oriented, though.

Thanks!

EDIT: Nevermind, it was actually enabled the same way as Raphael's audio script -- the only issue is that the audio script seems to reference different audio device names. I'm gonna see if I can figure out how to tweak it (not super familiar with alsa/pulse).

Specifically I had to change -c to 1, not 0, for Fedora. Though now I'm hearing a lot of chassis noise, which is strange -- I can hear myself typing through the headphones, and there's a whine. I'm sure it's just a setting in alsamixer I can futz with. I'll have to tweak some stuff to see if I can fix it.

[dsem]

Glad you got it working. Sounds like your mic might not be muted or might be too sensitive. Or maybe there's some sort of noise-cancelling module that's not properly working.

[raphael]

The latest release v4.9-1 is running stock v4.9 with no patch. Sound is working fine for me (speakers, mic and headphones).

[raphael]

I'll close this issue. The only value this repo provides now is the customized config.

[chepurko]

@raphael so if I just installed the upstream kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.9/ where would I obtain the linux-firmware package? Or is firmware included in linux-image for the Pixel 2? Basically I'm wondering how Pixel 2 users, specifically on Ubuntu, would go about updating with the mainline kernels if they're fully supporting my hardware at this point.