Support USB OTG gadget serial with Zero 2 W

[joneskoo]

It'd be really cool to have USB OTG serial console work with Raspberry Zero 2 W. It should be possible to use the DWC2 g_serial to provide a serial console over USB OTG, without having to use UART for it. This would be great because it doesn't require a serial/TTL adapter and leaves the UART usable for Bluetooth.

[stapelberg]

Ah, so we’d boot the kernel with console=ttyUSB0?

I tried using kernel-console-on-ttyUSB for router7 a while ago:

• https://twitter.com/zekjur/status/1419921839074185217
• https://twitter.com/zekjur/status/1439612800561819649

As you can see, aside from the inconvenient setup, that console also lacks parts of the kernel log (possibly exactly the part that is relevant for debugging) and has reliability issues. I’m not a fan.

That said, if we can make it work, I’m not opposed to having it. But I don’t know yet whether it’ll be doable.

[joneskoo]

I think it'd be ttyGS0 for gadget serial but I think you're right about losing some of kernel messages. True for debugging gokrazy but less so for building appliances with gokrazy. Besides once you get to shell you can use dmesg to see the earlier messages.

It does have the side effect of using the only USB port obviously, so it's not for everyone either.

I think it's be still useful because on Zero you may not have working networking.

[stapelberg]

Now that https://github.com/gokrazy/kernel/commit/ee3d95c7f43687de895dda52ead895df9bf153da is submitted, I think we could take a more serious look at the OTG serial.

Can you outline how to enable/use it? Would it just be a matter of turning on a kernel option?

[joneskoo]

It needs dtb overlay dwc2 in config.txt and then the g_serial. Then /dev/ttyGS0 should be a serial port and it shows as a USB serial device on micro USB attached to the Zero/Zero 2 (data port, not charge port).

I should set it up again to have the specifics to cite.

[stapelberg]

I was curious about what the dwc2 overlay does, and according to https://raspberrypi.stackexchange.com/a/77061/132131, it switches from the dwc_otg driver to dwc2.

But, it looks like the dwc_otg driver is specific to the Raspberry Pi kernel and was never upstreamed to Linux itself.

AFAICT, we already use the dwc2 driver via our device tree:

                usb@7e980000 {
                        compatible = "brcm,bcm2835-usb";

(The Raspberry Pi dtb uses compatible = "brcm,bcm2708-usb";, which their driver matches in https://github.com/raspberrypi/linux/blob/80d7a012763d3ef3e63b2d4a1837c67ca935dd7b/drivers/usb/host/dwc_otg/dwc_otg_driver.c#L1048. The overlay file sets compatible = "brcm,bcm2835-usb";.)

So, perhaps we don’t need the overlay at all?

Perhaps we just need to enable CONFIG_USB_G_SERIAL=m and modprobe g_serial for this to work?

[joneskoo]

Sounds plausible. I'm fairly sure I had ttyGS0 working in alpine before (based on IRC logs I did).

https://gitlab.alpinelinux.org/alpine/aports/-/commit/f7c1d7e3a6a02a71501dbb9f6c84161f5eba83ea is related to making it work.

I think most likely these would be good for testing the gadget USB further.

CONFIG_USB_DWC2=m CONFIG_USB_ETH=m CONFIG_USB_G_SERIAL=m

Not sure if we need to configure dwc2 to switch from host to otg mode or something.

(For gadget ethernet too; could be useful if we can have USB ethernet for fallback?)

[joneskoo]

https://systemoverlord.com/2017/05/21/pi-zero-as-a-serial-gadget.html This works in Raspbian.

[   11.664086] dwc2 20980000.usb: supply vusb_d not found, using dummy regulator
[   11.668696] dwc2 20980000.usb: supply vusb_a not found, using dummy regulator
[   11.731092] systemd[1]: Started File System Check Daemon to report status.
[   11.903097] systemd[1]: Finished File System Check on Root Device.
[   11.923127] dwc2 20980000.usb: EPs: 8, dedicated fifos, 4080 entries in SPRAM
[   11.943745] dwc2 20980000.usb: DWC OTG Controller
[   11.947760] dwc2 20980000.usb: new USB bus registered, assigned bus number 1
[   11.951669] dwc2 20980000.usb: irq 33, io mem 0x20980000
[   11.987584] systemd[1]: Mounted FUSE Control File System.
[   12.003471] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[   12.003500] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   12.023650] hub 1-0:1.0: 1 port detected
[   12.260939] systemd[1]: Mounted Kernel Configuration File System.
[   12.308519] g_serial gadget: Gadget Serial v2.4
[   12.312252] g_serial gadget: g_serial ready
[   12.315923] dwc2 20980000.usb: bound driver g_serial
[   12.703149] dwc2 20980000.usb: new device is high-speed
[   12.737124] dwc2 20980000.usb: new address 3

systemctl enable getty@ttyGS0.service && systemctl start getty@ttyGS0.service

When loaded, this shows up in System Information on Mac connected to the USB port:

Loading g_ether didn't work for me.

[   12.099281] udc-core: couldn't find an available UDC - added [g_ether] to list of pending drivers

[stapelberg]

Thanks for the pointers.

When adding

CONFIG_USB_ETH=m
CONFIG_USB_G_SERIAL=m

(we already have CONFIG_USB_DWC2=y)

…I get:

/ # modprobe libcomposite
/ # modprobe g_serial
[   98.130101] g_serial 3f980000.usb: failed to start g_serial: -2
[   98.172859] g_serial 3f980000.usb: failed to start g_serial: -2
[   98.221157] g_serial 3f980000.usb: failed to start g_serial: -2
[   98.264847] g_serial 3f980000.usb: failed to start g_serial: -2
modprobe: 'kernel/drivers/usb/gadget/legacy/g_serial.ko': unknown symbol in module or invalid parameter

Not sure why that is, or if USB gadget mode is supposed to work with the upstream Linux kernel at all.

The Raspberry Pi kernel has a different device tree (see https://github.com/raspberrypi/linux/pull/1086/files), but copying the reg and interrupts values to the gokrazy kernel dtb makes the USB port no longer work with a USB network card attached.

[joneskoo]

Here's some thoughts, could turn out false leads.

The DTB may still have dr_mode as host instead of peripheral. Not sure if this could explain the error, I think not based on Googling.

I see legacy/ in the path. I think there's two g_serial in Linux but I have no clue if they are part of same or incompatible?

Finally perhaps there's some other problem that could explain not finding symbols. Missing some dependency in config? Not copying all module objects in? Depmod?

Oh and finally, the gadget serial would not work if you have a device connected (port is host mode). To have it work you need another computer as host on the micro USB. This can only work on Zero and Zero 2. I don't think this should cause the error that you got but also it can never actually work if you use the port as host.

[stapelberg]

Thanks for the thoughts! Answers inline:

The DTB may still have dr_mode as host instead of peripheral. Not sure if this could explain the error, I think not based on Googling.

We set CONFIG_USB_DWC2_DUAL_ROLE=y, so we shouldn’t need to specify host or peripheral at the device tree level. Both should work.

I see legacy/ in the path. I think there's two g_serial in Linux but I have no clue if they are part of same or incompatible?

I can only find one, the one with legacy/ in the path.

Finally perhaps there's some other problem that could explain not finding symbols. Missing some dependency in config? Not copying all module objects in? Depmod?

gokr-packer grabs the whole lib directory, so it can’t be forgotten kernel module files. Since commit https://github.com/gokrazy/kernel/commit/bd68817a1d570b387f670109ef1d576da6c071b0 we also have proper module dependency files (though not sure if busybox’s modprobe respects them). The only listed dependency for g_serial is libcomposite, which is loaded.

Oh and finally, the gadget serial would not work if you have a device connected (port is host mode). To have it work you need another computer as host on the micro USB

Right, I did test with the Pi Zero 2 W connected (and powered) only via the USB data port from another computer. I also booted it up in that configuration, just to be sure.

[joneskoo]

Does the lsmod from Raspbian help identify the dependencies?

root@raspberrypi:/home/pi# lsmod
Module                  Size  Used by
rfcomm                 49152  4 
cmac                   16384  3 
algif_hash             16384  1 
aes_arm                16384  3 
aes_generic            40960  1 aes_arm
ecb                    16384  1 
algif_skcipher         16384  1 
af_alg                 28672  6 algif_hash,algif_skcipher
bnep                   20480  2 
snd_soc_hdmi_codec     20480  1 
hci_uart               40960  1 
btbcm                  16384  1 hci_uart
bluetooth             417792  31 hci_uart,bnep,btbcm,rfcomm
ecdh_generic           16384  2 bluetooth
ecc                    40960  1 ecdh_generic
libaes                 20480  3 bluetooth,aes_arm,aes_generic
8021q                  32768  0 
garp                   16384  1 8021q
stp                    16384  1 garp
llc                    16384  2 garp,stp
brcmfmac              303104  0 
vc4                   270336  3 
brcmutil               20480  1 brcmfmac
cec                    49152  1 vc4
sha256_generic         16384  0
libsha256              20480  1 sha256_generic
drm_kms_helper        233472  2 vc4
cfg80211              770048  1 brcmfmac
snd_soc_core          225280  2 vc4,snd_soc_hdmi_codec
snd_compress           20480  1 snd_soc_core
snd_pcm_dmaengine      16384  1 snd_soc_core
syscopyarea            16384  1 drm_kms_helper
rfkill                 32768  6 bluetooth,cfg80211
sysfillrect            16384  1 drm_kms_helper
raspberrypi_hwmon      16384  0
sysimgblt              16384  1 drm_kms_helper
fb_sys_fops            16384  1 drm_kms_helper
i2c_bcm2835            16384  0
bcm2835_codec          40960  0
v4l2_mem2mem           36864  1 bcm2835_codec
bcm2835_v4l2           45056  0
bcm2835_isp            28672  0
bcm2835_mmal_vchiq     32768  3 bcm2835_isp,bcm2835_codec,bcm2835_v4l2
snd_bcm2835            24576  0
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_dma_contig    20480  2 bcm2835_isp,bcm2835_codec
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2         32768  4 bcm2835_isp,bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common       61440  5 bcm2835_isp,bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
snd_pcm               110592  5 snd_compress,snd_pcm_dmaengine,snd_soc_hdmi_codec,snd_bcm2835,snd_soc_core
snd_timer              32768  1 snd_pcm
snd                    77824  6 snd_compress,snd_soc_hdmi_codec,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
videodev              241664  6 bcm2835_isp,bcm2835_codec,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
mc                     45056  6 bcm2835_isp,bcm2835_codec,videobuf2_common,videodev,v4l2_mem2mem,videobuf2_v4l2
vc_sm_cma              32768  2 bcm2835_isp,bcm2835_mmal_vchiq
fixed                  16384  0
uio_pdrv_genirq        16384  0
uio                    20480  1 uio_pdrv_genirq
i2c_dev                20480  0
g_ether                16384  0
usb_f_rndis            28672  1 g_ether
u_ether                24576  2 g_ether,usb_f_rndis
usb_f_acm              16384  1
u_serial               20480  3 usb_f_acm
g_serial               16384  0
libcomposite           57344  4 g_serial,usb_f_acm,g_ether,usb_f_rndis
dwc2                  163840  0
udc_core               65536  6 usb_f_acm,u_serial,dwc2,u_ether,usb_f_rndis,libcomposite
roles                  16384  1 dwc2
fuse                  118784  3
drm                   512000  4 vc4,drm_kms_helper
drm_panel_orientation_quirks    16384  1 drm
backlight              20480  1 drm
ip_tables              28672  0
x_tables               32768  1 ip_tables
ipv6                  475136  36

[joneskoo]

Dependencies as a graph:

https://go.dev/play/p/r_VGByiQ9Jw

graph TD
    aes_generic --> aes_arm
    af_alg --> algif_hash
    af_alg --> algif_skcipher
    btbcm --> hci_uart
    bluetooth --> hci_uart
    bluetooth --> bnep
    bluetooth --> btbcm
    bluetooth --> rfcomm
    ecdh_generic --> bluetooth
    ecc --> ecdh_generic
    libaes --> bluetooth
    libaes --> aes_arm
    libaes --> aes_generic
    garp --> 8021q
    stp --> garp
    llc --> garp
    llc --> stp
    brcmutil --> brcmfmac
    cec --> vc4
    libsha256 --> sha256_generic
    drm_kms_helper --> vc4
    cfg80211 --> brcmfmac
    snd_soc_core --> vc4
    snd_soc_core --> snd_soc_hdmi_codec
    snd_compress --> snd_soc_core
    snd_pcm_dmaengine --> snd_soc_core
    syscopyarea --> drm_kms_helper
    rfkill --> bluetooth
    rfkill --> cfg80211
    sysfillrect --> drm_kms_helper
    sysimgblt --> drm_kms_helper
    fb_sys_fops --> drm_kms_helper
    v4l2_mem2mem --> bcm2835_codec
    bcm2835_mmal_vchiq --> bcm2835_isp
    bcm2835_mmal_vchiq --> bcm2835_codec
    bcm2835_mmal_vchiq --> bcm2835_v4l2
    videobuf2_vmalloc --> bcm2835_v4l2
    videobuf2_dma_contig --> bcm2835_isp
    videobuf2_dma_contig --> bcm2835_codec
    videobuf2_memops --> videobuf2_dma_contig
    videobuf2_memops --> videobuf2_vmalloc
    videobuf2_v4l2 --> bcm2835_isp
    videobuf2_v4l2 --> bcm2835_codec
    videobuf2_v4l2 --> bcm2835_v4l2
    videobuf2_v4l2 --> v4l2_mem2mem
    videobuf2_common --> bcm2835_isp
    videobuf2_common --> bcm2835_codec
    videobuf2_common --> bcm2835_v4l2
    videobuf2_common --> v4l2_mem2mem
    videobuf2_common --> videobuf2_v4l2
    snd_pcm --> snd_compress
    snd_pcm --> snd_pcm_dmaengine
    snd_pcm --> snd_soc_hdmi_codec
    snd_pcm --> snd_bcm2835
    snd_pcm --> snd_soc_core
    snd_timer --> snd_pcm
    snd --> snd_compress
    snd --> snd_soc_hdmi_codec
    snd --> snd_timer
    snd --> snd_bcm2835
    snd --> snd_soc_core
    snd --> snd_pcm
    videodev --> bcm2835_isp
    videodev --> bcm2835_codec
    videodev --> videobuf2_common
    videodev --> bcm2835_v4l2
    videodev --> v4l2_mem2mem
    videodev --> videobuf2_v4l2
    mc --> bcm2835_isp
    mc --> bcm2835_codec
    mc --> videobuf2_common
    mc --> videodev
    mc --> v4l2_mem2mem
    mc --> videobuf2_v4l2
    vc_sm_cma --> bcm2835_isp
    vc_sm_cma --> bcm2835_mmal_vchiq
    uio --> uio_pdrv_genirq
    usb_f_rndis --> g_ether
    u_ether --> g_ether
    u_ether --> usb_f_rndis
    u_serial --> usb_f_acm
    libcomposite --> g_serial
    libcomposite --> usb_f_acm
    libcomposite --> g_ether
    libcomposite --> usb_f_rndis
    udc_core --> usb_f_acm
    udc_core --> u_serial
    udc_core --> dwc2
    udc_core --> u_ether
    udc_core --> usb_f_rndis
    udc_core --> libcomposite
    roles --> dwc2
    drm --> vc4
    drm --> drm_kms_helper
    drm_panel_orientation_quirks --> drm
    backlight --> drm
    x_tables --> ip_tables

[stapelberg]

Does the lsmod from Raspbian help identify the dependencies?

That output just confirms that g_serial only has libcomposite as a dependency