picow_freertos_iperf_server_sys: LED stops blinking when iperf is actualy being run

[kripton]

Since I wanted to learn FreeRTOS and lwIP in combination on the RP2040, I compiled the "Pico-W FreeRTOS iperf server" example.

The access point I used is another Pico-W running the "Pico-W Access Point" example without any modifications.

I also connect my Linux laptop to the same access point and all devices get their IPv4 addresses via DHCP server integrated with the access point example. No other devices except the Pico-W and the laptop are connect to the access point.

When I run the iperf server in "NO_SYS" mode, everything is fine, iperf reports around 12MBit/s. I can happily repeat the test, ping the device in parallel and even run nmap against it. The LED just keeps on blinking.

With the "sys" example however, the LED starts blinking as soon the board has connected to it's AP and is ready to run the tests. However, when I start the iperf client on my laptop, the LED will stop blinking eventually. The Pico-W running the iperf sys example is still pingable and futher iperf runs are possible. However, the LED will just stay off.

I haven't yet digged deep into it. It might be a bug in FreeRTOS or its rp2040 "port" that makes it stop scheduling tasks properly, it could be a bug in the cyw driver or firmware that either stops driving the LED or the SPI channel is blocked by the WiFi traffic and as such the SPI traffic regarding the LED gets lost. It could also be lwIP or the cyw driver corrupting some parts of memory and as such the LED blinking tasks is no longer functioning ...

Again, no issues with the NO_SYS variant of the freertos iperf server example.

[peterharperuk]

I don't think this is a bug. If I remember correctly, the led is displayed in a lower priority thread which won't get to run while iperf is running. I'll have to check the source.

[kripton]

Even if so, the LED should resume blinking when the iperf is stopped. This is not the case. If the LED stops blinking, it will stay off until board reset. Even stopping iperf or disconnecting the laptop from the WiFi AP doesn't make the LED blink again.

[kripton]

Just did another small test: With "SYS" the LED just stops blinking and never returns. With "NO_SYS", the LED blinks slower while iperf is running (understandable) but returns to the normal/idle blinking frequency when the iperf client server is stopped.

I added the pico_stdio_usb library to the example and there's nothing suspicious in the logs in both cases.

[peterharperuk]

It's possible it's running out of stack space

[peterharperuk]

It seems to work ok for me. The led doesn't ever stop blinking.

the LED will stop blinking eventually

How long do you have to wait for the led to stop blinking?

[kripton]

Oh, this not being reproducable was unexpected to me. What are you using as the WiFi AP? Also the "access_point" example or something else?

The LED blinking doesn't stop right away but after 20 seconds it stopped almost every time. I will try another pico_w board (and maybe another WiFi AP) to see if this is a process-temp-age thing but I would be surprised. Just as I am thinking of it, it could also be a toolchain issue. I'm not using the binary gcc distribution recommended, I'm on gcc version 12.2.0 (Gentoo 12.2.0 p1) here. Let me try to set up an "official" toolchain in a container or on a Pi. UF2 file attached, maybe you could try this on yours? picow_freertos_iperf_server.zip

[kripton]

This is indeed toolchain-related. It works (= LED does NOT stop blinking) when using gcc 9.x that comes with this docker image. It breaks when I use gcc 11 or gcc 12 on two machines I tried here. Sorry for the noise :/ Closing this as this seems to be an issue local to my machines. Feel free to re-open if you think it might be a more general problem with gcc versions newer than 9.x.x and want to track it.

[kripton]

Oh, funny side story: It took me quite some tries to verify this. I loaded many UF2s to the board and the LED was just "ON" all the time. At some point I figured out that I loaded the code onto a non-WiFi Pico instead of a Pico-W :D

[peterharperuk]

Interesting. Thanks for persisting and finding the cause. By coincidence I've just had to recreate my build environment which required me to reinstall everything. Let me try and reproduce it again.

[kripton]

Sorry, I'm re-opening this. I did some more tests using different toolchains (using also the official, binary gcc distribution from arm.com on my Gentoo machine, recompiling newlib for the Cortex-M0+ several times using different compilers) and even with the "most official ones", it's still failing: I do have a RPi 4 CM running "Raspbian GNU/Linux 11 (bullseye)", "Linux raspberrypi 5.15.32-v7l+" and I followed the official documentation on how to install the SDK. Host gcc:

pi@raspberrypi:~ $ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/10/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 10.2.1-6+rpi1' --with-bugurl=file:///usr/share/doc/gcc-10/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-10 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.1 20210110 (Raspbian 10.2.1-6+rpi1)

Target gcc:

pi@raspberrypi:~ $ arm-none-eabi-gcc -v
Using built-in specs.
COLLECT_GCC=arm-none-eabi-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-none-eabi/8.3.1/lto-wrapper
Target: arm-none-eabi
Configured with: ../configure --build=arm-linux-gnueabihf --prefix=/usr --includedir='/usr/lib/include' --mandir='/usr/lib/share/man' --infodir='/usr/lib/share/info' --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir='/usr/lib/lib/arm-linux-gnueabihf' --libexecdir='/usr/lib/lib/arm-linux-gnueabihf' --disable-maintainer-mode --disable-dependency-tracking --mandir=/usr/share/man --enable-languages=c,c++,lto --enable-multilib --disable-decimal-float --disable-libffi --disable-libgomp --disable-libmudflap --disable-libquadmath --disable-libssp --disable-libstdcxx-pch --disable-nls --disable-shared --disable-threads --enable-tls --build=arm-linux-gnueabihf --target=arm-none-eabi --with-system-zlib --with-gnu-as --with-gnu-ld --with-pkgversion=15:8-2019-q3-1+b1 --without-included-gettext --prefix=/usr/lib --infodir=/usr/share/doc/gcc-arm-none-eabi/info --htmldir=/usr/share/doc/gcc-arm-none-eabi/html --pdfdir=/usr/share/doc/gcc-arm-none-eabi/pdf --bindir=/usr/bin --libexecdir=/usr/lib --libdir=/usr/lib --disable-libstdc++-v3 --host=arm-linux-gnueabihf --with-headers=no --without-newlib --with-multilib-list=rmprofile CFLAGS='-g -O2 -fdebug-prefix-map=/build/gcc-arm-none-eabi-3BFy9A/gcc-arm-none-eabi-8-2019-q3=. -fstack-protector-strong' CPPFLAGS='-Wdate-time -D_FORTIFY_SOURCE=2' CXXFLAGS='-g -O2 -fdebug-prefix-map=/build/gcc-arm-none-eabi-3BFy9A/gcc-arm-none-eabi-8-2019-q3=. -fstack-protector-strong' FCFLAGS='-g -O2 -fdebug-prefix-map=/build/gcc-arm-none-eabi-3BFy9A/gcc-arm-none-eabi-8-2019-q3=. -fstack-protector-strong' FFLAGS='-g -O2 -fdebug-prefix-map=/build/gcc-arm-none-eabi-3BFy9A/gcc-arm-none-eabi-8-2019-q3=. -fstack-protector-strong' GCJFLAGS='-g -O2 -fdebug-prefix-map=/build/gcc-arm-none-eabi-3BFy9A/gcc-arm-none-eabi-8-2019-q3=. -fstack-protector-strong' LDFLAGS=-Wl,-z,relro OBJCFLAGS='-g -O2 -fdebug-prefix-map=/build/gcc-arm-none-eabi-3BFy9A/gcc-arm-none-eabi-8-2019-q3=. -fstack-protector-strong' OBJCXXFLAGS='-g -O2 -fdebug-prefix-map=/build/gcc-arm-none-eabi-3BFy9A/gcc-arm-none-eabi-8-2019-q3=. -fstack-protector-strong' INHIBIT_LIBC_CFLAGS=-DUSE_TM_CLONE_REGISTRY=0 AR_FOR_TARGET=arm-none-eabi-ar AS_FOR_TARGET=arm-none-eabi-as LD_FOR_TARGET=arm-none-eabi-ld NM_FOR_TARGET=arm-none-eabi-nm OBJDUMP_FOR_TARGET=arm-none-eabi-objdump RANLIB_FOR_TARGET=arm-none-eabi-ranlib READELF_FOR_TARGET=arm-none-eabi-readelf STRIP_FOR_TARGET=arm-none-eabi-strip
Thread model: single
gcc version 8.3.1 20190703 (release) [gcc-8-branch revision 273027] (15:8-2019-q3-1+b1)

Newlib: libnewlib-arm-none-eabi/stable,now 3.3.0-1 all [installed,automatic]

I cloned fresh copies of the pico-sdk, pico-examples and FreeRTOS-Kernel, compiled the example on the CM4, copied over the resulting UF2 file to my laptop to load it onto the Pico-W. The LED blinks for some more seconds but after about 15 seconds, it stops as well.

To rule out process parameters, I tried using two different Pico-Ws running the iperf program but I didn't yet change the one acting as the access point. The command I ran on my laptop (iperf client) is: while [ true ]; do iperf -c 192.168.4.16 -d ; done;

Any ideas what I could do to find the cause of the problem?

[kripton]

Might be related to #302

[peterharperuk]

I couldn't get picow_freertos_iperf_server_sys to work on the (soon to be old) master branch. I had to use develop. It could be the thread being starved of CPU I guess - assuming that the iperf to the Pico is still working. I can't reproduce it but my wifi here is a bit slow. We could try adding a new fangled async callback if thread starvation is your problem - although it seems a bit unlikely to me.