Build missing network interface support routines

[Memotech-Bill]

As per topic on Raspberry Pi forums, Richard Russel's build of PicoBB is missing network support routines.

>PRINT SYS "net_freeall"
         0

However my build has them:

>PRINT SYS "net_freeall"
 268532725

My build sequence is simply:

cd PicoBB/console/pico_w
make

We need to identify what is different about the way Richard builds the program.

[Memotech-Bill]

Looking at the signon messages, Richard's build has:

BBC BASIC for Pico Console v0.46, Build Jul 2 2024, USB Console, UART Console, Flash Filesystem, cyw43=gpio, SDL Sound, /dev/uart, Stack Check 4, RTC

Whereas my build has:

BBC BASIC for Pico W Console v0.46, Build Jul 3 2024, USB Console, UART Console, Flash Filesystem, cyw43=background, SDL Sound, /dev/uart, Min Stack, Stack Check 4, RTC

From this, I guess that Richard is building from the console/pico folder not the console/pico_w folder. Specifying BOARD=pico+w is not sufficient to enable WiFi support.

[rtrussell]

From this, I guess that Richard is building from the console/pico folder not the console/pico_w folder.

That's correct. My understanding has always been that the various build options are achieved by specifying them in the make command line. Indeed, referring back to the issue where this was previously discussed, these were the instructions you gave for building a Pico W version:

Then build the experimental version:

cd console/pico
make BOARD=pico_w CYW43=Y

Apart from the change to BOARD=pico+w that's what I am still doing.

[Memotech-Bill]

That's correct. My understanding has always been that the various build options are achieved by specifying them in the make command line. Indeed, referring back to the issue where this was previously discussed, these were the instructions you gave for building a Pico W version:

That is correct in principle. Doing make in the different folders just selects different default values.

However, as noted later in that thread:

* The CYW43 switch now supports a number of options to select the different SDK libraries:

  * CYW43=GPIO - Only support the GPIO functions (the onboard LED and power options). This version has the same amount of BASIC memory as the standard Pico build. It could become the default build, replacing the version for the standard Pico.
  * CYW43=POLL Use the cyw43_arch_poll library. Note that this does not eliminate callback functions. The callbacks are triggered when you call cyw43_arch_poll(), rather thai in response to a timer interrupt.
  * CYW43=BACKGROUND. Use my fixed version of the cyw43_arch_threadsafe_background library. I believe that this is the most convenient version to use with BASIC.

Looking at the CMakeLists.txt file, CYW43=Y should be equivalent to CYW43=BACKGROUND, so I am not quite sure why your build does not include network support. Are you still doing git checkout lwip_raw_tcp? That was an experimental branch for initial development of networking, and does not contain all the latest revisions.

Building in the console/pico_w folder was documented at the end of that thread.

[rtrussell]

However, as noted later in that thread:

Hmm, "later in the thread" is only useful if there was a reason for me to read it, and there probably wasn't!

Are you still doing git checkout lwip_raw_tcp?

I don't think so..

Most of my BBCSDL builds are as 'idiot proof' as I can make them (using a bog-standard makefile). Unfortunately there are two that aren't - Android and Pico - because in each case I'm reliant on a build process that I'm not familiar with: Android Studio in the first case and CMake in the second.

So in those cases I just blindly follow instructions that I've written down from a previous occasion and seem to work. I don't know why I'm doing what I'm doing, and unless some error message or obvious malfunction results, I am none the wiser.

To cut to the chase, what should I be doing to build the most 'universal' version, with support for the Pico W but with the minimum degree of impact (e.g. in respect of available memory) when it's running on a non-W Pico?

[Memotech-Bill]

To cut to the chase, what should I be doing to build the most 'universal' version, with support for the Pico W but with the minimum degree of impact (e.g. in respect of available memory) when it's running on a non-W Pico?

That depends upon whether or not you want network support. For a version that will run on either Pico or Pico W, but without network support:

cd PicoBB
git pull --recurse-submodules
git checkout master
cd console/pico
make BBC_SRC=<path to your source>

For a version that will run on both Pico and Pico W, and has network support on the Pico W, at the cost of reduced memory available to BASIC on either host, replace cd console/pico with cd console/pico_w.

A few words of explanation:

git pull will download from GitHub any changes I have made to my repository. The --recurse-submodules means that any changes to your repository, or the LittleFS repository, will also be downloaded.

get checkout master ensures that you are on the master branch. You probably will be anyway, this is just a precaution.

make builds the program. The option BBC_SRC=<path to your source> is only necessary if you have a local copy of your source that contains changes that you have not yet committed to GitHub.

As a check, after building the program, you can use the command:

picotool info <filename.uf2>

to show the signon message without having to load the program into a Pico. The significant part is the cyw43=... string. cyw43=gpio indicates support for the Pico W LED, while cyw43=background indicates network support (as well as the Pico W LED).

[rtrussell]

That depends upon whether or not you want network support.

I did say 'universal' version, so naturally that means it needs to support the key feature of the Pico W, i.e. WiFi (which I assume is what the W stands for). Other differences between the non-W and W versions are largely incidental as far as I'm concerned, and I would expect that they can be handled within, for example, a GPIO library.

How far did you get in making the memory costs of networking at least partly conditional on actually using the socklib library? The sort of thing I would expect is that calling PROC_initsockets would rejig the memory map, moving HIMEM down to make space for the networking buffers, before enabling the 'background' functionality.

Ideally (but less important, because one can always restart the Pico) calling PROC_exitsockets should disable the background functionality and then restore the memory map. But if that's not practical or too difficult so be it.

cd PicoBB git pull --recurse-submodules git checkout master cd console/pico make BBC_SRC=

That's similar to what I do, except that I think I use git clone rather than git pull, I don't know if that is significant. I don't use the BBC_SRC= option because at one point it didn't work properly, so my build includes a step in which I replace the source files in your cloned repository with mine. It may well be that I could use that option now.

I also explicitly replace the example programs and libraries with my versions, because commonly I will have made recent changes that aren't reflected in your repository.

[Memotech-Bill]

That depends upon whether or not you want network support.

I did say 'universal' version, so naturally that means it needs to support the key feature of the Pico W, i.e. WiFi (which I assume is what the W stands for). Other differences between the non-W and W versions are largely incidental as far as I'm concerned, and I would expect that they can be handled within, for example, a GPIO library.

The LED on the Pico W is controlled by a GPIO on the CYW43 chip, not a GPIO on the RP2040. It requires a different set of SDK routines to control this. If these are not present, no BBC BASIC library could control the Pico W LED.

How far did you get in making the memory costs of networking at least partly conditional on actually using the socklib library? The sort of thing I would expect is that calling PROC_initsockets would rejig the memory map, moving HIMEM down to make space for the networking buffers, before enabling the 'background' functionality.

All such memory allocation is deep within the lwip library. It is not obvious how one would donate some memory from that available to BBC BASIC.

cd PicoBB git pull --recurse-submodules git checkout master cd console/pico make BBC_SRC=

That's similar to what I do, except that I think I use git clone rather than git pull, I don't know if that is significant. I don't use the BBC_SRC= option because at one point it didn't work properly, so my build includes a step in which I replace the source files in your cloned repository with mine. It may well be that I could use that option now.

git clone downloads a copy of the entire repository, whereas git pull only downloads any changes sice the previous clone or pull.

[rtrussell]

The LED on the Pico W is controlled by a GPIO on the CYW43 chip, not a GPIO on the RP2040. It requires a different set of SDK routines to control this. If these are not present, no BBC BASIC library could control the Pico W LED.

Is that not very similar to the difference between the GPIO on the RPi 1-4 and the GPIO on the RPi 5? On the earlier models the GPIO is driven by the main SoC chip, but on the RPi 5 it's driven by a separate chip which communicates with the SoC using the I²C bus. The software interfaces are completely different, but I don't need to build different versions of BBC BASIC for the different models, it's entirely handled within the gpiolib.bbc library.

If you're saying that it's impossible for a single UF2 to be able to drive the GPIO on both the original Pico and the Pico W, I don't understand why that would be the case. Surely a UF2 could contain the drivers for both and a (notional) GPIO library use whichever interface is appropriate for the model on which it is running, similar to how it works on the Raspberry Pi itself?

If it really is impossible, it makes the whole idea of a 'universal' UF2 moot.

All such memory allocation is deep within the lwip library. It is not obvious how one would donate some memory from that available to BBC BASIC.

I thought that the CPU in the Pico was a relatively simple 'microcontroller' type device, lacking the kind of hardware memory-management which could enforce such partitioning. In its absence, I don't see how anything in the system could 'know' that BASIC is sharing the 'network' memory, so long as any background processes also accessing that memory are disabled.

If you examine the memory map of the WiFi-enabled version, is the memory allocated to networking contiguous with the memory that BASIC uses? If it is, or it can be persuaded to be (e.g. by controlling the order in which various modules are initialised) can't you disable the background processes and simply allow BASIC to extend its memory into that region. How could anything know you'd done that?

[Memotech-Bill]

The LED on the Pico W is controlled by a GPIO on the CYW43 chip, not a GPIO on the RP2040. It requires a different set of SDK routines to control this. If these are not present, no BBC BASIC library could control the Pico W LED.

Is that not very similar to the difference between the GPIO on the RPi 1-4 and the GPIO on the RPi 5? On the earlier models the GPIO is driven by the main SoC chip, but on the RPi 5 it's driven by a separate chip which communicates with the SoC using the I²C bus. The software interfaces are completely different, but I don't need to build different versions of BBC BASIC for the different models, it's entirely handled within the gpiolib.bbc library.

If you're saying that it's impossible for a single UF2 to be able to drive the GPIO on both the original Pico and the Pico W, I don't understand why that would be the case. Surely a UF2 could contain the drivers for both and a (notional) GPIO library use whichever interface is appropriate for the model on which it is running, similar to how it works on the Raspberry Pi itself?

I think we are talking at cross purposes. On the Pico, the LED is controlled by one of the RP2040 GPIO's. On the Pico W, that GPIO is repurposed to control the CYW43 chip. The LED is then controlled by one of the extra GPIOs on the CYW43 chip, which do not exist on the Pico. The option CYW43=GPIO (or CYW43=BACKGROUND) selects including the drivers for the CYW43 GPIO, necessary to control these additional GPIO, in order to provide a "Universal" UF2.

All such memory allocation is deep within the lwip library. It is not obvious how one would donate some memory from that available to BBC BASIC.

I thought that the CPU in the Pico was a relatively simple 'microcontroller' type device, lacking the kind of hardware memory-management which could enforce such partitioning. In its absence, I don't see how anything in the system could 'know' that BASIC is sharing the 'network' memory, so long as any background processes also accessing that memory are disabled.

If you examine the memory map of the WiFi-enabled version, is the memory allocated to networking contiguous with the memory that BASIC uses? If it is, or it can be persuaded to be (e.g. by controlling the order in which various modules are initialised) can't you disable the background processes and simply allow BASIC to extend its memory into that region. How could anything know you'd done that?

Moving HIMEM down makes space available right in the middle of the RAM used by BASIC, between the program and CPU stack.

I am not sure without looking whether the lwip network buffer allocation is static or dynamic.

If it is static, then the compiler will locate the buffers within the .bss region. Even if it could be arranged that the buffers are at the end of the region, and that this region was adjacent to the BASIC region, then in order to move this memory into or out of that available to BASIC, it would be necessary to dynamically adjust PAGE, not HIMEM. I don't think that can be done while a program is running.

If they are dynamically allocated, then they will be on the C heap. Either the heap is located below BASIC RAM, in which case the same issue as for static buffers arises, or the entire heap is located at HIMEM. Even then, moving HIMEM down would introduce space below the existing heap, but the heap grows upwards, not down. Also, the socklib library would be sitting between the original C heap and the freed space.

The only way I can see to make what you suggest work would be to modify the lwip code to use some form of custom memory allocation for the network buffers, making use of space either freed at HIMEM or perhaps allocated as BASIC variables. I am not currently inclined to undertake that.

[Memotech-Bill]

The current RAM memory map for the console/pico_w build is:

• RAM(rwx) : ORIGIN = 0x20000000, LENGTH = 72k
  • .data
  • .bss
  • .heap
• SCRATCH_Y(rwx) : ORIGIN = 0x20012000, LENGTH = 190k
  • Bottom of BASIC RAM
    • Directory paths
    • Command line
  • PAGE
    • BASIC program
  • LOMEM
    • BASIC heap
    • BASIC stack
  • HIGHMEM
    • BASIC libraries
    • Core 0 CPU stack
• SCRATCH_X(rwx) : ORIGIN = 0x20041800, LENGTH = 2k
  • Core 1 CPU stack

[rtrussell]

I think we are talking at cross purposes.

Let me try to clarify things with a couple of simple questions:

1. Is it possible to create a 'universal' UF2 which contains code for driving both the Pico W GPIO (including the LED) and the non-W Pico GPIO (selected at runtime if necessary)?
2. If that is possible, why make the inclusion of the Pico W driver code controllable by the CYW43=GPIO switch? Surely it's easier and less confusing to include it in every build as standard, rather than making it optional.

The only way I can see to make what you suggest work would be to modify the lwip code to use some form of custom memory allocation for the network buffers,

That would be the best way, certainly; other approaches are likely to involve some nasty assembly-language tricks to move the CPU's stack around without it noticing!

Obviously I've not seen the code itself, but I'm surprised that you don't think it's worth attempting. If we are still talking in terns of 40K of memory, which I think is what you originally suggested, that's a large amount to lose 'permanently' when it's only actually needed whilst socklib is in use.

[Memotech-Bill]

1. Is it possible to create a 'universal' UF2 which contains code for driving both the Pico W GPIO (including the LED) and the non-W Pico GPIO (selected at runtime if necessary)?

Yes

2. If that is possible, why make the inclusion of the Pico W driver code controllable by the CYW43=GPIO switch? Surely it's easier and less confusing to include it in every build as standard, rather than making it optional.

The makefiles specify CYW43=GPIO or CYW43=BACKGROUND (which includes GPIO) by default. It is necessary to explicitly specify CYW43=NONE to omit the driver.

Obviously I've not seen the code itself, but I'm surprised that you don't think it's worth attempting. If we are still talking in terns of 40K of memory, which I think is what you originally suggested, that's a large amount to lose 'permanently' when it's only actually needed whilst socklib is in use.

lwip is a large and complex library. It will take some research to even work out what and how memory is allocated. Depending on how the code is structured it may be necessary to fork the entire library in order to change the memory allocation, with subsequent support issues.

Also it is not clear that the network buffers are the main issue. It may be that much of the memory loss is due to the background processing of the CYW43 full driver.

I will do a bit more research.

[rtrussell]

It is necessary to explicitly specify CYW43=NONE to omit the driver.

OK. But - just so I'm absolutely clear - is that only if built in the console/pico_w directory or is it also included by default in the console/pico directory? I still haven't entirely got my head around the reason for having two different directories rather than a simple build option to enable networking, like all the other options.

lwip is a large and complex library. It will take some research to even work out what and how memory is allocated.

I don't doubt it, but even a complex library may well allocate the majority of its static and dynamic buffers quite near the start, and thereafter only refer to them symbolically (e.g. as a pointer), so it might not be complicated.

Also it is not clear that the network buffers are the main issue. It may be that much of the memory loss is due to the background processing of the CYW43 full driver.

Indeed, but if the background processing is completely disabled except for when socklib is active, they can hopefully all be lumped together.

[Memotech-Bill]

It is necessary to explicitly specify CYW43=NONE to omit the driver.

OK. But - just so I'm absolutely clear - is that only if built in the console/pico_w directory or is it also included by default in the console/pico directory? I still haven't entirely got my head around the reason for having two different directories rather than a simple build option to enable networking, like all the other options.

I currently maintain four builds:

• Console, No Network
• Console, With Network
• GUI, No Network
• GUI, With Network

For me it is most convenient to have separate folders for these. I simply have to do make without any parameters in each of console/pico, console/pico_w, bin/pico and bin/pico_w.

The "No Network" builds are retained because they do have more BASIC memory available than the "With Network" builds.

If you look at the Makefile in each of these folders, they are quite simple, they simply set the default value for each of the required parameters and then invoke src/pico/Makefile.

[rtrussell]

I currently maintain four builds:

Fair enough, but I only distribute a single 'as universal as possible' build for the Pico (just as I do for all the other platforms) and it's the build most people are going to come across - it seems to be what Paul Porcelijn is using.

The "No Network" builds are retained because they do have more BASIC memory available than the "With Network" builds.

Which of course is exactly what I am trying to avoid. It's not really viable to have to load a different UF2 into the Pico in order to run a BASIC program that uses network access and then revert back to the original UF2 in order to run a program that has a large memory usage.

In an ideal world it should be possible to write, for example, an application which initially accesses the network in order to download some data files and then performs some processing on that data which needs a lot of memory.

[Memotech-Bill]

I currently maintain four builds:

Fair enough, but I only distribute a single 'as universal as possible' build for the Pico (just as I do for all the other platforms) and it's the build most people are going to come across - it seems to be what Paul Porcelijn is using.

Which is currently the build in the console/pico_w folder. Although I have had support requests for the other builds.

The "No Network" builds are retained because they do have more BASIC memory available than the "With Network" builds.

Which of course is exactly what I am trying to avoid. It's not really viable to have to load a different UF2 into the Pico in order to run a BASIC program that uses network access and then revert back to the original UF2 in order to run a program that has a large memory usage.

I have identified in the memory map a fairly large static allocation associated with network buffers:

 COMMON         0x200059a8     0x98c7 CMakeFiles/bbcbasic.dir/home/pi/pico/pico-sdk/lib/lwip/src/core/memp.c.obj
                0x200059a8                memp_memory_RAW_PCB_base
                0x20005a1c                memp_memory_TCP_SEG_base
                0x20005c20                memp_memory_PBUF_POOL_base
                0x2000ebc4                memp_memory_PBUF_base
                0x2000ecc8                memp_memory_TCP_PCB_LISTEN_base
                0x2000edac                memp_memory_REASSDATA_base
                0x2000ee50                memp_memory_UDP_PCB_base
                0x2000eed4                memp_memory_TCP_PCB_base
                0x2000f20c                memp_memory_SYS_TIMEOUT_base

At present I have not worked out how to replace this.

Even if I do manage to modify the lwip code it is not clear where within the BASIC memory I can allocate storage,

• socklib subroutines cannot change HIMEM. The library is on one side, and the BASIC stack on the other.
• It might be possible to allocate above the loaded libraries. But there is not much space there unless HIMEM is lowered before loading the socklib library.
• The other option is to allocate space on the BASIC heap. Will such allocations get mistaken as BASIC variables? Which commands removes data from the heap? CLEAR, NEW, any others?

[rtrussell]

• socklib subroutines cannot change HIMEM.

Subroutines can change HIMEM but what they can't (easily and safely) do is change BASIC's stack pointer whilst it is in use!

But the key thing is easily and safely, libraries are allowed to (and the ones I write often do) break the normal rules of good coding practice, because they are 'black boxes' that end users shouldn't be fiddling with.

Here's a procedure that does succeed in lowering HIMEM, and BASIC's stack pointer, to free up some memory (here 10,000 bytes) which could be used for networking buffers:

      DEF PROCtest
      LOCAL b%%, d%%, s%%
      DIM s%% LOCAL TRUE, b%% LOCAL 10000, d%% LOCAL TRUE
      WHILE s%% < HIMEM
        ?d%% = ?s%%
        d%% += 1 : s%% += 1
      ENDWHILE
      HIMEM = d%%
      ENDPROC

Which commands removes data from the heap? CLEAR, NEW, any others?

CHAIN.

[Memotech-Bill]

Here's a procedure that does succeed in lowering HIMEM, and BASIC's stack pointer, to free up some memory (here 10,000 bytes) which could be used for networking buffers:

And suppose some intermediate subroutine has taken the address of a LOCAL variable?

[rtrussell]

And suppose some intermediate subroutine has taken the address of a LOCAL variable?

Give me an example, I can't see how it could fail so long as that code is the first thing in the procedure.

[rtrussell]

It could be useful to return the original value of HIMEM to the caller, for example so you can restore it later:

      DEF PROCtest(RETURN s%%)
      LOCAL b%%, d%%
      DIM s%% LOCAL TRUE, b%% LOCAL 10000, d%% LOCAL TRUE
      WHILE s%% < HIMEM
        ?d%% = ?s%%
        d%% += 1 : s%% += 1
      ENDWHILE
      HIMEM = d%%
      ENDPROC

In case it isn't obvious, the reserved memory for the networking buffer is at (the new value of) HIMEM, not at b%%.