Build missing network interface support routines

[Memotech-Bill]

As per topic on Raspberry Pi forums, Richard Russel's build of PicoBB is missing network support routines.

>PRINT SYS "net_freeall"
         0

However my build has them:

>PRINT SYS "net_freeall"
 268532725

My build sequence is simply:

cd PicoBB/console/pico_w
make

We need to identify what is different about the way Richard builds the program.

[Memotech-Bill]

The following command will show the total size of the files on the RPi disc before copying into the LFS filesystem:

du -sh build/tree

The LFS filesystem is 1MB (1024KB). A small amount of that space will be used for the directory entries, not sure how to estimate how much.

In terms of redundant files there are a number of cases where I have saved both .bbc and .bas versions of a file.

[Memotech-Bill]

I have now pushed a small change which should halt the make if assembling the LFS filesystem fails.

Even before that, I am not sure why you had to look a long way back to find the error, assembling build/tree and then creating the LFS filesystem are the last steps in the make.

[rtrussell]

I am not sure why you had to look a long way back to find the error

It could be because I did make a couple more times (just in case it was a transient fault, or a circular dependency) and doing that seems to generate quite a lot of output each time which doesn't include re-making the filesystem (no doubt because make determines that it's already been done). So to find the informative line I had to scroll back through two sets of output before getting to the build containing it.

[rtrussell]

In terms of redundant files there are a number of cases where I have saved both .bbc and .bas versions of a file.

Sadly deleting the .bas files hasn't freed up enough space. I guess I am going to have to find an alternative, smaller, demo file for picowav.bbc, and it took me ages to choose the one I'm currently using (prague.wav)!

Incidentally, although the build does now abort on filling the filesystem, it does so in a very 'dirty' way which doesn't look deliberate:

Wrote -28 of 3576 bytes of /mysqldem.bbc!
Traceback (most recent call last):
  File "../../src/lfsutil/pico_examples.py", line 97, in <module>
    Run ()
  File "../../src/lfsutil/pico_examples.py", line 95, in Run
    Build (parser.parse.args ())
  File "../../src/lfsutil/pico_examples.py", line 80, in Build
     err = os.waitstatus_to_exitcode (err)
AttributeError: module 'os' has no attribute 'waitstatus_to_exitcode'

[Memotech-Bill]

Incidentally, although the build does now abort on filling the filesystem, it does so in a very 'dirty' way which doesn't look deliberate

That looks as though you are sunning an old version of Python. The function waitstatus_to_exitcode was apparently added to Python v3.9

I will try and re-code without that.

How far short of space are you? The BBC BASIC interpreter takes somewhat less than 1MB, so it is possible to make the LFS filesystem a bit bigger than 1MB.

[Memotech-Bill]

I have pushed a few changes, including a fix for the Python issue.

It looks as though it should be possible to increase the size of the LFS to 1.25MB. However, that would create an incompatibility between the new executable and old LFS images, as currently the start address of the LFS is hard-coded in the executable. I will have to see whether it is possible to search for the start address of the LFS.

[rtrussell]

How far short of space are you?

I haven't been able to ascertain that accurately, but I don't think very much. However we want to leave enough free space in the filesystem for the end user to save his own work!

I've found a possible alternative demo file for picowav.bbc , it's a 7-second sound clip rather than 10-seconds, so that should hopefully save enough with some left over.

I may not get around to trying it today.

[Memotech-Bill]

With my latest commit, try:

rm filesystem*.lfs
make LFS_ORIG=768 LFS_SIZE=1280

Origin and size are specified in KB.

[rtrussell]

With my latest commit, try:

Now I've reduced the size of the filesystem by replacing the large prague.wav file with a shorter audio clip, is there still a benefit in doing this?

The only residual issue I'm trying to solve (and it's almost certainly 'finger trouble' on my part) is that despite using --recurse-submodules I'm still not picking up the latest source files from the BBCSDL repository. Maybe it's because I've been doing git pull to fetch your recent changes without having to start from scratch. This git stuff confuses me!

[Memotech-Bill]

Now I've reduced the size of the filesystem by replacing the large prague.wav file with a shorter audio clip, is there still a benefit in doing this?

Only to provide the user with more space for their files.

The only residual issue I'm trying to solve (and it's almost certainly 'finger trouble' on my part) is that despite using --recurse-submodules I'm still not picking up the latest source files from the BBCSDL repository. Maybe it's because I've been doing git pull to fetch your recent changes without having to start from scratch. This git stuff confuses me!

I am not a git expert, but I believe a sub-module refers to a specific commit of the parent repository. It is done that way so that any breaking changes to the parent repository does not break any projects using the repository as a sub-module.

I believe that the solution is:

cd PicoBB/BBCSDL
git pull

to pull your latest source files.

Alternately use

make BBCSDL=/path/to/your/source

At some point I need to do a new pull of your source into my repository, and then commit the change so that the sub-module points to your latest source.

[rtrussell]

I believe that the solution is:

cd PicoBB/BBCSDL
git pull

That sounds like a good solution, but unfortunately it doesn't work:

You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

Alternately use

make BBCSDL=/path/to/your/source

I know I can do that*, but it requires the up-to-date BBCSDL files to be on the same machine, which is something I cannot always guarantee (they won't be if I've committed a change to the BBCSDL repository but never built it on this machine).

*Actually it doesn't seem to work. Are you sure it redirects both the /src and the /include directories, because superficially it's not picking up the up-to-date header files.

[Memotech-Bill]

That sounds like a good solution, but unfortunately it doesn't work:

OK, I have pulled your latest repository into mine, then committed and pushed the change. I think that you now need to do:

git pull --recurse-submodules

Alternately use

make BBCSDL=/path/to/your/source

*Actually it doesn't seem to work. Are you sure it redirects both the /src and the /include directories, because superficially it's not picking up the up-to-date header files.

It may not redirect the include files. I will have to check.

[Memotech-Bill]

Alternately use

make BBCSDL=/path/to/your/source

*Actually it doesn't seem to work. Are you sure it redirects both the /src and the /include directories, because superficially it's not picking up the up-to-date header files.

It may not redirect the include files. I will have to check.

My error :(

The command should be:

make BBC_SRC=/path/to/your/source

And yes, it does redirect the include files as well as the C source.

[rtrussell]

OK, I have pulled your latest repository into mine, then committed and pushed the change.

I'm sorry to report that doing this now builds an invalid filesystem_pwc.uf2 (at least, my Pico doesn't accept it):

git pull --recurse-submodules
cd console/pico_w
make clean
make

filesystem_pwc.uf2 is 2,097,152 bytes but when copied to the Pico it eventually times out and the LED doesn't start flashing. On running BBC BASIC the filesystem contains no files, only some empty directories dev/, lib/, tmp/ and user/

[Memotech-Bill]

This has been caused by preparation for supporting the Pico 2.

For the original Pico, there was only one UF2 family ID value (0xe48bff56). For the Pico 2 there are a number of possible ID values, see section 5.5.3 of the RP2350 datasheet.

I have modified uf2merge so that it (optionally) picks up the family ID from the first file to be merged, and applies it to the subsequent files. Hence the bbcbasic+filesystem_*.uf2 file has the correct family ID (matching that of bbcbasic_*.uf2). However, when building filesystem_*.uf2 (which could apply to Pico or Pico 2) it is not obvious which family ID to apply.

The fundamental problem is that (in general) it is not easy (in the Makefile) to go from board name to chip type. It is done in CMake.

I need to give this some more thought.

[Memotech-Bill]

A bit of a cludge: I have added (and pushed) an option to uf2conv to read a reference UF2 file and use the family ID from the reference file as the family ID for the file being created.

Using this, my Makefile now gives filesystem_*.uf2 the same family ID as bbcbasic_*.uf2.

[rtrussell]

I have modified uf2merge so that it (optionally) picks up the family ID from the first file to be merged, and applies it to the subsequent files.

OK. For my release version I don't use your uf2merge, I use mine (written a long time ago) because it creates a significantly smaller file (or at least it used to, things may have changed for the pico_w). So I'll need to make the same modification to that, thanks for the heads-up.

[Memotech-Bill]

OK. For my release version I don't use your uf2merge, I use mine (written a long time ago) because it creates a significantly smaller file (or at least it used to, things may have changed for the pico_w).

That is because my version used to zero fill all the space between the executable and the filesystem. I have revised this recently.

[rtrussell]

That is because my version used to zero fill all the space between the executable and the filesystem.

I guessed that, mine never has. I'll modify my version so it ignores the family ID in the filesystem file altogether.

[rtrussell]

I'll modify my version so it ignores the family ID in the filesystem file altogether.

That seems to be working, both your merged file and mine now load.

Pretty much everything seems to be resolved now. I thought I'd try running mysqldem.bbc as in principle that should run if the socket interface is working, but it reports:

Error -7 connecting to access point at line 93

Is that anything I should worry about?

[Memotech-Bill]

Don't know.

When I try it, I get:

Connecting to MySQL server 'mysql-rfam-public.ebi.ac.uk'... failed to connect, aborting

Do any of the other network programs, such as lanchat work for you?

[rtrussell]

When I try it, I get: Connecting to MySQL server 'mysql-rfam-public.ebi.ac.uk'... failed to connect, aborting

Does the PicoW do DNS lookups, or do you have to supply an IP address? I notice that wifi-cfg.bbc doesn't ask for a DNS server (or even a netmask).

nslookup mysql-rfam-public.ebi.ac.uk
Server:  fritz.box
Address:  fd00::de15:c8ff:fe70:f7e9

Non-authoritative answer:
Name:    mysql-vm-084.ebi.ac.uk
Address:  193.62.194.222
Aliases:  mysql-rfam-public.ebi.ac.uk

Do any of the other network programs, such as lanchat work for you?

lanchat worked fine for me yesterday, that's how I concluded the PicoW had successfully connected to the network, but today I can't get the WiFi to work at all. Everything results in a hard fault, e.g.:

LittleFS image v2.1, Size = 1024KB, Origin = 0x10100000
BBC BASIC for Pico W Console v0.47, Build Oct 19 2024, USB Console, UART Console
, Flash Filesystem, cyw43=background, SDL Sound, /dev/uart, Min Stack, Stack Che
ck 4, RTC
(C) Copyright R. T. Russell, 2024
cyw43 initialisation succeded
>CH."wifi_scan"

R0 = 00000027  R8  = 20041698
R1 = 00000000  R9  = FFFFFFFD
R2 = 00000000  R10 = 00000000
R3 = 2000371C  R11 = 00000000
R4 = FFFFFFD8  R12 = 20005324
R5 = 00000020  SP  = 20041440
R6 = 200052A4  LR  = 1002E221
R7 = 200052A4  PC  = 1001828A
SG = 20028840  PSP = 01000000

Hard fault at line 70
>

I've no idea what has changed.

[Memotech-Bill]

Does the PicoW do DNS lookups, or do you have to supply an IP address? I notice that wifi-cfg.bbc doesn't ask for a DNS server (or even a netmask).

Yes it should do DNS lookups. It relies upon DHCP from the WiFi to provide netmask, gateway and DNS configuration.

[rtrussell]

Yes it should do DNS lookups. It relies upon DHCP from the WiFi to provide netmask, gateway and DNS configuration.

It should find mysql-rfam-public.ebi.ac.uk then, I've never had problems connecting to it from any other client.

Edit: Modifying mysqldem.bbc to use the explicit IP address 193.62.194.222 works about one time out of three here. The other times I get a Hard Fault or a Error -7 connecting to access point at line 93.

I see that wifi_scan.bbc (when it works) lists MAC addresses, which is useful here because we have a mesh, so there are multiple APs with the same SSID. Is there a way of forcing the PicoW to connect to a specific MAC address (hence mesh node)?

[Memotech-Bill]

The current version of wifi_scan is broken. It does not call PROC_initsockets so it does not reserve the space above HIMEM needed for the network routines.

Once you get a hard fault you will need to do a QUIT to re-initialise variables before any networking will work.

PROC_initsockets does too much, since it connects to an access point. I need to add a routine to socklib to just reserve the memory. This is not currently working properly, I need further investigation.

Is there a way of forcing the PicoW to connect to a specific MAC address (hence mesh node)?

PROC_initsockets includes:

SYS "cyw43_arch_wifi_connect_timeout_ms", ssid$, pwd$, &400004, 30000 TO err%

I see that the C SDK also lists:

int cyw43_arch_wifi_connect_bssid_timeout_ms (const char * ssid, const uint8_t * bssid, const char * pw, uint32_t auth, uint32_t timeout)

I assume that the BSSID is the MAC address.

It would be necessary to write some code to detect colons on the SSID, convert the string to bytes, and call the alternate connection routine.

[rtrussell]

The current version of wifi_scan is broken.

Ah. I was about to release the latest build but it sounds like I should hold fire.

I need to add a routine to socklib to just reserve the memory.

These were the routines I originally suggested you use (I think you made a small change because of an off-by-one error when run on the Pico rather than in Windows). Of course the 64-bit pointers aren't necessary on a 32-bit platform:

      PRINT "On entry:"
      PRINT "HIMEM = &"; STR$~HIMEM
      DIM s%% LOCAL -1
      PRINT "Stack pointer = &"; ~s%%
      PROCtest
      PRINT "After PROCtest:"
      PRINT "HIMEM = &"; STR$~HIMEM
      DIM s%% LOCAL -1
      PRINT "Stack pointer = &"; ~s%%
      PROCfree
      PRINT "After PROCfree:"
      PRINT "HIMEM = &"; STR$~HIMEM
      DIM s%% LOCAL -1
      PRINT "Stack pointer = &"; ~s%%
      END

      DEF PROCtest
      LOCAL b%%, d%%, s%%
      DIM s%% LOCAL TRUE, b%% LOCAL 10000, d%% LOCAL TRUE
      WHILE s%% < HIMEM
        ?d%% = ?s%%
        d%% += 1 : s%% += 1
      ENDWHILE
      HIMEM = d%%
      ENDPROC

      DEF PROCfree
      LOCAL b%%, d%%, s%%
      DIM s%% LOCAL TRUE, b%% LOCAL 7, d%% LOCAL TRUE
      b%% = 10000 + s%% - d%% - 4
      s%%!-12 += b%% : d%% = s%% + b%%
      SYS "memmove", d%%, s%%, HIMEM - s%%
      HIMEM = d%% + HIMEM - s%%
      ENDPROC

It would be necessary to write some code to detect colons on the SSID, convert the string to bytes, and call the alternate connection routine.

OK. This is low priority, but could be useful as a future enhancement for those who have mesh systems.

[Memotech-Bill]

The issues with wifi_scan have now been resolved.

It took a while due to tracking down a memory corruption issue that was eventually traced to an out by one error in the sizes of a structure between C & BASIC.

[Memotech-Bill]

Yes it should do DNS lookups. It relies upon DHCP from the WiFi to provide netmask, gateway and DNS configuration.

It should find mysql-rfam-public.ebi.ac.uk then, I've never had problems connecting to it from any other client.

Edit: Modifying mysqldem.bbc to use the explicit IP address 193.62.194.222 works about one time out of three here. The other times I get a Hard Fault or a Error -7 connecting to access point at line 93.

I have fixed an issue with DNS lookup.

I am now failing to connect to the server due to a 10 second timeout.

[rtrussell]

I am now failing to connect to the server due to a 10 second timeout.

When it works here (using the IP address) it gives the expected output:

[rtrussell]

The issues with wifi_scan have now been resolved.

Thank you. I plan to release the latest version; it's still very prone to Hard Faults here, including in the circumstances described in the other issue, but perhaps my Pico W is faulty in some way.

[Memotech-Bill]

It appears that the LWIP DNS resolver is limited in its capability. It often fails to resolve mysql-rfam-public.ebi.ac.uk although it seems reliable at resolving bbc.co.uk or google.co.uk. I have pushed a simple test program console/pico_w/examples/dns_lookup.bas.

Replacing the domain name by IP address in mysqldem.bbc I obtain:

Connecting to MySQL server '193.62.194.222'... connected successfully

Connecting to database 'Rfam'... connected successfully
MySQL version = 5.6.36-log

Fetching database names... DIM space at line 314

Are you running a modified version of the program?

Edit:

After changing line 314 to:

DIM payload&(1024)

I also obtain:

Connecting to MySQL server '193.62.194.222'... connected successfully

Connecting to database 'Rfam'... connected successfully
MySQL version = 5.6.36-log

Fetching database names... 12 database(s) enumerated: 'information_schema' 'Rfam' 'mysql' 'performance_schema' 'rfam_12_2' 'rfam_13_0' 
'rfam_14_0' 'rfam_14_10' 'rfam_14_8' 'rfam_14_9' 'sj_test' 'sys'

Fetching table names... 14 table(s) enumerated: '_annotated_file' '_family_file' '_genome_data' '_lock' '_overlap' '_overlap_membership
' '_post_process' 'alignment_and_tree' 'author' 'clan' 'clan_database_link' 'clan_literature_reference' 'clan_membership' 'database_lin
k'

Querying database 'Rfam' for table 'author'...

Fetching column titles... number of columns = 6
Fetching table data... number of rows read = 14

author_id  name                  last_name             initials    orcid                 synonyms
1          Ames T                Ames                  T                                 
2          Argasinska J          Argasinska            J           0000-0003-2678-2824   
3          Bachellerie JP        Bachellerie           JP                                
4          Barquist LE           Barquist              LE          0000-0003-4732-2667   
5          Barrick JE            Barrick               JE                                
6          Bateman A             Bateman               A           0000-0002-6982-4660   
7          Boese B               Boese                 B                                 
8          Boursnell C           Boursnell             C           0000-0002-3494-4451   
9          Breaker RR            Breaker               RR          0000-0002-2165-536X   
10         Brown C               Brown                 C                                 
11         Burge SW              Burge                 SW          0000-0002-2506-927X   Burge S
12         Cartinhour SW         Cartinhour            SW                                
13         Chen A                Chen                  A                                 
14         Collins JA            Collins               JA                                

Transaction completed successfully

Five runs in succession with no failures.

[rtrussell]

Are you running a modified version of the program?

As I noted in my previous post "When it works here (using the IP address) it gives the expected output"; using the IP address rather than the URL was the only change I made.

After changing line 314 to:

DIM payload&(1024)

What was the purpose of that change? I've run the program hundreds of times on different platforms (including the Pico W) and the 256-byte array has always been sufficient. I've never seen an error -4 (payload too large) reported.

[Memotech-Bill]

The version of the program I had was originally:

DIM payload&(65536)

[rtrussell]

The version of the program I had was originally:

DIM payload&(65536)

Where did that come from? mysqllib.bbc has always had DIM payload&(255) right back to when it was first written in 2022. In any case I would never write 65536 since the array is one element longer than the specified subscript, so if written by me it would be payload&(65535) or more likely payload&(&FFFF).

[Memotech-Bill]

So I mis-remembered when typing the comment, if is 65535 not 65536.

65535 is what is currently in BBCSDL/lib/mysqllib.bbc in your repository.

[rtrussell]

65535 is what is currently in BBCSDL/lib/mysqllib.bbc in your repository.

There is some confusion here. In mysqllib.bbc there are two separate lines in which a LOCAL payload&() array is declared: in FN_sqlColumns there is a DIM payload&(255) and in FN_sqlFetch there is a DIM payload&(65535). The second one needs to be big, because one has no idea how big a database element might be and it therefore shouldn't limit the size any more than necessary. The 65536-byte array still fits when mysqldem is run on the Pico W.

The earlier, smaller, array is used when reading the column headings and therefore doesn't need to be so big. That's the one I assumed you were talking about, because your modification to payload&(1024) could easily be too small for a database element in the general case.

I see no reason for changing mysqllib.bbc from the version which is supplied with BBC BASIC for SDL 2.0 and all the other BBC BASIC Console Mode editions. Indeed, none of the libraries supplied with the Pico edition is different from those supplied with the other Console Mode editions (apart from socklib.bbc of course) because my build process copies the master libraries on top of any in your repository.

Incidentally I also delete the .bas files which I pull from your repository because they are Linux-style text files with LF line endings, which don't display sensibly with TYPE.

[rtrussell]

Have you accidentally left some debugging code in socklib (or elsewhere)? I'm seeing this unwanted output when I connect:

H% = DEC23EC1
ipaddr = DEC23EC1 ^ipaddr = 2001E974

[Memotech-Bill]

Apologies, yes.

Fixed now.