Enhancement suggestion: make LFS2 available

[rkompass]

Hello @nspsck,

thanks for making these binaries available. Saves me some work :-) . Also it's nice to see the larger flash size of 155K. Only think I'm missing now is LFS, as it better uses this flash and includes some form of wear leveling.

Inclusion of LFS2 is just an option. If you don't mind: It would be beneficial for me and perhaps others to just have it.

[nspsck]

Hi,

It's done. The firmware now supports Lfs2 out of the box. But it still uses Fat as default, so you can easily verify that the device is functioning properly.

Also it's nice to see the larger flash size of 155K.

Well, there is a draw back: The size of your user c modules is going to suffer, but since not so many user c modules out there supports the stm32 port, I made the decision to give the user more flash spaces.

To use Lfs2 you simply do the following:

import os, pyb
os.umount('/flash')
os.VfsLfs2.mkfs(pyb.Flash(start=0))
os.mount(pyb.Flash(start=0), '/flash')
os.chdir('/flash')

[davefes]

+1, thanks for that addition.

[rkompass]

Thanks @nspsck from me too,

also I noted that I have an info now in Thonny (omg - still using this) in the connect info line about the "Blackpill". Very nice. Two years ago I soldered a bunch of Blackpills with 16MB flash, and capacitors, of course. If you are in the flow and don't mind me asking: Could you just build the binaries for those - it would just save me time. I'm making progress, but am also struggling with the Lora driver, as you noted - nested interrupts is the problem there. So every chance of avoiding another diversion is a help. (Old man can think well, but suffers more from fragmented attention, it seems). I'm perfectly o.k. of course if you reply: Do it yourself :-).

And I'm wondering why the Blackpill is not in the automatic Build/Download pipeline, at least the version without extra flash, as it is quite common and usable now.

[nspsck]

Hi,

Sure! It's now uploaded under firmware/16MB_Version, I do not have a board to test it's behavior but I hope it works just fine on your board!

And I'm wondering why the Blackpill is not in the automatic Build/Download pipeline, at least the version without extra flash, as it is quite common and usable now.

Ugh.. Me neither..

[rkompass]

Thanks a lot!

[nspsck]

@rkompass @davefes I recently just found out, that you can not fully utilize all these 155k/176k(if you were using lfs2) flash memory and this configuration leads into a file corruption on lfs2.

The reason for that is mpy uses a FS_CACHE to cache the content before writing to the flash memory, mpy can not perform a write correctly to a sector with it's used area greater than FS_CACHE. Since the sectors on the STM32F4 series are usually bigger than that (only 0, 1, 2, 3 are 16K), the write will fail. So I am returning back to the initial flash definition.

If there were any trouble I've caused, please accept my sincere apologies!

[davefes]

Not a problem thanks for the update. Does the Pyboard use LittleFS?

[nspsck]

Yes, the Pyboards supports LittleFS natively. Tho not enabled by default.

[davefes]

Suggested here: https://github.com/littlefs-project/littlefs/issues/29 that you might be able to run LittleFS ... if you ran it on 8MB external flash how much real storage would you get? The reliability of LittleFS even if you only had as much flash as the internal part would that still be an advantage?

[nspsck]

if you ran it on 8MB external flash how much real storage would you get?

Thonny reports 8MB - 8KB available after excuting the filesystem. Tho I haven't fully tested it now, I assume that at least 7MB is usable. The assumption is based on my experience with the fixed firmware on the reagular version. I have uploaded a file of size 63.1KB to the device and it failed at roughly 62KB.

The reliability of LittleFS even if you only had as much flash as the internal part would that still be an advantage?

In the case of Micropython: In comparision to FAT, LittleFS not only is more reliable, it also unlocks more internal flash for you. On FAT, Thonny only reports 47KB of total space, while on LittleFS, it reports 64KB of total space. So, you are actually gaining some extra storage. Based on that, the only disadvantage of LittleFS is that you are losing the USB MSC functionality.

[davefes]

@nspsck As far as I am aware I have never used USB MSC on my ESP32 work, so does not seem to be a disadvantage losing it.

[rkompass]

@nspsck: I actually noted the corruption on the flash. Still nice to have the alternatives. Now of course there are these questions: What is the standard now? 48K flash and 16K FS_CACHE size. With 80K flash there should be only 32K FS_CACHE size needed. Is that correct? So you would get 32K more flash for 16K less memory, right (not the same as stated now in the README). Very interesting alternative, b.t.w..

I just see now that RAM and CACHE size are also defined in the same place, so your instructions are very clear. My questions are answered :-)

I would suggest that in the beginning of the README you change to

The 8MB version lets you use the 8MB spi storage to store your codes while the regular version offers 155KB flash storage for your codes. The trade of is that you lose SPI1.

The 8 MB version lets you use the 8 MB spi storage to store your codes while the regular version offers 48 KB flash storage for your codes. The trade-off is that you lose SPI1 together with Pins A4, A5, A6 and A7.

Perhaps in the README there should also be a note that the 8MB version is for boards that have an extra winbond chip + capacitor added. Afaiu this has to be done manually (or did you buy such a board?).

Perhaps you like to compare and copy some descriptions from here.

[nspsck]

What is the standard now? 48K flash and 16K FS_CACHE size. With 80K flash there should be only 32K FS_CACHE size needed. Is that correct? So you would get 32K more flash for 16K less memory, right (not the same as stated now in the README). Very interesting alternative, b.t.w.. I just see now that RAM and CACHE size are also defined in the same place, so your instructions are very clear. My questions are answered :-)

Hi, I just want to answer this question for the sake of us surely being on the same page. The standard is 64K flash and 16K FS_CACHE. It contains first 3 sectors and 16K of the 4th sector. You will see only 47K because it's formated with Fat by default, once you format the system to Lfs2, it gives you 63K flash storage. So the 80K version only gives you 16K more usable flash for file storage for exact 16K less RAM.

Also, am going to try to explain this behaviour in more details so it makes a little more sense. (Not a must read :D, just ignore it if you do not want to!) As you can see in this table provided by ST, the flash on the F411CE is separated into 8 sectors, from 0 to 7. Sector 0 is used for ISR (Interrupt Service Routine vector table) caching hence that must be there. We only have sectors 1 to 7 for our code and firmware. Before I go futher, here is a important mechanism that I have to mention: If a change is to be made to a sector, it has to erase the whole sector and then write to it. Because of this nature, micropython decided to cache everything that is going to be changed to a sector (content on that sector as well as changes to be made), once a timer elapses or a write comes in, it write the cached content to the sector. This improves life cycle and performance of the flash but it alseo causes some restraint. If we were putting the firmware to sector 1-6 and use sector 7 for storage, you will be needing 128K RAM for FS caching. But that poor thing only have 128K RAM total, so that is not the way to do it. That's actually why the Firmware uses sector 5-7 (384K total).

With all that in mind, I came up with a theory: if we were going to use only 16K RAM for Caching, as soon as we assign more than 16K from sector 4 to to the filesystem, erasing/writting on the sector 4 will be a problem. Let's assume the file is 17K and sector 1-3 are all filled up perfectly full. At first, the 16K cache is filled up fully, this triggers a erase and write cycle on the whole sector 4, this part we will not see any problem. Then the rest 1K still needs to be written to the storage. This 1K is loaded into the cache and a erase and write cycle is triggered again on the whole sector 4, and data corruption occurs. It does not have to be all true but it should be close to the root of the problem.

On the other hand, if you were using a SPI flash to store your files, you only need 4K cache because most SPI flashes use 4K sector size. Tho, this part is just a theory, I haven't tested it yet but I could load 200K files on to the a 8MB board using only 16K cache. So there is a good chance that this theory is not untrue.

I would suggest that in the beginning of the README you change to

Thanks for point that out! I have changed it now!

Perhaps in the README there should also be a note that the 8MB version is for boards that have an extra winbond chip + capacitor added. Afaiu this has to be done manually (or did you buy such a board?).

Ugh.. I thinks this is not very necessary to mention it since if people were buying this off the official store (aka, the only place you can actually get a 8MB flash soldered on board for a reasonable price), it will be pretty clear that there are 2 versions of this board, and which is which. Plus I do not know if I can even upload a snipshot of it's schematics since the official repo does not have a license file.

Perhaps you like to compare and copy some descriptions from here.

After carefully going through the descriptions by mcauser, I think a quick guide for using a flash module in combination with the 8MB-version would be quite useful. However I need to wait for my flash module to arrive in oder to test it before I make some bold assumptions just like "the 155K flash will work".

[davefes]

However I need to wait for my flash module to arrive

Was that just 8MB chips to put on standard BlackPills? Are you still going to try getting LFS2 working?

[nspsck]

Was that just 8MB chips to put on standard BlackPills?

No, but the formfactor has to match if you want to solder it yourself. WeAct Studio sells a 8MB version and that's the only reason why I am providing a 8MB version.

Are you still going to try getting LFS2 working?

I thought that works now. Are you having trouble with it?

[davefes]

The last comment I read was;

However I need to wait for my flash module to arrive in oder to test it before I make some bold assumptions just like "the 155K flash will work".

I will have my 8MB BlackPills tomorrow and will try the latest 8MB firmware.

[nspsck]

The 8MB version has been tested already and it's proven to work.

However I need to wait for my flash module to arrive in oder to test it before I make some bold assumptions just like "the 155K flash will work".

Sorry for the unclear statement, by that I was refering not to make mistakes about this guide:

After carefully going through the descriptions by mcauser, I think a quick guide for using a flash module in combination with the 8MB-version would be quite useful.

[davefes]

OK, thanks. I have micropython/lora running on the lower-level BlackPill, but for real-life applications need the 8MB variant.

[rkompass]

Hi @nspsck: Just started to read here again.

On the other hand, if you were using a SPI flash to store your files, you only need 4K cache because most SPI flashes use 4K sector size. Tho, this part is just a theory, I haven't tested it yet but I could load 200K files on to the a 8MB board using only 16K cache. So there is a good chance that this theory is not untrue.

That would be a nice revelation, actually, to have 12K (=16K-4K cache reduction) more RAM! So the 8M or 16M external flash variants could perhaps have this more RAM and at the same time give up on the internal flash altogether, making room for a much bigger binary? Like compiling all optional modules like network and ulab etc. in?

[nspsck]

That would be a nice revelation, actually, to have 12K (=16K-4K cache reduction) more RAM! So the 8M or 16M external flash variants could perhaps have this more RAM and at the same time give up on the internal flash altogether, making room for a much bigger binary?

Ye, thanks for bringing that up again. I have forgotten about this neat feature and now it's been added to the 8MB Firmware. The side effects are unclear now, but there should be none. Also, in fact, you can use the 8MB Firmware on a 16MB board... Magic...

Like compiling all optional modules like network and ulab etc. in?

Actually, now there should be112KB more space for frozen modules for the 8MB Version. But I haven't tested it yet.

[rkompass]

Hello @nspsck,

when trying the new firmware (I'm eager to check out the new RAM) you probably used ST-Link with STM32CubeProgrammer?

I can flash the firmware (8MB version) but after a Reset, there is no USB port.

The dfu process apparently only writes one portion of firmware at the first address, so this first header of 408 bytes seems missing. Is that the culprit?

#### old 8MB firmware.dfu:

Device returned transfer size 2048
DfuSe interface name: "Internal Flash  "
File contains 1 DFU images
Parsing DFU image 1
Target name: ST...
Image for alternate setting 0, (2 elements, total size = 324440)
Setting Alternate Interface #0 ...
Parsing element 1, address = 0x08000000, size = 408
Erase       [=========================] 100%          408 bytes
Erase    done.
Download    [=========================] 100%          408 bytes
Download done.
Parsing element 2, address = 0x08030000, size = 324016
Erase       [=========================] 100%       324016 bytes
Erase    done.
Download    [=========================] 100%       324016 bytes
Download done.
Done parsing DfuSe file

#### new 8MB firmware.dfu:
Device returned transfer size 2048
DfuSe interface name: "Internal Flash  "
File contains 1 DFU images
Parsing DFU image 1
Target name: ST...
Image for alternate setting 0, (1 elements, total size = 333616)
Setting Alternate Interface #0 ...
Parsing element 1, address = 0x08000000, size = 333608
Erase       [=========================] 100%       333608 bytes
Erase    done.
Download    [=========================] 100%       333608 bytes
Download done.
Done parsing DfuSe file

Thanks for your responsiveness and engagement :-)

Raul

---

O.K., Correction:

With the new NUCLEO firmware there is also these 2 parts of the firmware:

#### NNUCLEO_F411RE-20231005-v1.21.0.dfu:
Device returned transfer size 2048
DfuSe interface name: "Internal Flash  "
File contains 1 DFU images
Parsing DFU image 1
Target name: ST...
Image for alternate setting 0, (2 elements, total size = 291112)
Setting Alternate Interface #0 ...
Parsing element 1, address = 0x08000000, size = 14736
Erase       [=========================] 100%        14736 bytes
Erase    done.
Download    [=========================] 100%        14736 bytes
Download done.
Parsing element 2, address = 0x08020000, size = 276360
Erase       [=========================] 100%       276360 bytes
Erase    done.
Download    [=========================] 100%       276360 bytes
Download done.
Done parsing DfuSe file

and yet no USB port. I seem to understand now, that the addresses of the portions correspond to the table you show at the start of this repo.

[rkompass]

Seeing

MEMORY
{
    FLASH (rx)      : ORIGIN = 0x08000000, LENGTH = 512K /* entire flash */
    FLASH_ISR (rx)  : ORIGIN = 0x08000000, LENGTH = 16K /* sector 0 */
    /* This is your space for storing Code. */
    FLASH_FS (rx)   : ORIGIN = 0x08004000, LENGTH = 80K /* sectors 1,2,3,4: 16k+16k+16k+32K(of 64K)=80K */
    /* This is where the firmware is stored. */
    FLASH_TEXT (rx) : ORIGIN = 0x08020000, LENGTH = 384K /* sectors 5, 6, 7: 128K+128K+128K=384K */
    RAM (xrw)       : ORIGIN = 0x20000000, LENGTH = 96K
    FS_CACHE (xrw)  : ORIGIN = 0x20018000, LENGTH = 32K
}

and comparing with the flash module organization png shouldn't it be now:

MEMORY
{
    FLASH (rx)      : ORIGIN = 0x08000000, LENGTH = 512K /* entire flash */
    FLASH_ISR (rx)  : ORIGIN = 0x08000000, LENGTH = 16K /* sector 0 */
    /* This is where the firmware is stored. */
    FLASH_TEXT (rx) : ORIGIN = 0x08004000, LENGTH = 384K /* secs 1,2,3,4,5,6,7: 3x16K+64K+128K+128K+128K=384K */
    RAM (xrw)       : ORIGIN = 0x20000000, LENGTH = 124K
    FS_CACHE (xrw)  : ORIGIN = 0x2001f000, LENGTH = 4K
}

and I don't know how to do the settings that there is no FLASH_FS used, of course.

[nspsck]

What you might want to do is a full chip erase. (Because the layout has been changed) I believe there's a full chip erase guide using dfu in the how to flash section.

[rkompass]

I did that dfu-util -s :mass-erase:force -a 0 -d 0483:df11 -D ff.bin.

Is that FLASH_ISR (rx) : ORIGIN = 0x08000000, LENGTH = 16K /* sector 0 */not needed anymore?

[nspsck]

I did that dfu-util -s :mass-erase:force -a 0 -d 0483:df11 -D ff.bin.

Did that work out for you?

Is that FLASH_ISR (rx) : ORIGIN = 0x08000000, LENGTH = 16K / sector 0 / not needed anymore?

This is used for the .isr_vector as you can check out. Basically this is the start up code and they must sit at the beginning of the flash. Hence it is needed.

Ugh, and, even tho I haven't used the STM32CubeProgrammer on linux, I think it will be worth a shot, also, if the programm hangs after you erased the entire flash, just press rst on the BlackPill and it will continue to run again. (At least on windows..)

[rkompass]

Hello,

what I wanted to tell is that even the mass erase and of course, a reset did not work for me. I was wondering whether the .isr_vector is still present in the 8MB dfu file. Apparently it is not, what I deduce from the fact noted above, that the dfu-util only reports one pice of bytes:

#### new 8MB firmware.dfu:
Device returned transfer size 2048
DfuSe interface name: "Internal Flash  "
File contains 1 DFU images
Parsing DFU image 1
Target name: ST...
Image for alternate setting 0, (1 elements, total size = 333616)
Setting Alternate Interface #0 ...
Parsing element 1, address = 0x08000000, size = 333608
Erase       [=========================] 100%       333608 bytes
Erase    done.
Download    [=========================] 100%       333608 bytes
Download done.
Done parsing DfuSe file

The first small piece is gone. If you managed to put both pieces together, (i.e. integrate the .isr_vector and the other part of binary into one block) that might be fine and I have to look for other reasons why I cannot see the board.

[nspsck]

Ugh... That is weird. Because as long as the firmware starts by 0x08000000 the fimrware should boot up normally. It was split into 2 parts but that is only relevant for mboot. As for me, it works perfectly. The only thing that might have gone wrong is the erase part. But I am not an expert on dfu-util, so the following is the only information that I can provide you:

# content of file.sh
set -x
echo -e -n "\xff" > ff.bin
dfu-util -s :mass-erase:force -a 0 -d 0483:df11 -D ff.bin

# output for sudo sh file.sh
+ echo -e -n \xff
+ dfu-util -s :mass-erase:force -a 0 -d 0483:df11 -D ff.bin
dfu-util 0.9

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2016 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

dfu-util: File too short for DFU suffix
dfu-util: A valid DFU suffix will be required in a future dfu-util release!!!
Opening DFU capable USB device...
ID 0483:df11
Run-time device DFU version 011a
Claiming USB DFU Interface...
Setting Alternate Setting #0 ...
Determining device status: state = dfuERROR, status = 10
dfuERROR, clearing status
Determining device status: state = dfuIDLE, status = 0
dfuIDLE, continuing
DFU mode device DFU version 011a
Device returned transfer size 2048
DfuSe interface name: "Internal Flash  "
Performing mass erase, this can take a moment
Setting timeout to 35 seconds
dfu-util: Only DfuSe file version 1.1a is supported
dfu-util: (for raw binary download, use the --dfuse-address option)

After that, I verified the content of the board with STM32CubeProgrammer, and turns out, all the content from 0x0800 0000 to 0x0808 0000 (which is 512KB total) are all set to 0xFFFF FFFF. Tho, the weird part is, unlike erasing then flashing with STM32CubeProgrammer, if I were using dfu-util, the firmware skipped the factory reset process. (Where the blue led should be on for a while until the flash is formatted correctly.)

Which leads me into thinking, that you might have a flash issue. Could you please try holding the user key during boot to do a factory reset? The blue led should flash during the whole process and after that, a USB device should pop up (on windows).

[rkompass]

Could you please try holding the user key during boot to do a factory reset? The blue led should flash during the whole process and after that, a USB device should pop up (on windows).

That was the solution. After writing the firmware holding it for triple flashing --> after this factory reset the USB port appeared. Voila:

>>> import gc
>>> gc.collect(); gc.mem_free()
96384

which is almost a + 16K !

Congratulations and a big thanks.

PS: Now there is this FAT filesystem appearing again, but I know what to do...

[davefes]

Holding down the USER button during the whole boot process does not give me a USB port on an official WeACT board. Actually, I don't care but will do a mass erase. I would just like to be on the "same page" as you guys.

gc.mem_free() gives me 97184

Which command does one use to tell how much flash is available? Tried pyb.info(), micropython.mem_info()

BTW, you are doing a great job documenting this as well as getting this board to work. Thanks.

[nspsck]

You can use pyb.info() and look for the GC: total to check the total RAM available.

And thanks for the compliment!

[davefes]

Sorry I meant flash storage not flash RAM.

`

import what_filesystem Littlefs v2 filesystem 0008 magic littlefs 0014 version 0x20001 0018 block_size 4096 001c block_count 2048 0020 name_max 255 0024 file_max 2147483647 0028 attr_max 1022 `

gives 8388608 so I guess all is good.

[nspsck]

Ah, then you look for the LFS free: right below GC.

But well, if you were using LFS2, there will be a chance that pyb.info() crashes because of how that function is implemented. It does not support LFS2 very well.

[davefes]

It does crash:

GC: 98880 total 1696 : 97184 1=28 2=7 m=18autoconnect: /dev/ttyACM0 action: remove

Is this "connected" with my USB not being mounted?

[nspsck]

The crash is caused by the pyb.info() assumes that the filesystem is fat.

[davefes]

Got that. Thanks

[davefes]

Observation ... doing a mass-erase does not remove the filesystem or the files at /flash Maybe, that is a good thing!