File Module: Can't see writes until board is reset

[iabdalkader]

When creating or writing to a file, I don't see changes after calling file.close() until after I reset the board, for example, if I run this:

>>> f = open("0:/test.txt", "w")
>>> f.write("HelloWorld")
>>> f.close()

the file is not created...but if I reset the board, the file is there and written...I'm not sure if this is intended, I can confirm though that f_close calls storage_flush() immediately which calls flash_cache_flush :

flash_cache_flush () at storage.c:26
26      if (flash_cache_dirty) {
(gdb) n
28          flash_write(flash_cache_sector_start, (const uint32_t*)CACHE_MEM_START_ADDR, flash_cache_sector_size / 4)

[stevie67]

Hi, my understanding is that this is unavoidable, or? The operating system assumes that it has full control over the file system. It will cache a lot of data and just assumes that the cache is always right. So it won't react to any change coming from the Python board.

Actually that points to a deeper issue: Writing to the file system from the board should only ever be done, when no write from the host computer is happening afterwards or at the same time. Otherwise you will get a corrupted file system. Maybe the storage should be put into read-only mode for the host, before writing from the board is enabled...

[iabdalkader]

Hi, my understanding is that this is unavoidable, or? The operating system assumes that it has full control over the file system. It will cache a lot of data and just assumes that the cache is always right. So it won't react to any change coming from the Python board.

you're probably right, because I traced the code and it does flush the data, I just need to remount the file system to see my changes, maybe a sync from the host will work too...

Actually that points to a deeper issue: Writing to the file system from the board should only ever be done, when no write from the host computer is happening afterwards or at the same time. Otherwise you will get a corrupted file system. Maybe the storage should be put into read-only mode for the host, before writing from the board is enabled...

right, personally I don't really need to write to the internal flash, I was just messing around, I just need to read from the flash, I think fatfs can mount the flash fs as read only, and the sdcard as read/write and somehow make it read only for the host... This way you can copy scripts to the flash, and read data from the sdcard...

[dpgeorge]

Yes, @stevie67 is correct: you can't have either the flash or sd card mounted by both the MCU and your PC (via USB) at the same time.

It's easy to make flash and sd card read only, to either the MCU or the USB MSD. But this is still not going to solve the problem of "MCU writes to drive, PC doesn't see the changes", because, even if the PC mounts read-only, it still uses a cache.

The other way round, "PC writes, MCU sees the changes" can work, since you can disable the caching of the filesystem blocks in the MCU.

[iabdalkader]

But this is still not going to solve the problem of "MCU writes to drive, PC doesn't see the changes", because, even if the PC mounts read-only, it still uses a cache.

You might want to add that to the FAQ :)

The other way round, "PC writes, MCU sees the changes" can work, since you can disable the caching of the filesystem blocks in the MCU.

Is it possible to disable the cache entirely ? for reading/writing ? I was thinking about it lately, I could use another flash block, swap blocks maybe, is there any other way around that ?

[dpgeorge]

Is it possible to disable the cache entirely ? for reading/writing ?

Just call storage_flush after a call to storage_write_block. That will keep the cache clean, which is functionally equivalent to disabling the cache.

Note that the SD card (now fully implemented, I hope!) doesn't have a cache. Any writes are immediate.

[iabdalkader]

That will keep the cache clean, which is functionally equivalent to disabling the cache.

this will keep it sync'd right ? I actually want to use all the cache memory...there's no way to write directly to a flash block without erasing it first, right ?

[stevie67]

Correct. To my knowledge there is no way. That's a pity, because some of the flash regions are actually bigger than the RAM which Damien has allocated for the cache, so those can not be used in the file system - well unless you would use some of the flash as a cache together with the RAM, as Damien suggested at some point. But that sounds a bit strange for me... I guess, the SD card is the better choice, then.

[iabdalkader]

That's a pity, because some of the flash regions are actually bigger than the RAM which Damien has allocated for the cache

And in my case I'm only using 16KB of the CCM for cache, so I'm limited to 16KB blocks :D, and now I need those 16KB too.

unless you would use some of the flash as a cache together with the RAM, as Damien suggested at some point

that could work, one the 128KB sectors can be used to temporarily store the current sector while it's erased and then write it back with changes, but if I limit myself to 16KB I can do this in memory, this way I won't need to reserve 16KB of memory all the time

[dhylands]

Well, generally speaking, the way you deal with flash is that you have to erase a large block (sometime called an erase block). This converts the memory to all 0xff's. There are also typically smaller write blocks.

Depending on the flash, you can then do much smaller "writes" which will convert 1's to 0's. There is no way to convert a 0 back to a 1 unless you erase.

With NOR flash, (which is what I believe the onboard flash is) you can generally do many many smaller writes to a block (again converting some of the 1 bits to 0 bits. With NAND flash, there are often limits on the number of writes that can occur to a block before you need to erase it.

Does anybody remember WORM drives? (Write-Once Read Many). These had a very similar behaviour, except they didn't have any erase capabilities.

So you can general write the first time without erasing, but to re-write requires an erase.

On Mon, Feb 17, 2014 at 10:29 AM, Ibrahim Abd Elkader < notifications@github.com> wrote:

That will keep the cache clean, which is functionally equivalent to disabling the cache.

this will keep it sync'd right ? I actually want to use all the cache memory...there's no way to write directly to a flash block without erasing it first, right ?

Dave Hylands Shuswap, BC, Canada http://www.davehylands.com

[iabdalkader]

What if we used a few buffers in RAM for cache ? this way you can use all the flash storage, including 128KB sectors and have the 64KB CCM free for other things ?

[pfalcon]

Closed? So what's the resolution on this one? I marked this ticket as "wontfix" because there doesn't seem to be a way to fix it with the current state of USB filesystem gadgets (and that's great shame for the industry!). But this is important matter, which should be well documented in a way that no user could ever miss it, and until then, I suggest to keep this open.

[dpgeorge]

This problem does need attention, at least to make the FS behave in a predictable way.

[iabdalkader]

Hi, I closed it because I thought it wasn't really an issue with MP, anyway, we could disable the MSC if the storage (flash/SD) is used by MP ? (when opening files for example), this way MCU/Host will have exclusive access to the storage ?

[dpgeorge]

With an SD card inserted, there are 2 independent filesystems: flash and SD. One could be for USB MSC, the other for uPy.

[lurch]

Sounds like the pyb module needs a function so that it can either mark the flash as "read-only to the MCU, read-write to the USB host" or "read-write to the MCU, read-only to the USB host" (which will presumably need a "soft disconnect and reconnect" of the filesystem for the USB host side to pick up the read-only status change). Currently the flash appears writable to both the host and MCU which I guess is just asking for trouble!

Hmmm, actually I guess if the host is in read-write mode, we can't know if it still has data sitting in write buffers (waiting for the user to "safely eject" it). So maybe the "initial mode" (whether host or MCU sees the flash as read-write) could be selected in boot.py (with the default obviously being host-has-write-access), and if the initial mode is MCU-has-write-access, then from the MCU side you could later do the swap to host-has-write-access, but you could never switch it back again without resetting the pyboard.

And I guess if you're still in MCU-has-write-access mode, you could do a "soft disconnect and reconnect" of the filesystem for the host side, but then still keep it "read-write to the MCU, read-only to the USB host" which should force the host to drop its caches, and then see the new changes you've written to the flash from the MCU side?

And obviously all of the above should also apply to the sdcard too, if present (I guess with the difference being that if an sdcard is inserted, the MCU will still be able to see both 'drives' but the host will only be able to see the sdcard). Hmmm, maybe if the host is in read-only mode, the "soft disconnect and reconnect" could also be used to switch whether the host sees the flash or the sdcard?

So in conclusion maybe default to:

• no sd card inserted at bootup -> flash is ro to the mcu but rw to the host
• sd card inserted at bootup -> flash is rw to the mcu, sd card is rw to the host but ro to the mcu

And then have a function like host_filesystem_readonly() that you could optionally call in boot.py[1], and if the host is in readonly mode, then you could later call a function like host_filesystem_remount(drive, readwrite_mode=False), with drive indicating flash or sd card. And once the host has been switched into readwrite mode, any further calls to host_filesystem_remount would raise an exception. I think just those two functions would cover everything discussed in this thread? :-) They'd also allow you to switch the sd card in the pyboard and "remount" it to the host (assuming that the host had been kept in readonly mode of course)

[1] Assuming there's a reliable way to ensure this only gets called in boot.py, i.e. before the host gets a chance to mount the MSC.

[pfalcon]

And I guess if you're still in MCU-has-write-access mode, you could do a "soft disconnect and reconnect" of the filesystem for the host side, but then still keep it "read-write to the MCU, read-only to the USB host" which should force the host to drop its caches, and then see the new changes you've written to the flash from the MCU side?

Why do you think that host after getting "soft disconnect" will proceed to resetting its caches and not throwing error into user's face?

That's not going to work, the problem is not solvable in general case. Because wrong access method is used. With mass storage, host controls passive file system. And passive file system cannot signal anything to host. Trying to perform device-side USB disconnect when host is actively engages in higher-level protocol is certainly an error condition (unless you can quote USB spec on allowing device-initiated signalling for mass-store).

Again, the problem is that "active-passive" model was selected, with 2 active participants. So, model with 2 active parties should be used, and then one should be leading (take control of single resource). Client/Server, and apparently server should be the board. Then it should be for example FTP protocol. Problem: not as convenient as just mounted disk, and a bit of chore to implement. Or it can be X-, Y-, or Z-modem transfer within serial console - but how many people on Earth still know what it is and how to use it?

All in all, client/server model is the only which may offer correct concurrent usage results - a host got to ask board to do something, instead of grabbing its flash. And by "a host" is of course meant "a user", so it's gotta be user-friendly. That means not just #362, but probably some other convenience functions to essentially provide pythonistic mini-shell.

That's of course if FTP sounds scary. I'm actually surprised noone started to implement SLIP in Python yet - I'd do just that, if someone else would work on core language, asyncio, etc. But then SLIP is also not the most user-friendly protocol (can Windows still do it?).

[lurch]

Why do you think that host after getting "soft disconnect" will proceed to resetting its caches and not throwing error into user's face?

Because that's the behaviour I've observed (sort of!) on my Windows laptop.

1. Plug in pyboard
2. MSC device shows up
3. Browse pyboard drive in Windows Explorer
4. Create new file in flash on MCU side
5. Wait for red LED to go off
6. Refresh in Windows Explorer
7. Windows doesn't see new file
8. Press reset button on pyboard
9. Refresh in Windows Explorer
10. Windows can now see new file

[dhylands]

@lurch What you wrote out is exactly what I'd suspect. The Windows side thinks it owns the flash and it will have its own in memory cache. The only time it will see anything done by the MCU, is if the entry isn't in the PC cache, and it decides it wants it, then it will go and reread the flash.

The problem with UMS is that its a block level thing, not a file level thing.

I think it makes sense to have uPy be able to control whether and when the flash gets shared with the PC.

Basically, when the flash is being shared with the PC, uPy shouldn't write anything, because the Host side could decide to write something in the same area that uPy did, wiping out whatever uPy wrote. You're basically just getting lucky that you uPy written file is showing up.

The way I think of it is that its really mutually exclusive. Only the PC or uPy should be able to write to the volume, and as long as the volume is being shared witht the PC, then uPy shouldn't be writing (or reading for that matter) from the flash.

[pbeart]

Presumably if you are using software that is stored on the host machine, you could continually write to a different portion of the same file, thus clearing the cache? Actually, on second thoughts, appending to a file relies on having an accurate representation of the file, then adding to that, so I suppose that wouldn't work. Anyway, does anyone know how to use windows batch to completely disable the cache for a specific drive?

[lurch]

Lots of hits in google about disabling write-caching for a drive, but not disabling the read-cache (or the metadata-cache, which is the bigger problem here). But then I found http://msdn.microsoft.com/en-us/library/windows/desktop/aa364218%28v=vs.85%29.aspx which describes how to open individual files with the cache (both write and read) disabled. But I haven't experimented so I dunno if it will actually help in this specific scenario.