stmhal: os.sync() [apparently] doesn't flush FatFS read buffers

[pfalcon]

My time to ask dumb questions. I created a new file on pyboard's internal flash over USB mass-storage . Expectedly, it didn't show up. I issued sync on Linux side, it didn't show up. I unmounted filesystem, and it still didn't show up. I issues os.sync() on pyboard, and it still didn't show up. I had to reboot pyboard to get hold of it. I'd say, unmounting mass-storage and os.sync() should be 100% enough.

[dhylands]

Not really. Personally, I think that USB Mass Storage on the pyboard is more or less broken.

The host thinks it has exclusive access, and the pyboard thinks it has exclusive access, so of course, there will most definitely be conflicts.

I think that the only reliable way to make USB Mass Storage work is to turn off local file access while the volume is being shared with the host. Anything else will lead to inconsistencies, corruption, etc.

So I personally turn off USB Mass Storage, and use rshell to copy files in and out.

[pfalcon]

I think that the only reliable way to make USB Mass Storage work is to turn off locl file access while the volume is being shared with the host.

This is exactly what I do - access MS via host, make host flush all buffers, then make pyboard flush its buffers and reread actual state (unless os.sync() does something else, and then it should do that too). That should work (because nothing else can, indeed).

[dhylands]

I think that os.sync() only does the flushing part. I don't think it rereads any buffers.

If the pyboard had written anything to the filesystem and it was still cached in memory, then it's entirely possible that the act of calling os.sync() on pyboard would corrupt data which was synced by the linux host (since both parties believe that they have exclusive access they both assume that nobody other than themselves wrote anything.

[pfalcon]

I think that os.sync() only does the flushing part. I don't think it rereads any buffers.

Yes, quick checking shows the same. If there's desire to somehow make USB mass storage usable, then besides flushing write buffer, fatfs apparently simply needs to be reinitialized.

If the pyboard had written anything to the filesystem and it was still cached in memory, then it's entirely possible

This is pretty obvious (as for bug report submitted and respond by developers), so whoever does that, does wrong. Of course, care should be taken to convey this fact to folks to whom it's not obvious, but it's beyond scope of this ticket.

[dhylands]

One approach that would partially work (as long as the pyboard isn't writing anything) would be to invalidate all internal caches everytime a write by the the host is detected. Then you wouldn't have to call anything on the pyboard, it would just work.

And, if we can detect when the host is connected (I think that this might be possible, but I'd need to investigate further), then we could mark the filesystem on the pyboard side as read-only.

[pfalcon]

Yes, those ideas are good follow-up ideas. But I really think we should do it step by step and first step should be making it possible to see data correctly written by host to an idle pyboard, after issuing os.sync(), not after reboot. If that works, we can look how to automate it.

[dhylands]

I don't think we want it tied to os.sync(). I think you want an os.invalidate().

Calling os.sync() in its current form WILL corrupt things if there is any unwritten data. So since you basically don't want ANY of the current os.sync() functionality, I really don't think it makes sense to add the invalidate to it.

I think that adding an os.invalidate() which will toss any internally cached buffers and force the pyboard to pick up changes made by the host would be an excellent first step.

Then added the automation, and other stuff.

[dhylands]

I created PR #1589 which adds an os.invalidate() function.