ESP32 S2/S3 USB MSC support?

[MichaelDu9226]

Are there any plans to increase MSC support here?

CDC+MSC may be great

Thanks

[peterhinch]

MSC mode is a can of worms for this reason.

[stephanelsmith]

Bump, this would be a neat feature. In my application, it'd be preferable to present a controlled list as opposed to automatically using the literal file listing. Maybe like how ble does it, an irq callback when a listing is requested, file read or write. I know very little about USB Msc support, maybe a naive idea on my part, but I think would be cool!

[weilian1977]

test result shows filesystem corruption maybe happen

[t35tB0t]

@peterhinch - From experience I'll counter that USB MSC is not at all a can of worms but actually a significant development accelerator. It already works beautifully on STM32's. I'd never cripple my development system by turning it off (during development). Espressif also realized the value of USB and so went to great trouble to trade silicon real estate for USB support. As such, I'm have a serious look at ESP32-S3 as a challenging STM32's in some use-cases.

IMHO, it would be a huge mistake for the micropython community to delay or limit TinyUSB integration. Many will simply drop mpy and move on to circuitpython over something like this. For good reasons, Adafruit appears to be making it a policy to focus on devices which support USB MSC. After doing some significant development on STM32's with USB MSC and having zero issues with flash corruption, I'll strongly favor devices with USB MSC.

Note that ESP32-S3 with USB MSC storage enabled works right now on the circuitpython fork. I begrudgingly flashed cpy onto my hardware and it "just worked". Now looking forward to same on micropython (or micropython on zephyr).

[Red-M]

The only time this would be an actual issue is if someone wrote self writing code which edited main.py and/or boot.py or another file that was being actively developed on, but someone that would be doing that would understand what they're doing and know how to safely exit those loops to be able to write to them again.

I feel like this answer is another cop out for not wanting to fix the mess that is the USB stack on micropython. Expect contributions soon™

[peterhinch]

File corruption is a risk if a program modifies the filesystem in any way. The issue crops up repeatedly in the forum. I found myself answering the question so frequently that I wrote the FAQ referenced above.

[robert-hh]

@Red-M @t35tB0t As Peter said, the problem with MSC origins from a PC and Pyboard assuming to have exclusive access to the filesystem and being able to modify it - writing files, directories and the FAT. In addition to the PC caching and deferring accesses this will a file system corruption sooner or later. Even if you understand the risks and try to avoid it. Circuitpython avoids that problem by making file system modifications exclusive. Either it's the PC or CircuitPython which can modify the file system, and to switch between the modes you have to unmount and remount the file system. That's safe, but inconvenient. @iabdalkader suggested for his RP2040 implementation of MSC a kind of dynamic access arbitration. As long as the board is mounted to a PC, the local MicroPython is not allowed to write. Once unmounted form the PC, local access is allowed. That avoids the need for manual switching. I did not follow the discussion about whether that was implemented. Still, the problem of caching by the PC exists. But that is not different to USB drives. A minor drawback of using MSC is the limitation to a FAT file system for the MicroPython board, which does not support wear leveling. b.t.w.: TinyUSB is used by some MicroPython ports.

[iabdalkader]

I did not follow the discussion about whether that was implemented.

It was implemented, but it was never merged. I kept the commit that adds it here: https://github.com/micropython/micropython/pull/7402/commits/35895830e6f5052cd2d09ffe312267e6c8afbd4d if anyone wants to reuse it somewhere feel free to do so.

[t35tB0t]

@Red-M Yes, thank you for detailing the factors for others here. However, I really do understand these factors and have been using USB MSC on mounted SDcards with STM32 for 1000's hours with no corruption. The repeated assertion here that using USB MSC will sooner or later corrupt the file system is not accurate.

What will corrupt your flash is power disruption, or bad code, or bad user behavior. But fundamentally, an embedded system that is writing to its system flash is fundamentally less robust regardless if you have USB or not. Note that I'm somewhat centric to SDcard usage because I'm developing applications that are too large for typical MCU internal flash. The latest ESP32's are interesting but have storage performance issues. SD cards have an embedded ARM controller and do their own wear leveling now. If they are not powered down properly and are crashed during a write, their wear leveling will corrupt the entire file system. So, the recommendation is to use Industrial Grade SDcards which have more support for avoiding such corruption.

As far as making USB MSC safe to use, simply consider the following and you'll be fine...

• User apps should not be randomly writing to "system" flash - this is not "best practice" and user design problem
• There is no need for access arbitration. Dynamic automatic arbitration (locking) would be cool but isn't essential
• Host PC write caching should be turned OFF, that's a user dev environment problem not MicroPython's
• Reboot MicroPython after writing to flash from Host PC. OK to write during normal execution, but need to reboot to use
• If apps such as FTP are active, don't write to the flash from PC until FTP disconnects. Be a responsible developer
• Never power off a target while flash is being written do (obviously)
• Never remove an SDcard while system is running

I hope I'm not just in a minority who is writing larger/more complex MicroPython applications (500K to 1MB source code)? Unfortunately, I have seen no alternative development path that compares to the efficiency and cycle time of using USB MSC. It really works really well. If unavoidable flash corruption exists, with these large projects and heavy USB MSC use, then file system corruption would be a problem for me. It isn't. What is a problem is a lack of any good alternative to USB MSC.

Thonny and rshell are incapable or supporting these projects effectively. If there is anything that compares to the performance and utility of USB MSC, please let us know here.

[peterhinch]

If there is anything that compares to the performance and utility of USB MSC, please let us know here.

mpremote is the official solution.

[t35tB0t]

@peterhinch - As far as link behavior goes, the official mpremote is not architecturally different than Thonny or rshell. I've found both Thonny and rshell to be unusable for larger projects. A faster interface is needed: USB MSC is an obvious choice that has proven to be fast and useful without corrupting SDcards. USB over Ethernet might be interesting as an extension but even having FTP in c-code below micropython would be a relief.

I must discount these other tools which are using REPL over a slow default serial connection. The raw USB rate is nearly 100x faster. The default UART speed at 0.1152 Mbits/sec is actually 10x slower than the UART hardware can actually do. Even with a UART speed-up, the USB will still be 10x faster to we should be using it.

Thus, we have no comparison to USB MSC. Two key facts here are that USB MSC has been extremely reliable on STM32s and that CircuitPython has USB MSC working on the ESP32-S3 already.

Are we aware of any intention to implement USB MSC on Micropython for the ESP32-S3?

[Red-M]

I plan to but I'd prefer a regular dev to provide such a feature as this codebase seems to have very particular rules for contributions.

I really want to get MSC working and then work on the actual thing I want being USB HID as honestly, MP should really be noticing the features that CP has since its a major driving force to the amount of people around it.

If I can get some traction on #8689 as I have a bunch of expresif ESP-S3 devboards that need this compile time flag to work (I have N8R8 DevKits), I can start to work on some of the shortcomings of USB under MP.

[t35tB0t]

@Red-M - I have some N8R8's as well. I fiddled with TinyUSB in MP but only got it partly integrated in an evening. The USB event callback with ESP SDK is a bit convoluted and CP's approach is vastly different than the legacy MP USB so I stopped there. If you do make progress on USB MSC on the N8R8s, I'll gladly do some debugging/testing. Fortunately, I also have a stack of STM32's - apparently the global supply of STM32 ICs has all but vanished.

[Red-M]

Realistically I just want to make a keyboard with some custom firmware but this would be way easier with threads on an RTOS because I want low latency on the key loop.

Somehow this is a hard ask as CP has a good USB stack but has decided that threads are "too hard" for most people and thus have removed threads entirely (not disabled). MP on the other hand will let you use such interfaces but the USB stack is a mess and doesn't have a concept of HALs for interfacing with hardware so the same library interfaces have to be maintained between ports instead of a unified interface into the port specific HAL, which makes maintenance a huge strain.

I'm fine to try for USB MSC support but I already have some really dumb tooling that did what I needed enough to test out and play with MP but this constant dodging of features is making people move to CP instead because they have features but not much power in terms of multicore support (where as MP has much better support)

[t35tB0t]

@Red-M - Agreed. CP as a dumbed down MP fork was a waste of developer time and my time to try to make something useful with it. Perhaps Zephyr will mature and change the solution space here. Meanwhile, patching in USB MSC (along with other USB modes) to the ESP32-S3 would make these N8R8s pretty nice. It would be a leap forward in usefulness. Of course, some serious work addressing Octal Flash and PSRAM issue(s) is also needed.

We could also try to find some high-speed UART rate combos on PC/target to work around the divider mis-matched bauds. Not as sweet as USB MSC but we might at least give folks more useful interface performance with e.g. Thonny. I've done this before so just need to dig up some notes, re-implement, and then test stability perhaps with a BER pattern.

[M-D-777]

Hi, all

Any interest in adding MSC support for ESP32 S3? https://github.com/micropython/micropython/pull/8699

I have added this feature by referring to the pull request of @iabdalkader https://github.com/micropython/micropython/pull/7402

The MSC feature is configurable with sdkconfig.usb_msc.

MSC with sdcard will be later.

[t35tB0t]

@M-D-777 - Nice. Thank you for this. Will try your MSC implementation this weekend then see if I can add SDcard and maybe RAM disk blk dev support as well. But these are just stop-gap measures. Zephyr is moving to fill this hardware interface gap far better than MP or CP ever can. So we're eager to have that kind of full featured platform running MP.

[tsaost]

First of all, I am not familiar with the internals of MP at all, so maybe this is a bad idea that has been rejected already.

Why not have two partitions in flash, one that is writable by PC but only readable by MP (so this partition will hold .py code, configuration etc.), and another partition that is only writable by MP (it would be nice, but not strictly necessary for it to be readable by PC) so that it can be used by MP code for logging data, saving states, etc.)

This should cover most of the common usage that people want to use USB MSC for, i.e., be able to save modified Python code quickly to the ESP32, and for MP code to save data without worrying about data corruption. The only issue is how to get the saved data out from the ESP32 to PC, but this can probably done using some kind of file transfer over Wifi.

If this a bad idea, or it is too much trouble to implement, I'd say make USB MSC an optional feature that can be turn on and off. It should be OFF by default, and one should be given a warning that this is an "adult feature" that could cause data corruption if used indiscriminately. Those who consider themselves adult can then turn it on and bear the responsibility for the consequences :-)

Edit: Sorry for making this unnecessary suggestion. Lots of ideas, pros and cons, have already been discussed in rp2: Add MSC support.

[jimmo]

Why not have two partitions in flash, one that is writable by PC but only readable by MP (so this partition will hold .py code, configuration etc.), and another partition that is only writable by MP (it would be nice, but not strictly necessary for it to be readable by PC) so that it can be used by MP code for logging data, saving states, etc.)

You can already set up this exact scenario on the stm32 and rp2 ports (and it would work on ESP32 if we had MSC support). It's unfortunate to have to fragment the flash into two parts though (in terms of wasted space).

With a read-only-from-micropython partition, you still have to have a way to know exactly when the filesystem is in a valid state for reading. i.e. if the PC is in the middle of of a file operation (e.g. saving code.py), the device needs to know that all blocks are flushed (at every layer from the host PC to the host disk cache, to the device's flash cache) before being able to read. Obviously it's not nearly as bad as two concurrent writers, but even a concurrent writer and reader isn't safe.

I think CircuitPython has some heuristics to detect completed write operations to make this work (e.g. idle timeout?), i.e. this is how they trigger the "code.py has change, do soft reset" thing. Honestly I think the way CircuitPython implement this is a good compromise in the complexity-vs-ease-of-use tradeoff.

If only MTP was usable on the host PC... and we wouldn't need to worry about this.

I'd say make USB MSC an optional feature that can be turn on and off. It should be OFF by default, and one should be given a warning that this is an "adult feature" that could cause data corruption if used indiscriminately.

Yes, see https://github.com/micropython/micropython/pull/7402#issuecomment-1048777491

(Edit: sorry just saw your edit to link to the exact same comment :) )

[dhalbert]

I think CircuitPython has some heuristics to detect completed write operations to make this work (e.g. idle timeout?), i.e. this is how they trigger the "code.py has change, do soft reset" thing. Honestly I think the way CircuitPython implement this is a good compromise in the complexity-vs-ease-of-use tradeoff.

CircuitPython waits 0.75 seconds (used to be 0.5 seconds) after the last write before triggering an auto-reload. We still see delayed writes on Linux and Windows. macOS is much better about it. Windows supposedly fixed the worst delayed writes (which were on FAT12 only, probably a legacy of floppy support) many years after we reported the problem, but I haven't checked again recently.

[robert-hh]

@iabdalkader made a PR #8814 for rp2 and I did for the mimxrt port #8820, setting the filesystem to r/o by Micropython as long as the drive is mounted at the PC, and setting it to r/w after it had been unmounted. For my test configuration of mimxrt boards and debian Linux I found, that the Micro had to go through a warm reset to ensure, that all modifications done by the PC are noticed by the Micro. So it it a three state logic. That's not overly comfortable, but seems safe. For the rp2 set-up this did not seem to be necessary, which I'm skeptic about. There is also a boot mode option to disable MSC at all. And yes, MPT will solve that. I looked around for MTP implementations, but did not find any.

[t35tB0t]

The dual-mode USB solutions here are good ideas but, IMHO, they unnecessarily complicated. The STM32's work perfectly well as-is. So, unless there's something adversely different about the ESP32, I see no good reason for putting much effort into that complexity.  Try developing with USB on STM32's and then come back here with real problems instead of imagined collisions and contrived scenarios.

From hundreds of application development hours on the STM32's I am able to assert that, as-is, the MPY USB functionality is by far the most superior interface solution (for development).  This includes working from windows and linux host machines.  The Micropython paradigm with the STM32's has been there for a very long time and it "just works".

Why make the ESP32's different?  What's sad is that the ESP32's hardware with USB was a welcome arrival.  Having it broken like this is quite a letdown.  Seeing that CircuitPython made it work (albeit in a gimped way) is irritating.  The alternate development interfaces perform poorly to the point of being unusable for larger application development.  I challenge anyone to prove otherwise.

So, having USB work the same in MPY, across supported  hardware, is a reasonable expectation.

[dhylands]

I have encountered numerous corruptions when using MSC with STM32, which is what motivated me to write rshell, so while they may be less frequent, they are definitely still there.

With MSC, the host assumes it has exclusive access and it will happily overwrite changes made by the micropython board. This is why the volume needs to be RO when the host has it mounted.

[t35tB0t]

The fundamental issue here is that the flash access is way too slow for any reasonably sized project when using any currently available interface than USB.  We welcome any similar solution but are dismayed with the apparent lack of understanding surrounding USB and flash usage. rshell is clever but useless for any large development project.  USB on STM32 has been the only useful solution to that. So, this is not a code development or host issue - if we're trying to break a paradigm of protecting code flash from corrupted execution, then we're already on a path to compromised designs.  Good luck when the power fails suddenly or the code has an evil bug.

But firstly, hat's off to the awesome work MPY Devs have done to make MPY as stable as it is.  MPY has greatly matured and works very well for real-world professional applications --- when used with solid architecture.  If you have encountered numerous corruptions when using MSC with STM32 then, in all due respect, you were not using that flash "properly".    I'll explain...

 A robust target application should not be actively writing to program flash.  Why on earth would one do that?!  Its not safe for an application to be actively writing to code flash during runtime.   Consider what corruption can happen when power is lost.  This is one of those things you can do but should not.  We do not want to accidentally create self-modifying code here.

My team has had zero flash corruption issues with MicroPython running in a production environment running million dollar products in complex test environments.  The MPY applications are relatively large, complex, and must reliably operate the networked production test equipment.  In addition to local stepper motors on RS485, we have many socket services (had to increase socket resources significantly).  And have many async tasks running cooperatively. Yet we are very restrictive on any writing to flash.  Without USB development, the project would have been impractical.

We would not be using USB if it were going to put expensive products at risk.  And that's why we never write to flash during normal runtime execution either.  So, no, the volume does NOT need to be RO when a development host is mounted - MPY needs to use the flash RO when the host is mounted or, smarter, use it RO all the time.  This is a fundamental architecture that most professional products use.  Add separate data flash or support data partitions if you must log continuously.  But NEVER write to your own code space.  That's like dumping garbage in your own bed.  Don't do it.

[tsaost]

@jimmo Thank you for your comments. I agree that even with the two partitions solution, in theory there are still synchronization issues. But frankly most people will not see the problem because developers will need to be "trained" to soft reset MP after uploading new code so that the new code will take effect (CircuitPython's auto reload is neat but I'd rather have manual control). IMHO there can be no foolproof solution to any problem because the world will just produce a better fool :-)

I also agree that there will be wasted space, but the two partitions solution also offer the advantage that the "internal" partition writable by MP can then be set to Littlefs and have better wear leveling. Actually, as you pointed out, the current STM32 port already offered this exact solution, as explained by Hybrid (STM32)

@t35tB0t I agree with your sentiment regarding the usefulness of USB MSC, and I'd like to see it supported by MP whenever the hardware supports it. But wouldn't the slow uploading problem be partially alleviated by breaking a large project into a number of smaller .py modules during development?

[t35tB0t]

@tsaost USB is operating at 12MHz and SerialCom defaults to 0.115MHz.  Above ~400Kbits, the SerialCom BAUD error can cause issues.  With some effort, we may be able to get some target serial consoles running at ~1MBAUD.  However, this is just a serial port and in now way offers the simplicity of a file system mount over USB.  I edited the hard-coded Thonny baud rate to try this out and the experience was underwhelming.  There was also an issue EPS32-S3 flash caching when running PSRAM along with some very slow performing work-arounds.  This may be related to the reluctance from some here.

In any case, my example project has like a hundred py files and related configuration files.  We have large dictionaries so I created a caching dictionary object to preserve RAM.  Just the py files alone total 12,000 lines. So it isn't a tremendous amount of code.

So, it actually is mostly small files.  And, no, rshell and Thonny would fail to support development on a system like this. In looking for nearby alternatives, I'm looking at MTP - just like our cell phones use.  This would support file browsing but not in-situ editing.

Thus, there are no alternatives on the able here with the same anywhere near utility level as USB MSC both in terms of performance and functionality.  CircuitPython clear made their position on this.  Zephyr with MicroPython reduced to just being an application may be the future answer.  Especially if the MPY team has having so much difficult staying on top of powerful hardware development.

The request is simple:  add "adult" USB MSC support to all relevant platforms without complicated nannys.

[peterhinch]

@t35tB0t

Add separate data flash or support data partitions if you must log continuously. But NEVER write to your own code space.

That is good advice for professional users and advanced amateurs who can easily add flash chips. Many of the users we support are hobbyists whose first project is a simple data logger - they are unhappy when their only filesystem is corrupted.

[t35tB0t]

@peterhinch Agreed that implementing a data logger using only internal flash presents a problem.  And that problem is that data logging to code flash really isn't simple (because of the issues I've already highlighted).  If Devs really think that one tiny logger use-case is so important to focus on, then I'll suggest creating a special safe-data-logger function.  One which does protected writes to reserved data blocks so code flash can not be corrupted.  That's just one possible approach that adds value instead of reducing functionality for the rest of us.  Actually, I might even use such a function myself.

[Red-M]

Y'all heard of a semaphore?

[Meekdai]

I think ESP32 S2 S3 can completely follow the way of RP2(#7402 ) first, and it is up to the user to decide whether to use MSC or not.

// Enable USB Mass Storage with FatFS filesystem.
#define MICROPY_HW_USB_MSC  (1)

[paulftw]

I received a lolin s2 mini yesterday and my mind was blown when I saw how easy it is to update code in CP compared to the painfully slow rshell.

Seriously considering teaching CP instead of MP in my next workshop just for that reason alone.

The choice to withhold this extremely time saving and user friendly feature because it’ll affect some users down the road leads to bad learning experience, cripples adoption, slows down community growth.

if you care about a small subset of users who will write to flash and hit a bug it is logical to care even more about a much larger subset of users who will want to upload custom code to their board.

MSC should be supported, enabled by default or at least mentioned on the first pages of the tutorial.

If the problem is as serious as some argue (while others, seemingly just as experienced, are claiming it’s a non-issue in real life scenarios) I as a user would understand if enabling MSC disables writing to FS from Python. even a drastic measure of having FS read-only by default would be worth it as long as a one liner to enable writes is provided plus a mention of associated dangers.

[robert-hh]

Uploading custom code to flash is easy, using mpremote or ftp for devices with network support. If you just have applications that do not write to flash than having MSC support with the flash set to r/o is not an issue. Besides that, there is a common understanding that device which can support MSC should do that. Only it's not yet implemented or merged.

• For the STM32 port, MSC is implemented, with the board's file system in r/w mode. That cause me a lot of trouble, until I decided to disable MSC.
• For rp2 there is an implementation, enabled currently for just one board. And there is a PR open with a kind of automatic r/o mode as long as the board's drive is mounted to the PC.
• For the MIMXRT port, there is a PR open, with a automatic r/o mechanism.

For the other ports, someone has to implement it. I could do that for the SAMD port, once the other previous commits for that port are merged. An implementation for the NRF port should be possible based on the rp2 implementation. ESP32 is more difficult since it uses a different USB stack.

[t35tB0t]

@paulftw - Awesome to hear that you're teaching embedded Python and agreed that the utility of USB MSC is literally a 100x benefit.  The effect of using USB MSC as-is with STM32 is not serious and its not really a problem.  The issue is that it is an easy feature to access.  People expect it to "just work" and it does.  Where they get into trouble is when they run off and craft code to write to flash without understand what they are really doing and why the choice to write to code space is inherently dangerous irregardless of USB MSC.  Turning off USB MSC doesn't fix user ignorance of flash usage risks.

This can be ameliorated with a bunch of explanatory documentation and good code examples.  @Red-M might even provide some semaphore examples to show how MP can protect itself from aggressive users on the USB. Architecting for robust solutions and avoiding the pitfalls of flash corruption could be an informative workshop in itself.

@robert-hh - actually, we found that developing custom code using mpremote or ftp is not all that effective.  We wrote a multi-instancing ftp server in MP so can push data over e.g. 8 sockets in parallel.  Its cute and works well file tools like FileZilla - most of the time.  The ftp server is functional as a mass code update tool.  However, it is not effective for code development.  If MP crashes or is halted, then you're stuck. So, ftp and mpremote were found to not be truly useful for heavy development work.

[victorallume]

I too have had various issues with corruption using MSC on a pyboard, due to simultaneous FS access from the PC and microcontroller. However, it's pretty useful, as long as you're careful. The main way I ensure safety is to drop the microcontroller into a REPL before using MSC. Would it be feasible to only allow MSC access when the micro is in a REPL mode?

[t35tB0t]

@victorallume  Reading the source code, it certainly is feasible to block MSC access when MPy is not in REPL.  But really we don't need the added complexity and hassles from that.  We didn't need it for the more sophisticated STM32's so why add MSC nanny's for the ESP32s?  Just design well for what you have in hand and you will have zero problems with MSC. I've seen enough development and deployment hours with MSC on STM32s to assert that the much of the debate over MSC flash corruption here is FUD. Tell us a specific corruption scenario and we can discuss how to fix that system's design so MSC usage is not a flash corruption concern.

[robert-hh]

@iabdalkader has implemented a lock mechanism for the RP2 board, and I did so in the MSC PR for the mimxrt port. It sets the file system to read-only for the board as long as it is mounter by the host. Similar to what Adafruit does, but kind of automatic. To get R/W access at the board, you just have to eject it at the host. For most users it is useful. If you have a usage pattern which does not use mixed write accesses, then this is not needed.

[t35tB0t]

@peterhinch -   As a long time contributor to MicroPython, you and others have definitely produced a reliable and powerful tool for embedded systems. Many of us out here have benefited from a focus on efficient memory usage and practical implementation in constrained systems.   Hence, your concern regarding file system corruption is appreciated.  Yet, respectfully, I feel the concern is misapplied in the case of USB MSC.  I'll also suggest that your referenced FAQ is misses the point in calling USB MSC a "pitfall" when the true pitfall comes whens designing target code that writes to flash.

Generally speaking, constantly or randomly writing to flash file system with your code base is bad practice. Power disruption or other run-time issues can result in storage corruption.  So, this is not a just a USB MSC risk but a general flash-use risk that novice user's should be made aware of.   Add separate SPI flash for logging if you must.  Just stop writing code that modifies the system flash or you will eventually corrupt it.

The discussion here is regarding implementation of USB MSC on the ESP32-S3.   This feature is vastly more effective for complex systems development than using Thonny or rshell.  On the STM32 with USB MSC, we can attach am 8GB SDcard, lay down a git repo on the sdcard, and develop a complex project with many modules from a PC with a favorite editor.  Code, debug, check-in, push to server; fast cycle time; seamless development; zero file system corruption.

I tried Thonny and rshell with the ESP32-S3 and it was such a sluggish experience that I'm back to writing ESP32 code on the STM32 with USB MSC and then doing final segments on the ESP32.  I tried CircuitPython on the ESP32-S3 and, while the USB-MSC is working beautifully, the overall departures from MicroPython were beyond disappointing.  So, the plea here is for USB MSC on ESP32-S3.

[t35tB0t]

@Red-M ... semaphores? cute and even a funny redneck jab - but it misses the point.  The premise here is that USB MSC access to flash can result in flash corruption. The emphasis here is that ANY writes to code flash are dangerous operations that require operational considerations or you will get bit.  In addition, @peterhinch is asserting that data logging in a minimalist hardware configuration is a highly prized feature.

Semaphores and the like are all tiny mechanisms and not a complete solutions.  Let me disrupt the power while you're using your semaphore to demonstrate...

Writing to flash pages containing code is fundamentally dangerous and not a safe beginner move.  If the intention is to make these things unbrickable, then there's plenty of other MPY features that a curious beginner can use to break their system.  And thus a lot more thinking and code would need to be written - and it only protects users from learning.

I'm left here wondering if other challenges with the ESP32 itself or affection for LittleFS may be reluctance factors here?  After all, STM32 with USB MSC has been around for years and without a need for this discussion thread.  So why worry now?

Is there something fundamentally buggy in the current ESP32-S3 hardware?  We know the PSRAM caching design is problematic.  Is the ESP USB hw interface also suffering some bug that makes it undesirable?  Seem to function well enough with CPY...

[peterhinch]

I have no desire to enter a debate on this, merely to reiterate an empirical observation from forum support: novice Pyboard users were regularly scribbling over their filesystems. Advice from experienced users was to turn off MSC support in boot.py.

The automatic r/o mechanism described by @robert-hh sounds good.

[t35tB0t]

@peterhinch  yes, there is no debate here.  As a denizen of the MPy community there is great respect for you. Yet we shall correct your empirical observation because it may be misleading folks into thinking that all experienced users are telling folks to turn MSC off. Some have suggested otherwise.

Experienced users are making it clear that MSC must be available on hardware that was powerful enough that the chip designers included the USB core. Or they'll drop MPy and move on. Flash corruption is a trivial thing to recover from but slow development tools are a permanent impediment. There are plenty of MPy features that allow us to corrupt our flash and few will actually brick a unit. Novices should play; they be allowed to build fast and fail often.  Its good for them.

Having the auto r/o MSC mode as default as @robert-hh suggests would be fine.  As long as we can flip a switch and leave it where we want it, then its all good. But a MPy development policy of protecting novices at the expense of access to basic hardware capabilities isn't sustainable.

[jimmo]

@t35tB0t

Yet we shall correct your empirical observation because it may be misleading folks into thinking that all experienced users are telling folks to turn MSC off.

Peter is just reporting what he's seen. There's nothing to "correct". I have the same empirical observations, and as an experienced MicroPython user I also provide the same advice -- for most users, using MSC to transfer files to/from the device is not the best option when you weigh up all the competing factors (assumed experience, convenience, risk of corruption, etc).

But one of the key things about MicroPython is that we strive provide building blocks, and users who want to combine them in ways that make sense for them should definitely do so! https://github.com/micropython/micropython/pull/7402#issuecomment-1048777491 is a good summary from Damien for the same issue on the rp2 port. We're aiming to get to the point that this can all be configured in boot.py however the user likes, and at a per-partition level. (But there is some work to get there!)

For deployment of larger projects such as the sort of thing you're talking about, we're also working on the "mapfs" feature https://github.com/micropython/micropython/pull/8381 -- this has the advantage of saving significant amounts of RAM (because the .mpy files run from flash after being copied to the device) as well as making deployment a lot faster... see notes on this at the bottom of this post!

@tsaost USB is operating at 12MHz and SerialCom defaults to 0.115MHz.

This may be true when accessing serial over a USB/UART converter, but in this case the point is moot -- MSC would have been unavailable anyway. When accessing USB serial over a CDC ACM device it does not run at 115200 -- the speed setting when opening an ACM device is just forwarded through to the other end and it is free to do whatever it chooses with this information (e.g. if you were implementing a USB/UART that is the speed you should open your UART at).

Awesome to hear that you're teaching embedded Python and agreed that the utility of USB MSC is literally a 100x benefit.

When you are teaching MicroPython, from a lot of personal experience I find that although MSC is a huge benefit in terms of simplicity, it is basically guaranteed filesystem corruption for new MicroPython users. And the only really efficient workflow that involves working with MSC is to edit directly from the device, and so filesystem corruption means the student loses their code. CircuitPython has fine tuned their workflow to make this as reliable as possible (and it's one of the key differentiators of CircuitPython and one of the many reasons why I'm glad that CircuitPython exists -- it's a great alternative to MicroPython in the education space).

Another note -- one thing that trips up new users is that when they copy the files, they still need a serial console to do anything useful. So it's neat to be able to "just copy main.py" but anything above the most trivial program you need a serial terminal -- you might as well install mpremote or similar. And it's confusing when you copy the file over MSC, you still need to trigger a soft reset to remount the filesystem to be able to see the new files for importing). (CircuitPython again solve this by having a heuristic to trigger soft reset automatically on write, which works well for them).

For small projects, "mpremote mount" is an amazing workflow that gives you the best of all worlds -- the files live on the PC, they run automatically from the device, and no risk of filesystem corruption.

In any case, my example project has like a hundred py files and related configuration files.

I tried out a similar scenario on a pyboard v1.1 by creating 50 small files (random sizes between 200 and 1400 bytes).

• Copying with MSC to FAT: 41.3 seconds.
• Copying with mpremote to FAT: 90.8 seconds.
• Copying with mpremote to LittleFS: 64.0 seconds.
• Copying with mpremote deploy-mapfs: 3.0 seconds. (!!!!)

I think we can possibly optimise mpremote-littlefs a bit here, and in particular a rsync-like partial update would be useful. Additionally, at least half of the mpremote-littlefs copy time is rewriting the same pages over and over again and this could be improved. Given the other benefits of both LittleFS and mpremote, the extra copy time is a worthwhile tradeoff for most users and I will continue to recommend disabling MSC.

As maintainers though, as you can see the the mapfs feature is likely to take a higher priority because it is such a huge win in every way (reduced RAM, > 10x faster copy, etc). The copy is so fast because it does a single erase for the whole region and then it's just streaming writes.

[t35tB0t]

The "correction" is that experienced developers are also suggesting folks leave MSC on and avoid that nastly habit of writing data to code flash!

Projects many times larger than the tiny speed test referenced here will simply not be practical without MSC. mpremote-littlefs with an rsync like partial update is a workflow benedfit for projects as they get larger.

Also, the file system restart to sync process shouldn't be confusing - its so fast it becomes muscle-memory yet still and with intent and control. Teach why its necessary and move on. We need to be able to tweak multiple files on a running system without having it automatically restart with each edit. MSC paired with a terminal console has been by far the most streamlined tool chain we've tried with our Mpy projects.

I'm pretty sure I can find ways to corrupt flash even with MSC missing. As can we likely fix designs where MSC was deemed to be the corruption villain.  I have specific cases where MSC workflow works fine.  Where's a real example where we can recreate the flash corruption and study the root causes?  I'd like to see real cases where one must use a serial interface to be safe.

[TonyWeston]

Its been a while since this was updated, but to add my 2c.

Regardless of if MSC is a 'Can or worms' or not. The ESP32/S2/S3 has the hardware to support it, so support should be developed. Unless MSC is marked as deprecated across the entire micropython project, we should not be deciding what or what does not go into the specific ESP32 port.

Otherwise the ESP32 will become divergent from the micropython standard - this is the worse option, in my opinion, and will hurt micropython as a whole, with developers now uncertain which boards their code will run on, or not.

A solution to the corruption problem, would simply to lock the filesystem to python writes, if that filesystem is opened on the host PC. If, MSC is never developed (because the ESP32 micropython developers have decided that MSC should not be there) then solutions like this would never come to light.

[robert-hh]

MSC is atm supported for the stm32 port and one rp2040 board variant. So it is not very common across MP ports. There is an PR to support Python coded USB devices. That would be a portable way to have various USB device types and modes. Then users can decide whether the install it to their boards or not. I had implemented MSC support for the mimxrt port including a locking mechanism that you mentioned. It worked. The PR for it is not online any more. I closed it in favour of the python variant. Note that the initial ESP32 MCUs cannot support MSC, lacking USB hardware.

[t35tB0t]

MSC is not the problem with corruption - humans are.

If a system on a chip includes a mainstream hardware feature - it is expected to be supported by firmware. Cutting out mainstream features in order to chase a goal of fully normalizing MicroPython across all platforms is not going to end well. An exaggerated example would be limiting precision to integers because not all hardware has floating point.

The reason a particular piece of silicon invests all the transistors needed to implement hardware USB block is likely because that device has the horsepower to justify such an interface. It follows then that a more powerful development interface is going to be useful if not essential if one expects to use all that horsepower effectively. From experience, I'll assert that its really simple to use MSC safely and effortlessly even without extra locks. What is the "PR for Python-coded USB devices" trying to solve? The MSC Interface needs to be fast and not running in python environment.

Plan A - beg for MicroPython Devs to retain and extend embedded USB support to all relevant hardware modules Plan B - use Zephyr RTOS and microPython as an app so my hardware isn't gimped Plan C - learn all the low-level USB stuff and code it myself (not likely because I'm busy frying bigger fi$h here) Plan D - use CircuitPython (just kidding, I tried it and it was worse than awful - they perverted Python beyond tolerance).

After so much time has past on this topic and ESP-S3's still don't have a solid MSC solution, I'm suspecting that only Plan B will yield useful results in the long run.

[Red-M]

I was going with Plan B anyway/aswell, its probably better that micropython as a whole moves to becoming a zepher app because its easier for board/hardware support.

[jonnor]

USBDevice support landed in recent MicroPython. And there is an MR for ESP32-S3 at https://github.com/micropython/micropython/pull/15108