Add a bunch of handy new commands

[gitcnd]

Most awesome micropython project ever !

I've found the new included commands I wrote indispensable for working on my hardware, so I'm sharing it back with everyone.

[vsolina]

I'm crying from joy right now! This is a next level contribution

I had no idea you can use espcam from micropython! Storing a jpeg file to a FS from python in conjunction with ramdisk (to not wear-out the flash) will allow me to do some really cool projects. Thank you

Wget upgrade will also be very useful to me personally. And ADC definitely And set command is totally amazing! it's essentially a gpioset from linux Basically every command here is gold

You also updated the readme and added useful comments to each command implementation and integrated touch into the shell. You have no idea how many times my muscle memory wrote touch and the shell responded with command not found. :D

You really made this project your own ❤️

Please give me a day or so to take a deeper look at the changes and test it out a bit and then I'll merge this PR. But from what I've seen so far it's good as is. Changes to the base shell are minimal and unlikely to break anything and commands are totally isolated and even if there's a problem with any of them it will not affect basic functionality. One thing you could do is to remove the unnecessary lines which are commented out, e.g. in adc.py:

6: #import os
48: #print("adc pin"+adcpin+" delay"+adcdelay+" loop"+adcloop)

Definitely not things like:

18: # 27, 32, 33 are good candidates

But this is not that important, I'll merge it as is if you don't get a chance to remove them.

The only problem here is if you continue adding more commands I'll need to improve the project structure 😱 and separate basic utilities from advanced/specific ones - e.g.: two bin directories. In order to make it easier to deploy essentials on boards with a tiny flash.

[gitcnd]

So glad you liked my contribs!!

I'm about to teach a micropython course, and your code is absolutely necessary to help people use this stuff.

I've got a large number of things I want to do...

1. figure out git hooks, so we can auto-generate minified versions of the larger files (no point for files under 4096 bytes though, because that's the min flash block size)
2. sanitize input parameters (introduce switches)
3. find the best possible way to make all the commands work from more places than just the sh.py environment. e.g. this is the best I've dreamed up so-far ARGV=[4,3,0.1,0.1];exec(open('bin/blink.py').read()) - but I want to try and make it easier still, and need to test it with mpremote and other things.
4. introduce some kind of config system, so custom stuff (e.g. pins like LED_BUILTIN, camera, LCD pins and screen type, etc) have their own place to live, and our code like "photo.py" and "blink.py" etc can always "just work" no matter what board uses them. Same for wifi ssid lists and passwords, etc
5. maybe get telnet hooked up always, with security
6. more commands - especially "git" - so users can "git pull" to auto-update everything
7. and some harder stuff: I'm working on OTA flash updates without requiring any OTA partition, and custom flash partition sizes (which gives you 30% more filesystem space https://github.com/micropython/micropython/issues/13445 )
8. I'm also trying to fix a bunch of bugs in micropython itself (bad handling of esp32cam serial hardware bugs) as well as adding the esp32cam board itself - but the micropython PR system is a total shambles and my lack-of-git-knowledge isn't helping.

[gitcnd]

Are you open to a chat about the best way I can add more stuff to both your repos?

I'm "old to programming" (40 years!) but newish to python (2y) and even newer to micropython (2mo) [but a decade with MCUs, and 40y with electronics]... so I'm not sure what kind of code-style modern people expect to see.

My general plan is to work out the minimalist switch-handling code, and an invocation method that's compatible with different ways to run programs, a versioning system so we can add an "update" command for the MCU which understands how to automatically come back to this git and grab needed updates... and then insert this into all the existing things.

I've got more new commands already (LCD screen driver stuff, espnow server/client stuff, radar chip decoder, and assorted other hardware testers).

The config system is a bit more complicated, since it ideally lives "outside" your system (defining the camera/lcd/LED/etc pins and drivers and device name, wifi settings, users and passwords, "environment" strings, etc are all more "global" kinds of stuff). I'm not aware of anyone doing this yet though, and it's a tricky problem. i.e. If we make one up and make it some other repo, then yours is going to depend on it. if we add it into your repo, then other people will depend on yours, even if they don't want shell...

Also: there's 900 people in my group, about 50 of which are likely to actively participate, and quite of few of those will probably be open to "giving back" as part of their learning process (all 900 of these are already good programmers) - I've got a half-baked plan to encourage them to implement anything they think is needed as well...

Thoughts?

[vsolina]

That's awesome, good luck with the course, GJ for spreading the knowledge! We need more teachers like you.

You're proposing really good ideas, I was thinking about some of the similar problems and solutions. A few thoughts:

1. Git hooks + 7. git commands There's a git implementation in pure python: Dulwich https://github.com/jelmer/dulwich and a wrapper called Gittle https://github.com/FriendCode/gittle These might be portable to micropython but it seems to me that running full git on microcontrollers with very limited memory be an overkill. Especially since it would implement non-essential features (branches, tags, multiple origins, etc.) for this usecase. Also checking out a git repo by default fetches and stores the entire history of all branches which would be problematic on the file system sizes we're working with. Instead I was thinking this environment needs a package manager (like aptitude, yum, brew) that would allow on-demand installation of available packages directly on controllers. This would require a package index and versioned packages (package version and MPY version, possibly even MCU architecture specific ones to support compiled binaries). It could be hosted on a dedicated github repository that would contain both the index and packages themselves. This would be more powerful and customizable than using micropython's pip and pypy. In addition to installing packages, a simple way to publish package updates directly from a microcontroller would also be useful. This could be solved with some kind of small VPS that acts as an intermediary between the MCU and github repo, automatically building/preparing/testing packages for each MPY version, architecture; and would be able to do minification, (optional) compilation to .mpy - for speed and size reduction.
2. My initial "standard" for defining main() functions in commands is peculiar and could definitely be improved.
3. Cool idea, an alternative could be something like from bin.blink import main; main(args)
4. Yup, this would be great; IMO config should be a single file that lives on the MCU and there's a version (potentially in the package index) available for every supported board. This way it would be available for all commands, custom programs, and would be isolated so each board only fetches config for itself.
5. There's no way to make telnet secure since it's unencrypted, that's why we switched to SSH. Boris implemented an ssh client for MPY https://github.com/loboris/MicroPython_ESP32_psRAM_LoBo/wiki/ssh (among other amazing things, e.g.: powerful threading library), but I've never seen a ssh server that would would work on MPY with little effort. I suspect that 500KB RAM available on esp32 might not be enough to run a sshd in addition to web server and custom scripts. Running it by default with a password is mostly a matter of minimal configuration changes to boot.py. Could be done, but not everyone wants to run it.

One more thing you reminded me of is the need for an installer to do initial setup (install MPY and optionally shell, web editor) which could allow a user to configure the MCU the way they want, e.g.: which MPY version they want, which WiFi to connect to, should telnet be on or off by default, what repl password to use, should shell run by default, which pkgs to install, etc. Tasmota has an awesome installer similar to that: https://tasmota.github.io/install/

"the best way I can add more stuff to both your repos?" - opening a PR should suffice.

Modern is overvalued, the essence of coding did not change for a very long time. From the code you're contributing I can tell that you don't need my help, you're doing really well.

"The config system is a bit more complicated, since it ideally lives "outside" your system..." - IMO this aligns really well with a dedicated repository for package managment index, packages and config files. But the devil is in the details.

"...I've got a half-baked plan to encourage them to implement anything they think is needed as well" Sounds awesome, but simplifying a process to contribute would definitely help reduce friction.

Main issue here is that I have limited time and energy to contribute to these projects, so any contributions are always welcome. To summarize, major projects I was considering (and might implement in the future) are:

1. package management system
2. web installer
3. Extension of the editor to add support for compiling C code. This can be implemented using x86 VM in browser that would (down)load and run on demand when a user runs a compile command. And this seems like the best bet for adding support for git - implementing it in editor instead of on the MCU.