How is the source/target device specified in the file management commands?

[fake-name]

Looking through the documentation, rshell seems to support moving files to the micropython device's storage:

usage: cp SOURCE DEST
usage: rsync [-m|--mirror] [-n|--dry-run] [-q|--quiet] SRC_DIR DEST_DIR

Etc...

However, what's not clear is which direction the copying is, and how to specify it.

For example, how do you disambiguate between cp pyboard.file.py local.filesystem.file.py and cp local-filesystem.file.py pyboard.file.py? What happens if both ends have a identical file, and I want to clobber the local version with the one on the micropython board? Or vice versa? Is the second command always assumed to be the python board?

There seems to be some sort of heuristic at play, at least for rsync:

/home/durr/Programming/ESPScreen/workSpace> rsync . /
Adding //creds.py
Adding //boot.py
Adding //main.py
Adding //app.py
/home/durr/Programming/ESPScreen/workSpace> rsync ./ /
Adding //creds.py
Adding //boot.py
Adding //main.py
Adding //app.py
/home/durr/Programming/ESPScreen/workSpace> rsync * /
Source directory /home/durr/Programming/ESPScreen/workSpace/* does not exist.
/home/durr/Programming/ESPScreen/workSpace> rsync . .
Checking /home/durr/Programming/ESPScreen/workSpace/creds.py
Checking /home/durr/Programming/ESPScreen/workSpace/boot.py
Checking /home/durr/Programming/ESPScreen/workSpace/main.py
Checking /home/durr/Programming/ESPScreen/workSpace/app.py
/home/durr/Programming/ESPScreen/workSpace> repl
Entering REPL. Use Control-X to exit.

>>> import os; os.listdir("/")
['boot.py', 'creds.py']

I'm not sure what "adding" means in this context, since it doesn't seem to actually mean anything is getting copied. Also, there's the spurious // which shouldn't be an issue normally but potentially could be messing something up.

I'm not sure what the solution here could be, short of using some sort of rsync host:<path> board:<path> sentinel or similar. Or it could be hard-coded so src is always the host machine filesystem, and dest is always the target board, but that would break getting files from the board.

Really, this is an issue for basically every command that operates on files. I'd assume that things like cat cat's files on the python board, but I can interpret it both ways.

Things like cd. Which end is it changing the directory on? The python board? The local board? Both?

Other bugs:

/home/durr/Programming/ESPScreen/workSpace> repl ~ import os; os.listdir("/") ~
Entering REPL. Use Control-X to exit.

>>> import os
>>>

(I hit Ctrl+X manually)
>>>
Unrecognized command: os.listdir("/") ~
/home/durr/Programming/ESPScreen/workSpace>

However the "pass a line to repl to execute" works, it seems to explode if there are more then one statement in a line, which renders the concept kind of useless.

[davehylands]

rshell has the notion of "virtual directories". When you have a pyboard attached, for example, you'll find a virtual directory called /flash and a virtual directory called /sd (if you have an SD card mounted). rshell also allows multiple Microopython boards to be connected simultaneously, in which case each one gets named. The default name is pyboard. You can change this by creating a board.py file which specifies a different name. Each board will also get a virtual directory created.

So with a pyboard 1.1, you'll wind up with /flash and /pyboard/flash both being virtual directories which represent the /flash directory on the pyboard.

rshell also has a notion of "current working directory". You use the cd command to change the current working directory.

rshell allows the source or destination directories to be either on the host or on the device. So you can copy host-to-host, host-to-device, device-to-host, or device-to-device.

When a filename (source or destination) doesn't started with a / then it gets appended to the current working directory. So:

cp foo.py /flash

will copy foo.py from the current directory to the /flash directory on your device (assuming that the device has a /flash diretocry). You could also use cp foo.py /pyboard/flash.

If you did the following:

cd /pyboard/flash
cp ~/foo.py .

then that would copy foo.py from the users home directory (on the host) to the /flash directory on the device.

All of the commands behave this way (cp, rsync, ls, etc).

So you view the filesystem as one big tree, with virtual subdirectories that represent the files on your device(s). When you cd, you're cd'ing in the one big tree. This follows the linux/unox model where volumes get mounted into your hosts directory tree. There is only one single current directory.

[fake-name]

Why doesn't ls follow those rules then?

/home/durr/Programming/ESPScreen/workSpace> ls
app.py   boot.py  creds.py main.py
/home/durr/Programming/ESPScreen/workSpace> ls /
boot/       dev/        home/       media/      opt/        root/       srv/        tmp/        var/        lib32@      lib@        sbin@
cdrom/      etc/        lost+found/ mnt/        proc/       run/        sys/        usr/        bin@        lib64@      libx32@     swap.img
/home/durr/Programming/ESPScreen/workSpace> cd /flash
Directory '/flash' does not exist
/home/durr/Programming/ESPScreen/workSpace> ls /flash
Cannot access '/flash': No such file or directory
/home/durr/Programming/ESPScreen/workSpace> ls /pyboard
lib/     boot.py  creds.py main.py
/home/durr/Programming/ESPScreen/workSpace> ls /pyboard/flash
Cannot access '/pyboard/flash': No such file or directory
/home/durr/Programming/ESPScreen/workSpace> boards
pyboard @ /dev/ttyUSB0 connected Epoch: 2000 Dirs: /boot.py /creds.py /lib /main.py /pyboard/boot.py /pyboard/creds.py /pyboard/lib /pyboard/main.py

So the virtual directory /flash doesn't seem to exist for ls. /pyboard does, but /pyboard/flash doesn't. And none of them are visible when doing a ls of /, so you have to know there's a magic directory path that does \~\~things\~\~ to make use of them. And I'm not using a pyboard, so that makes things even more confusing. And somehow, what should be located in /pyboard/flash seems to be at /pyboard, at least for my use (I'm using an ESP32).

Also, what happens if there's a local directory named /pyboard?

The virtual file system idea is neat, but the bizarre semantics of how it's implemented are.... problematic. The major issue is the fact that it doesn't follow the linux "one big tree" behaviour, but rather behaves as several trees overlaid on each other where sometimes a file on one masks a file on the other one, and sometimes you have to know ahead of time that accessing magic paths will retrieve content on one of the other trees. Currently, it rather works like something along the lines of overlayfs.

It seems like it'd make much more sense to have a /local directory that is equivalent to the "mount" point of the local filesystem, or some similar mechanism (/cwd)? At minimum, if you're going to have magic paths, they need to be injected into the return of ls so the can actually be found.

[davehylands]

ls does follow those rules (as far as I know).

/flash doesn't exist for all boards. It does for the pyboard 1.0 and 1.1. It doesn't for the RPi Pico or ESP32 (for example).

For the board you have, the internal filesystem is showing up at /pyboard because the board mounts its internal filesystem at /. For these boards you see the files on the device under /pyboard.

The boards command needs to be improved (it's currently showing all of the files in /)

I'm not aware of any "masking" behaviour that you're describing. A particular directory comes from the host or the device, never both. Your ls output seems entirely correct for a board that mounts its filesystem at / (on the device).

[fake-name]

/flash doesn't exist for all boards. It does for the pyboard 1.0 and 1.1. It doesn't for the RPi Pico or ESP32 (for example).

For the board you have, the internal filesystem is showing up at /pyboard because the board mounts its internal filesystem at /. For these boards you see the files on the device under /pyboard.

I understand why they're at different paths. The problem is ls doesn't show any of them.

/home/durr/Programming/ESPScreen/workSpace> ls /
boot/       dev/        home/       media/      opt/        root/       srv/        tmp/        var/        lib32@      lib@        sbin@
cdrom/      etc/        lost+found/ mnt/        proc/       run/        sys/        usr/        bin@        lib64@      libx32@     swap.img

Note what's not in the above. ls doesn't show any of the virtual mount points!

This renders the mount points completely undiscoverable from within the tool.

Effectively, ls is unaware of the mount points and doesn't show them.

A particular directory comes from the host or the device, never both.

So if the host has a directory named /pyboard, and I cd /pyboard with an attached device, what happens? I assume I'll wind up on one of the two directories.

This is what I mean by masking, ~~and how the model here is more like overlayfs or another layered filesystem. If something is present in the top layer (presumably the pyboard), it's returned. Otherwise, the next layer (PC local filesystem) is searched, etc... As such, if something is present on the pyboard, it "masks" anything with the same name further down.~~

Thinking about it more, this is like how linux handle it, in that linux also masks things like this. If you have a directory with contents in it, mounting something to that directory renders the contents of the directory unavailable while the mount is present.

Whether this is a good way to handle this is.... debatable. I still think it should be /local/ -> PC /, /device/ -> pyboards. Then, ls /pyboards would show all connected pyboards, etc...

[davehylands]

Note what's not in the above. ls doesn't show any of the virtual mount points!

This renders the mount points completely undiscoverable from within the tool.

Effectively, ls is unaware of the mount points and doesn't show them.

That's a good point and one that can can be addressed. Thanks for pointing this out.

Whether this is a good way to handle this is.... debatable. I still think it should be /local/ -> PC /, /device/ -> pyboards. Then, ls /pyboards would show all connected pyboards, etc...

That would also be possible, although you still have the same "masking" problem if you have a real directory named /local

If there is a collision, I would be inclined to make the device directory take precedence over the local one, otherwise you might not be able to access the device at all. Most people don't create additional directories in / but rather create them in ~ (their home directory). And so far there have been no reports of clashes.

I would be inclined to add something lile /pyboards/BOARDNAME (i.e. /pyboards/pyboard for the default case) as an additional location to /pyboard mostly for backwards compatability. /pyboard can be changed already by "naming" your board. I've also been wanting to create an rshell configuration file which would allow some of these things to be controllable.

I also need to fixup the boards output for boards which mount their filesystem on /.

[fake-name]

That would also be possible, although you still have the same "masking" problem if you have a real directory named /local

No, I mean / is accessible only at /local/. Effectively, you mount the PC filesystem / to /local/ in rshell, so the ls of / would show: /pyboard /local only (assuming one connected pyboard).

The rshell / would then not point to the actual local drive root, but rather a virtual directory that lists all the connected devices and the local disk.

[davehylands]

OK - I get what you're saying. When I wrote rshell I went with the linux model. Whatever it does, I think I have to preserve the existing behavior.

I'll have to think about this some more, it might be possible to support both models.