cp -r and wildcard copy fail

dclaar commented 2 years ago

Recursive copy doesn't copy files in subdirectory:

> cp -r flash /flash
> ls /flash/lib
> !dir flash\lib
 Volume in drive N is public

 Directory of N:\Shared\Doug\projects\m5stack\weather_and_AQI\flash\lib

12/23/2021  03:58 PM    <DIR>          .
12/23/2021  03:55 PM    <DIR>          ..
12/20/2021  08:56 PM             4,324 aqi_and_color.py
12/23/2021  03:58 PM             3,858 brightness.py
...

Where flash/lib has multiple files only creates flash/lib, not copying any files.

Attempting a wildcard copy, rshell creates a bogus source directory, then says it can't find it (note that it copies the current directory twice, which makes the path invalid):

cp flash/lib/* /flash/lib
File 'N:\Shared\Doug\projects\m5stack\weather_and_AQI/N:\Shared\Doug\projects\m5stack\weather_and_AQI/flash/lib/aqi_and_color.py' doesn't exist

This is on windows 10 with 0.0.30.

dclaar commented 2 years ago

resolve_path:

    if path[0] != '/':
        # Relative path
        if cur_dir[-1] == '/':
            path = cur_dir + path
        else:
            path = cur_dir + '/' + path

doesn't work on windows. Figuring out how best to fix it.

davehylands commented 2 years ago

forward slashes are actually fine on windows (if that's what you're worried about). The only place on windows that doesn't like forward slashes is the CMD.EXE program. All of the internal APIs etc have worked fine with forward slashes since forever.

The thing that rshell doesn't support well under windows is multiple drives. It assumes that everything is on the current drive.

dclaar commented 2 years ago

It's true that cp foo /flash has worked forever.

But if cur_dir has a windows style path--which it gets from os.getcwd():

>>> os.getcwd()
'N:\\Shared\\Doug\\projects\\m5stack\\weather_and_AQI\\flash'

then it works for that single cp, as the path foo is relative.

But for cp lib/* /flash/lib, resolve_path returns the absolute windows path:

resolve_path:  N:\Shared\Doug\projects\m5stack\weather_and_AQI\flash/lib

Then process_path returns a list of absolute paths:

process_pattern:  ['N:\\Shared\\Doug\\projects\\m5stack\\weather_and_AQI\\flash/lib/aqi_and_color.py', 'N:\\Shared\\Doug\\projects\\m5stack\\weather_and_AQI\\flash/lib/brightness.py'...]

And then resolve_path is called again for each pattern, e.g. N:\Shared\Doug\projects\m5stack\weather_and_AQI\flash/lib/aqi_and_color.py

Since that pattern does not start with '/', resolve_path prepends cur_dir:

resolve_path:  N:\Shared\Doug\projects\m5stack\weather_and_AQI\flash/N:\Shared\Doug\projects\m5stack\weather_and_AQI\flash/lib/aqi_and_color.py

and the command fails.

I found that the python 3 way to handle all of this is pathlib, which also obviates all of the "is there a '/' at the end" stuff.

The current path I'm heading down is to write resolve_path as:

def resolve_path(path, *a):
    """Resolves path and converts it into an absolute path.

    Converts path to absolute on all platforms.
    If extra arguments are given, they are appended to path in 
    the correct format for the OS.
    if path[0] == '~':
        # ~ or ~user
        path = os.path.expanduser(path)

    path = pathlib.PurePath(path, *a)
    if not path.anchor:
        # Relative path
        path = pathlib.PurePath(cur_dir, path)

    # This is a hack because the dest is posix, but source may be windows:
    #  On windows, '/a/b/c' will return false
    #  But 'n:\a\b\c' or '//net/a/b/c' will return true.
    #  On posix, we will have '/a/b/c', and will return true.
    if path.is_absolute():
      return '%s' % path
    else:
      return '%s' % path.as_posix()

and to call it everywhere path stuff is done, e.g.:

def process_pattern(fn):
    """Return a list of paths matching a pattern (or None on error).
    """
    directory, pattern = validate_pattern(fn)
    if directory is not None:
        filenames = fnmatch.filter(auto(listdir, directory), pattern)
        if filenames:
            return [resolve_path(directory, sfn) for sfn in filenames]

rsync:

    for src_basename in to_add:  # Name in source but absent from destination
        src_filename = resolve_path(src_dir, src_basename)
        dst_filename = resolve_path(dst_dir, src_basename)
        print_func("Adding %s" % dst_filename)
        src_stat = d_src[src_basename]
        src_mode = stat_mode(src_stat)
        if not dry_run:
            if not mode_isdir(src_mode):
                cp(resolve_path(src_filename), resolve_path(dst_filename))
        if mode_isdir(src_mode):
            rsync(src_filename, dst_filename, mirror=mirror, dry_run=dry_run,
                  print_func=print_func, recursed=True, sync_hidden=sync_hidden)

    if mirror:  # May delete
        for dst_basename in to_del:  # In dest but not in source
            dst_filename = resolve_path(dst_dir, dst_basename)

and so on. I haven't finished finding all the places that this needs to be done yet though.

bereldhuin commented 2 years ago

I do have the same problem on Windows.

Is there a complete fix already ?

dhylands / rshell

cp -r and wildcard copy fail #172