GoogleCloudPlatform / gsutil

A command line tool for interacting with cloud storage services.
Apache License 2.0
873 stars 333 forks source link

gcloud storage rsync output is messy #1790

Open jsoref opened 3 months ago

jsoref commented 3 months ago

Windows 11 Version 23H2 (OS Build 22631.3737) Your current Google Cloud CLI version is: 480.0.0

PS F:\test> gcloud storage rsync   --no-clobber --ignore-symlinks --recursive  gs://sample-bucket-name/test-with-colons/ .
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.At gs://sample-bucket-name/test-with-colons/**, worker process 16680 thread 15056 listed 1...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_** to 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_$2$2WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming .\hello:cruel:world:how:are:you to .\hello$1cruel$1world$1how$1are$1youCopying gs://sample-bucket-name/test-with-colons/hello:cruel:world:how:are:you to file://.\hello$1cruel$1world$1how$1are$1you
  Completed files 1/1 | 5.0B/5.0B

Problems

There are missing line breaks

Ideally, the output would be closer to:

PS F:\test> gcloud storage rsync   --no-clobber --ignore-symlinks --recursive  gs://sample-bucket-name/test-with-colons/ .
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.
At gs://sample-bucket-name/test-with-colons/**, worker process 16680 thread 15056 listed 1...
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_** to 375b0fab3be7fc9544fbaa6c2b97da43ff6d2112.t-with-colons_$2$2
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file:__. to 3fec0319fc0cfde4a47aa40b533691f14e03e4a1.file$1__.
WARNING: The following characters are invalid in Windows file and directory names: /:*?"<>|
Renaming .\hello:cruel:world:how:are:you to .\hello$1cruel$1world$1how$1are$1you
Copying gs://sample-bucket-name/test-with-colons/hello:cruel:world:how:are:you to file://.\hello$1cruel$1world$1how$1are$1you
  Completed files 1/1 | 5.0B/5.0B

The choice of replacement characters is unfortunate

macOS has picked some replacement characters -- which means that people who use these tools on macOS to touch a FAT volume will already have things encoded in a certain way -- specifically, : is encoded as U+F022, not $1.

The output is more or less unreadable

What is the :__ stuff -- where did it come from? Where is it going? When I look on disk, I can't find it. Testing with an NTFS volume doesn't show any streams in the file, so it doesn't appear to be a stream.

Where is the .t-with-colons_$2$2 ? I can't find it.

At most, I'd expect one warning per file.

I'd also expect to get 0 warnings for a basic case of gcloud storage rsync --no-clobber --ignore-symlinks --recursive gs://sample-bucket-name/test-without-colons .

But even in that case (and using NTFS as the destination volume), I'm getting 3 WARNINGS.