Files missing when using copy-files

[GoogleCodeExporter]

What steps will reproduce the problem?
$ ls ./source/*.xml  | wc -l
62125
$ gcloud compute copy-files ./source/*.xml 
user@my-instance:/mnt/pd1/home/target --zone us-central1-b
$ ls /mnt/pd1/home/target/*.xml | wc -l
61347

What is the expected output? What do you see instead?
The tool missed 778 files. It should copy all the 62125 files.

What is the output of 'gcloud info'?
Google Cloud SDK [0.9.43]
Platform: [Linux, x86_64]
Python Version: [2.7.3 (default, Mar 13 2014, 11:03:55)  [GCC 4.7.2]]
Site Packages: [Disabled]
Installed Components:
  core: [2015.01.12]
  core-nix: [2014.10.20]
  compute: [2015.01.12]
  gsutil-nix: [4.6]
  gsutil: [4.7]
  bq: [2.0.18]
  dns: [2015.01.12]
  sql: [2015.01.12]
  gcutil: [1.16.5]
  gcutil-nix: [1.16.5]
  bq-nix: [2.0.18]
Please provide any additional information below.
I tried the same command on another directory with over 62 files and it missed 
779 files.

Original issue reported on code.google.com by khalid3ahmed on 6 Jun 2015 at 3:06

[GoogleCodeExporter]

I'm somewhat surprised this copies as many files as it does.  Most shells have 
a limit on the number of positional argument that they accept.  So when I tried 
to reproduce, I got:

  $ ls | wc -l
     62125

  $ gcloud compute copy-files --zone us-central1-f *.xml my-inst4:tmp
  -bash: /Users/jeffvaughan/google-cloud-sdk/bin/gcloud: Argument list too long

  $ bash --version
  bash --version
  GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin14)
  Copyright (C) 2007 Free Software Foundation, Inc.

I would suggest tarring and gz compressing the the xml files (use `tar czf`) 
and transferring one larger file.  This should be both more reliable and 
perform better.  You can uncompress on the destination using `gcloud compute 
ssh --command 'cd somewhere; tar xzf`.

Regardless, it's bad if gcloud + the environment is letting this be a silent 
failure.  Did you get an error messages?

Original comment by jeffvaughan@google.com on 8 Jun 2015 at 5:45

• Changed state: NeedsMoreInfo

[GoogleCodeExporter]

I tarred the files in one tar.bz2 file and uploaded it using copy-files when I 
had noticed the missing files. I'd expect copy-files to throw an error or 
reject the command but that never happened. It logged the regular info messages 
to stdout so it was hard to see if there's an error for over 62K files. It 
would be great to add the limitations of positional arguments and this issue to 
the documentation since it failed silently which is not good.

Original comment by khalid3ahmed on 8 Jun 2015 at 7:03

[GoogleCodeExporter]

I agree that failing silently is bad behavior.  Are you not getting any output 
from your shell about in this case?  I've been able to reproduce: files don't 
copy + an error from bash.  I haven't been able to reproduce files don't + no 
error.

What shell and os are you using?

Original comment by jeffvaughan@google.com on 8 Jun 2015 at 7:15

[GoogleCodeExporter]

I just spotted an error message that says: ERROR: (gcloud.compute.copy-files) 
[/usr/bin/scp] exited with return code [1]. Unfortunately, I can not tell if 
there are more error messages like this because it's beyond the scrollback 
buffer size of tmux.

I'm using Bash, Debian 7
$ bash --version
GNU bash, version 4.2.37(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.

Original comment by khalid3ahmed on 8 Jun 2015 at 7:39

[GoogleCodeExporter]

Hmmm, do you know or could you check what the the exit code for gcloud overall 
was?  (I.e. from running `echo $?` after the copy?)

Original comment by jeffvaughan@google.com on 9 Jun 2015 at 3:06

[GoogleCodeExporter]

Because gcloud does display an error message in this case, I'll mark this 
working as intended.

Original comment by z...@google.com on 13 Oct 2015 at 4:07

• Changed state: WorkingAsIntended