search

jkeiser / knife-essentials

Knife commands to treat the Chef server like a filesystem
Apache License 2.0
81 stars 26 forks source link

knife download sometimes fails with EEXIST #98

Open jkeiser opened 11 years ago

jkeiser commented 11 years ago

I suspect parallelism. Just let this method continue silently or something.

Created data_bags Created data_bags/aws Created data_bags/aws/rs-preprod.json ERROR: Errno::EEXIST: File exists - /home/jkeiser/testit/data_bags

yoniLavi commented 11 years ago

I get this as well. I'm using 11.6.0, and when running knife download '*' it keeps failing similarly and leaving empty directories. When I then delete the directory mentioned in the error and retry, it gets further. So it takes me several such iterations to eventually finish the download.

jkeiser commented 11 years ago

I believe this was just fixed. If you see this continue, please reopen!

davekonopka commented 11 years ago

I'm still running into this with knife-essentials (1.5.3) installed running knife download . and knife download \* in an otherwise empty repo folder. It's seemingly random as to which folder triggers it.

jkeiser commented 11 years ago

OK, I'll take a look today.

davekonopka commented 11 years ago

Here's one thing I just tested. Against a Chef 10.20.0 server knife download . works without issue in an empty directory.

Against an 11.6.0 server it dies. Same setup: empty repo directory with just a .chef/ folder containing knife.rb. The exception is actually consistent:

ERROR: Errno::EEXIST: File exists - /Users/davek/projects/backup-test/clientsCreated cookbooks

I think the variation I was seeing in what folder the exception would hit came from some earlier tries against a partially populated repo folder.

jkeiser commented 11 years ago

The variation is actually almost certainly more to do with parallelism problems, so they will occur randomly.

On Tue, Aug 13, 2013 at 8:29 AM, Dave Konopka notifications@github.comwrote:

Here's one thing I just tested. Against a Chef 10.20.0 server knife download . works without issue in an empty directory.

Against an 11.6.0 server it dies. Same setup: empty repo directory with just a .chef/ folder containing knife.rb. The exception is actually consistent:

ERROR: Errno::EEXIST: File exists - /Users/davek/projects/backup-test/clientsCreated cookbooks

I think the variation I was seeing in what folder the exception would hit came from some earlier tries against a partially populated repo folder.

— Reply to this email directly or view it on GitHubhttps://github.com/jkeiser/knife-essentials/issues/98#issuecomment-22573290 .

luisbosque commented 10 years ago

This also happens to me. Chef - 10.16.4 knife-essentials - 1.5.5

I'm using "versioned_cookbooks true" The more cookbooks I have already downloaded the more frequently the error is raised. I'm at a point when I'm not able to download more cookbooks since the error is always raised in different cookbooks/versions.

luisbosque commented 10 years ago

Not sure if it's related to concurrency. I tried with --concurrency 1 and it also fails

luisbosque commented 10 years ago

Actually increasing (dramatically) the value of concurrency reduces the chance to get errors.

jkeiser commented 10 years ago

If it fails with concurrency 1, it's not concurrency at all. Would you mind posting a log with -VV on the command line of it failing with --concurrency 1? I can't be 100% sure I will know the error, but it could well help.

On Mon, Dec 16, 2013 at 8:12 AM, Luis Bosque notifications@github.comwrote:

Actually increasing (dramatically) the value of concurrency reduces the chance to get errors.

— Reply to this email directly or view it on GitHubhttps://github.com/jkeiser/knife-essentials/issues/98#issuecomment-30673272 .

lamont commented 10 years ago

Here's my repeatable crash. It occurs with anything from concurrency 1 to 50, and if I just re-run it seems to get stuck at the exact same directory each time. if I rmdir the (sometimes empty) directory and re-run it goes for a few minutes before getting stuck on the next one.

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system/file_system_entry.rb:53:in `mkdir': File exists - /home/users/me/transfer/cookbooks/cookbookWithLotsOfVersions-0.2.43/templates (Errno::EEXIST)
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system/file_system_entry.rb:53:in `create_child'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:313:in `copy_entries'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:356:in `block in copy_entries'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:94:in `call'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:94:in `process_input'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:69:in `each'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:421:in `to_a'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:421:in `parallel_do'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:355:in `copy_entries'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/file_system.rb:356:in `block in copy_entries'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:94:in `call'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:94:in `process_input'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:111:in `worker_loop'
    from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/knife-essentials-1.5.5/lib/chef_fs/parallelizer.rb:23:in `block (2 levels) in initialize'