search

poise / poise-archive

A Chef cookbook to unpack file archives like TAR and ZIP files.
Apache License 2.0
10 stars 13 forks source link

Downloading URL artifact can cause Errno::ENAMETOOLONG: File name too long #5

Open bbaugher opened 7 years ago

bbaugher commented 7 years ago

Looks like the code generates a unique name for the cached file by converting the URL into Base64. This can then easily go over the 255 character file name limit if the URL is long enough

(/tmp/kitchen/cache/cookbooks/poise-archive/files/halite_gem/poise_archive/archive_providers/base.rb line 84) had an error: Errno::ENAMETOOLONG: File name too long @ utime_internal - /tmp/kitchen/cache/aHR0cDovL3JlcG8uYmlnZGF0YS5jZXJuZXIuY29ycC9uZXh1cy9jb250ZW50L3JlcG9zaXRvcmllcy9iaWdkYXRhLXNuYXBzaG90L2NvbS9jZXJuZXIvdm9sdGUvdm9sdGUtdG9vbHMvMS4wLVNOQVBTSE9UL3ZvbHRlLXRvb2xzLTEuMC0yMDE2MTIwNi4xODA2MjctMjUtYXNzZW1ibHkudGFyLmd6_some-tools-1.0-20161206.180627-25-assembly.tar.gz
/opt/chef/embedded/lib/ruby/2.3.0/fileutils.rb:1154:in `utime'
/opt/chef/embedded/lib/ruby/2.3.0/fileutils.rb:1154:in `block in touch'
/opt/chef/embedded/lib/ruby/2.3.0/fileutils.rb:1151:in `each'
/opt/chef/embedded/lib/ruby/2.3.0/fileutils.rb:1151:in `touch'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/file_content_management/deploy/mv_unix.rb:34:in `create'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/provider/file.rb:375:in `block in do_create_file'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/mixin/why_run.rb:52:in `add_action'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/provider.rb:176:in `converge_by'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/provider/file.rb:374:in `do_create_file'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/provider/file.rb:153:in `action_create'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/provider.rb:145:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/resource.rb:622:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/runner.rb:69:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/runner.rb:97:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/runner.rb:97:in `each'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/runner.rb:97:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42/lib/chef/runner.rb:96:in `converge'
/tmp/kitchen/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
/tmp/kitchen/cache/cookbooks/poise-archive/files/halite_gem/poise_archive/archive_providers/base.rb:82:in `download_file'
/tmp/kitchen/cache/cookbooks/poise-archive/files/halite_gem/poise_archive/archive_providers/base.rb:59:in `action_unpack'
coderanger commented 7 years ago

Sad trombone dot gif, didn't think of this. I should probably check what the limit on Windows is too.

coderanger commented 7 years ago

Because you filed the bug you get to have an opinion, switch to a hash for anything over 255 chars (or whatever the max on Windows is)? It won't be as nicely readable but should be better than nothing.

bbaugher commented 7 years ago

I'm not totally sure I understand what we get out of adding anything like that to the file name.

coderanger commented 7 years ago

We need some kind of unique name for the cache file, the goal with Base64 was to make something that could be reversed to the original URL during manual debugging if needed, while a hash could not be, but is fixed-length.

bbaugher commented 7 years ago

I would have thought converting URL to the filename would be sufficient for most if not all. Generating a hash seems simplest. I don't think you lose a lot from doing that

coderanger commented 7 years ago

I did think that, but as you noticed this can make invalid filenames when very long ;-)

bbaugher commented 7 years ago

I meant converting http://www-eu.apache.org/dist/kafka/0.10.1.0/kafka_2.10-0.10.1.0.tgz into kafka_2.10-0.10.1.0.tgz.

coderanger commented 7 years ago

Unfortunately that is not unique enough, I ran into problems early on with downloads of the same name but in different folders.

coderanger commented 7 years ago

Anyways, I'll fix this up shortly.

tanner-bruce commented 7 years ago

What about hashing the URL with SHA1? We are also running into this problem. I can submit a PR if this solution is acceptable.

coderanger commented 7 years ago

I changed this a while ago to use a base64 of the resource name and only the last path component of the URL, you're still hitting something too long though?

gchpaco commented 7 years ago

Oh yes. Especially with consul-template. Why not just use Digest::SHA256.base64digest?

gchpaco commented 7 years ago

An example:

[2017-05-05T16:34:45+00:00] ERROR: consul_template_installation[0.16.0] (test_consul_template::default line 19) had an error: Errno::ENAMETOOLONG: poise_archive[https://releases.hashicorp.com/consul-template/0.16.0/consul-template_0.16.0_linux_amd64.zip] (/opt/kitchen/cache/cookbooks/consul_template/libraries/consul_template_installation_binary.rb line 42) had an error: Errno::ENAMETOOLONG: remote_file[/opt/kitchen/cache/aHR0cHM6Ly9yZWxlYXNlcy5oYXNoaWNvcnAuY29tL2NvbnN1bC10ZW1wbGF0ZS8wLjE2LjAvY29uc3VsLXRlbXBsYXRlXzAuMTYuMF9saW51eF9hbWQ2NC56aXA_consul-template_0.16.0_linux_amd64.zip] (/opt/kitchen/cache/cookbooks/poise-archive/files/halite_gem/poise_archive/archive_providers/base.rb line 84) had an error: Errno::ENAMETOOLONG: File name too long @ utime_internal - /opt/kitchen/cache/aHR0cHM6Ly9yZWxlYXNlcy5oYXNoaWNvcnAuY29tL2NvbnN1bC10ZW1wbGF0ZS8wLjE2LjAvY29uc3VsLXRlbXBsYXRlXzAuMTYuMF9saW51eF9hbWQ2NC56aXA_consul-template_0.16.0_linux_amd64.zip
coderanger commented 7 years ago

Ahh okay, so the part of the workaround I didn't mention is you can use the path property directly rather than the resource name for the long URL, I'll fix that in the next release to be smarter though.