downloaded release did not mach expected checksum

[PierrickLozach]

I have an error when I try to download modules using puppet-forge-server. I copied the puppetlabs-dism tarball from PuppetLabs' forge into my modules folder and I get this:

puppet module install --module_repository http://localhost:8080 puppetlabs-dism Notice: Preparing to install into C:/ProgramData/PuppetLabs/puppet/etc/modules ... Notice: Downloading from http://localhost:8080 ... Error: Downloaded release for puppetlabs-dism did not match expected checksum Error: Try 'puppet help module install' for usage

Can you help?

[PierrickLozach]

Using puppet version 3.7.5

[PierrickLozach]

Here is the log in debug mode:

Debug: Runtime environment: puppet_version=3.7.5, ruby_version=2.1.5, run_mode=user, default_encoding=IBM437 Notice: Preparing to install into C:/ProgramData/PuppetLabs/puppet/etc/modules ... Notice: Downloading from http://localhost:8080 ... Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-dism Debug: Failed to load library 'pe_license' for feature 'pe_license' Info: Resolving dependencies ... Info: Preparing to install ... Debug: HTTP GET http://localhost:8080/v3/files/puppetlabs-dism-1.1.0.tar.gz Debug: Failed to load library 'pe_license' for feature 'pe_license' Error: Downloaded release for puppetlabs-dism did not match expected checksum Error: Try 'puppet help module install' for usage

[i11]

Hi! Thanks for reporting it! I'll have a look in a few hours.

[i11]

Would you mind sharing the arguments used to start the server? I wasn't able to reproduce it.

Server:

$ bundle exec bin/puppet-forge-server -x https://forgeapi.puppetlabs.com
[2015-05-18 14:19:30] INFO  Detecting API version for https://forgeapi.puppetlabs.com...
[2015-05-18 14:19:32] INFO   +- Daemonizing: false
[2015-05-18 14:19:32] INFO   |- Port: 8080
[2015-05-18 14:19:32] INFO   |- Host: default
[2015-05-18 14:19:32] INFO   |- Pidfile: default
[2015-05-18 14:19:32] INFO   |- Server: default
[2015-05-18 14:19:32] INFO   `- Backends:
[2015-05-18 14:19:32] INFO      - #<PuppetForgeServer::Backends::ProxyV3:0x007f904d0b5e70 @url="https://forgeapi.puppetlabs.com", @cache_dir="/var/folders/7k/xd7b183115x4p064035t0w38ybnmxc/T/puppet-forge-server/cache/93c654b7b5d3bff7decd0a021bf7cd0a28040c3c", @http_client=#<PuppetForgeServer::Http::HttpClient:0x007f904d0b5d58>, @log=#<PuppetForgeServer::Logger:0x007f904c1ac730 @loggers=[#<Logger:0x007f904c1ac578 @progname=nil, @level=1, @default_formatter=#<Logger::Formatter:0x007f904c1ac500 @datetime_format=nil>, @formatter=#<Proc:0x007f904c1ac050@/Users/me/tmp/puppet-forge-server/lib/puppet_forge_server/logger.rb:30>, @logdev=#<Logger::LogDevice:0x007f904c1ac410 @shift_size=nil, @shift_age=nil, @filename=nil, @dev=#<IO:<STDERR>>, @mutex=#<Logger::LogDevice::LogDeviceMutex:0x007f904c1ac3c0 @mon_owner=nil, @mon_count=0, @mon_mutex=#<Mutex:0x007f904c1ac348>>>>]>>
[2015-05-18 14:19:32] INFO  WEBrick 1.3.1
[2015-05-18 14:19:32] INFO  ruby 2.0.0 (2014-05-08) [universal.x86_64-darwin13]
[2015-05-18 14:19:32] INFO  WEBrick::HTTPServer#start: pid=10259 port=8080
::1 - - [18/May/2015:14:19:53 +0200] "GET /v3/releases?module=puppetlabs-dism HTTP/1.1" 200 4254 0.7635
localhost - - [18/May/2015:14:19:52 CEST] "GET /v3/releases?module=puppetlabs-dism HTTP/1.1" 200 4254
- -> /v3/releases?module=puppetlabs-dism
::1 - - [18/May/2015:14:19:54 +0200] "GET /v3/files/v3/files/puppetlabs-dism-1.1.0.tar.gz HTTP/1.1" 200 6577 0.8016
localhost - - [18/May/2015:14:19:53 CEST] "GET /v3/files/v3/files/puppetlabs-dism-1.1.0.tar.gz HTTP/1.1" 200 6577
- -> /v3/files/v3/files/puppetlabs-dism-1.1.0.tar.gz

Puppet module command output:

$ puppet module install --module_repository http://localhost:8080 puppetlabs-dism --debug
Notice: Preparing to install into /Users/me/.puppet/modules ...
Notice: Downloading from http://localhost:8080 ...
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-dism
Info: Resolving dependencies ...
Info: Preparing to install ...
Debug: HTTP GET http://localhost:8080/v3/files/v3/files/puppetlabs-dism-1.1.0.tar.gz
Debug: Executing 'gzip -dc /Users/me/.puppet/var/puppet-module/cache/puppetlabs-dism20150518-10348-cgfor2 | tar xof -'
Debug: Executing 'find . -type d -exec chmod 755 {} +'
Debug: Executing 'find . -type f -exec chmod a-wst {} +'
Debug: Executing 'chown -R 1018876844:1884119166 .'
Notice: Installing -- do not interrupt ...
/Users/me/.puppet/modules
└── puppetlabs-dism (v1.1.0)

[PierrickLozach]

Sure. I am on Windows 8.1 using ruby version 2.1.5p273

I ran "bundle exec C:\tools\ruby215\bin\puppet-forge-server -m modules".

I do not understand why you would want to proxy the puppetlabs forge but that might be the reason that it is failing. In my situation, I intend to run the private forge for development purposes and to not have to publish modules to Puppetlab's forge without testing them from a separate environment.

[i11]

Any use-case you can think of is absolutely valid, so proxying definitely is not a requirement. We proxy to achieve traffic reduction, establish security perimeters and for having a single end point both for private and public modules.

Feels like calculated MD5 digest is wrong. To be honest I never tried running it on Windows, so there might something I did not take into account even though it should be directly portable.

Would you mind conducting an experiment for me? Download the upstream puppetlabs-dism place it in your empty modules directory and try re-running puppet module command.

Unfortunately I'll need some more time to try it on Windows myself.

[i11]

It would be also great if you could publish the JSON payload you get for http://localhost:8080/v3/releases?module=puppetlabs-dism. It might contain some hints as well.

[PierrickLozach]

What do you use to calculate the MD5? I can check whether it's there or not.

What do you mean by the "upstream" puppetlabs-dism? I downloaded the .tar.gz file from the puppet forget and placed it in my modules directory. At the moment, I have no other files in that directory. The output is still the same:

puppet module install --module_repository http://localhost:8080 puppetlabs-dism --verbose --debug
Debug: Runtime environment: puppet_version=3.7.5, ruby_version=2.1.5, run_mode=user, default_encoding=IBM437
Notice: Preparing to install into C:/ProgramData/PuppetLabs/puppet/etc/modules ...
Notice: Downloading from http://localhost:8080 ...
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-dism
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Info: Resolving dependencies ...
Info: Preparing to install ...
Debug: HTTP GET http://localhost:8080/v3/files/puppetlabs-dism-1.1.0.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Error: Downloaded release for puppetlabs-dism did not match expected checksum
Error: Try 'puppet help module install' for usage

Here is the payload you asked for:

{"pagination":{"next":false,"total":1},"results":[{"uri":"/v3/releases/puppetlabs-dism-1.1.0","module":{"uri":"/v3/modules/puppetlabs-dism","name":"dism","owner":{"username":"puppetlabs","uri":"/v3/users/puppetlabs"}},"metadata":{"name":"puppetlabs-dism","version":"1.1.0","author":"puppetlabs","summary":"Windows DISM puppet module","license":"Apache License, Version 2.0","source":"https://github.com/puppetlabs/puppetlabs-dism","project_page":"https://github.com/puppetlabs/puppetlabs-dism","issues_url":"https://tickets.puppetlabs.com/browse/MODULES","dependencies":[],"types":[]},"version":"1.1.0","tags":["puppetlabs","dism"],"file_uri":"/v3/files/puppetlabs-dism-1.1.0.tar.gz","file_md5":"f915b326788a601dc17217277f2409c7"}]}

Just so you know, I was only testing your forge. I do intend to run it on Linux for my dev environment so this isn't an urgent request.

[i11]

Digest::MD5 class. Both forge and puppet cli uses it, so there shouldn't be any issues. At least not from the first glance. By upstream I meant the official forge, so you got it. Payload has MD5 in it, so that eliminates my suspicion about it being missing.

Another wild guess I would try is replacing line 76 in lib/puppet_forge_server/backends/directory.rb with :checksum => options[:with_checksum] == true ? Digest::MD5.file(path).hexdigest : nil,, which is exactly how puppet cli calculates the checksum.

I'm out of ideas if that doesn't work. I'll try to reproduce it with Windows in the evening.

[PierrickLozach]

Your last change fixed it. Here is the log:

puppet module install --module_repository http://localhost:8080 puppetlabs-dism --verbose --debug
Debug: Runtime environment: puppet_version=3.7.5, ruby_version=2.1.5, run_mode=user, default_encoding=IBM437
Notice: Preparing to install into C:/ProgramData/PuppetLabs/puppet/etc/modules ...
Notice: Downloading from http://localhost:8080 ...
Debug: HTTP GET http://localhost:8080/v3/releases?module=puppetlabs-dism
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Info: Resolving dependencies ...
Info: Preparing to install ...
Debug: HTTP GET http://localhost:8080/v3/files/puppetlabs-dism-1.1.0.tar.gz
Debug: Failed to load library 'pe_license' for feature 'pe_license'
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/CHANGELOG.md Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/checksums.json
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/Gemfile
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/lib/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/LICENSE
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/metadata.json

Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/Rakefile
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/README.md
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/spec/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/spec/fixtures/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/spec/spec_helper.rb
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/spec/unit/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/spec/unit/provider/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/spec/unit/provider/dism_spec.rb
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/spec/fixtures/dism_get_features
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/spec/fixtures/site.pp
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/lib/puppet/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/lib/puppet/provider/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/lib/puppet/type/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/lib/puppet/type/dism.rb
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/lib/puppet/provider/dism/
Debug: Extracting: C:/ProgramData/PuppetLabs/puppet/var/puppet-module/cache/tmp-unpacker20150518-7564-17e71al/puppetlabs-dism-1.1.0/lib/puppet/provider/dism/dism.rb
Notice: Installing -- do not interrupt ...
C:/ProgramData/PuppetLabs/puppet/etc/modules
â""â"?â"? puppetlabs-dism (v1.1.0)

[i11]

Nice! I will release the patch shortly. Thanks for trying it out!

[i11]

1.5.2 released.