search

rnelson0 / puppet-generate-puppetfile

Generate or update a Puppetfile for use with r10k or librarian-puppet. Optionally, create a fixtures file for rspec-puppet.
92 stars 22 forks source link

generate-puppetfile downgrades existing Puppetfile #55

Open vchepkov opened 7 years ago

vchepkov commented 7 years ago

I receive strange results. here is the original file

#
# Forge modules
#
forge 'http://forge.puppetlabs.com'

mod 'puppet/extlib', '1.1.0'
mod 'puppet/make', '1.1.0'
mod 'puppet/puppetboard', '2.9.0'
mod 'puppet/r10k', '5.0.2'
mod 'puppetlabs/apache', '1.11.0'
mod 'puppetlabs/concat', '2.2.1'
mod 'puppetlabs/gcc', '0.3.0'
mod 'puppetlabs/git', '0.5.0'
mod 'puppetlabs/inifile', '1.6.0'
mod 'puppetlabs/java', '1.6.0'
mod 'puppetlabs/ntp', '6.1.0'
mod 'puppetlabs/postgresql', '4.9.0'
mod 'puppetlabs/puppetdb', '5.1.2'
mod 'puppetlabs/puppetserver_gem', '0.2.0'
mod 'puppetlabs/ruby', '0.6.0'
mod 'puppetlabs/stdlib', '4.16.0'
mod 'puppetlabs/vcsrepo', '1.5.0'
mod 'stahnma/epel', '1.2.2'
mod 'theforeman/puppet', '7.1.1'
#
# Github modules
#
mod 'mcollective',
  :git    => 'https://github.com/vchepkov/puppet-mcollective.git',
  :branch => 'development'
mod 'python',
  :git    => 'https://github.com/vchepkov/puppet-python.git',
  :branch => 'setuptools34'

Here is the output

$ generate-puppetfile -p Puppetfile 

Installing modules. This may take a few minutes.

Your Puppetfile has been generated. Copy and paste between the markers:

=======================================================================
forge 'http://forge.puppetlabs.com'

# Modules discovered by generate-puppetfile
mod 'gentoo/portage', '2.3.0'
mod 'puppet/extlib', '1.1.0'
mod 'puppet/make', '1.1.0'
mod 'puppet/puppetboard', '2.9.0'
mod 'puppet/r10k', '5.0.2'
mod 'puppetlabs/apache', '1.3.0'
mod 'puppetlabs/concat', '4.0.0'
mod 'puppetlabs/gcc', '0.3.0'
mod 'puppetlabs/git', '0.5.0'
mod 'puppetlabs/inifile', '1.6.0'
mod 'puppetlabs/java', '1.6.0'
mod 'puppetlabs/ntp', '6.1.0'
mod 'puppetlabs/puppetserver_gem', '0.2.0'
mod 'puppetlabs/ruby', '0.6.0'
mod 'puppetlabs/stdlib', '4.16.0'
mod 'puppetlabs/vcsrepo', '1.5.0'
mod 'stahnma/epel', '1.2.2'
mod 'stankevich/python', '1.18.2'
mod 'theforeman/concat_native', '1.5.0'
mod 'theforeman/puppet', '3.0.0'
# Discovered elements from existing Puppetfile
#
# Forge modules
#
#
# Github modules
#
mod 'mcollective',
  :git    => 'https://github.com/vchepkov/puppet-mcollective.git',
  :branch => 'development'
mod 'python',
  :git    => 'https://github.com/vchepkov/puppet-python.git',
  :branch => 'setuptools34'
=======================================================================

As you can see it downgrades apache and puppet for some reasons

rnelson0 commented 7 years ago

This is usually due to the underlying use of puppet module install - some module has a restrictive dependency on the apache and puppet modules. If you can run grep apache ${pathtomodules}/*/metadata.json, it may show you where that is coming from.

djjudas21 commented 6 years ago

I'm experiencing this too. Any way to work around it?

djjudas21 commented 6 years ago

I found that generate-puppetfile downgraded one of my modules from 0.2.4 to 0.1.0. I searched all my metadata.json files as you suggested. Several mentioned this module, most specified >= 0.1.0, one specified >= 0.1.1 but none of them specified a hard dependency or an upper bound. What's going wrong?

rnelson0 commented 6 years ago

Can you run the earlier suggest command as well as puppet module list and attach that output please?

djjudas21 commented 6 years ago

OK. In my example, running generate-puppetfile downgraded jgazeley/motd from 0.2.4 to 0.1.0. Here's the grep on my puppet server:

[root@puppet-prod production]# grep -r -A 1 motd */*/metadata.json
local_modules/eduroam/metadata.json:      "name":"jgazeley/motd",
local_modules/eduroam/metadata.json-      "version_requirement":">= 0.1.0"
--
local_modules/its_gitlab/metadata.json:      "name":"jgazeley/motd",
local_modules/its_gitlab/metadata.json-      "version_requirement":">= 0.1.0"
--
local_modules/mariadb/metadata.json:      "name":"jgazeley/motd",
local_modules/mariadb/metadata.json-      "version_requirement":">= 0.1.0"
--
local_modules/rainbow/metadata.json:      "name":"jgazeley/motd",
local_modules/rainbow/metadata.json-      "version_requirement":">= 0.1.0"
--
local_modules/resnet/metadata.json:      "name":"jgazeley/motd",
local_modules/resnet/metadata.json-      "version_requirement":">= 0.1.0"
--
local_modules/uob_dhcp/metadata.json:      "name":"jgazeley/motd",
local_modules/uob_dhcp/metadata.json-      "version_requirement":">= 0.1.1"
--
local_modules/uob_dns_rpz/metadata.json:      "name":"jgazeley/motd",
local_modules/uob_dns_rpz/metadata.json-      "version_requirement":">= 0.1.0"
--
local_modules/uob_nagios/metadata.json:      "name":"jgazeley/motd",
local_modules/uob_nagios/metadata.json-      "version_requirement":">= 0.1.0"
--
local_modules/uob_resnet_vhosts/metadata.json:      "name":"jgazeley/motd",
local_modules/uob_resnet_vhosts/metadata.json-      "version_requirement":">= 0.1.0"
--
local_modules/uob_tftp/metadata.json:      "name":"jgazeley/motd",
local_modules/uob_tftp/metadata.json-      "version_requirement":">= 0.1.0"
--
modules/iperf/metadata.json:    {"name":"jgazeley/motd","version_requirement":">= 0.1.0"}
modules/iperf/metadata.json-  ]
--
modules/motd/metadata.json:  "name": "jgazeley/motd",
modules/motd/metadata.json-  "version": "0.2.4",
--
modules/motd/metadata.json:  "source": "https://github.com/djjudas21/puppet-motd",
modules/motd/metadata.json:  "project_page": "https://github.com/djjudas21/puppet-motd",
modules/motd/metadata.json:  "issues_url": "https://github.com/djjudas21/puppet-motd/issues",
modules/motd/metadata.json-  "dependencies": [

And here's the output of puppet module list (excuse the legacy crap in local_modules):

[root@puppet-prod production]# puppet module list
Warning: Module 'puppet-logrotate' (v2.0.0) fails to meet some dependencies:
  'jgazeley-yumupdate' (v0.3.0) requires 'puppet-logrotate' (>= 3.0.0 < 4.0.0)
Warning: Module 'puppetlabs-apache' (v2.0.0) fails to meet some dependencies:
  'jgazeley-mod_auth_cas' (v0.4.0) requires 'puppetlabs-apache' (v1.x)
Warning: Module 'puppetlabs-apt' (v3.0.0) fails to meet some dependencies:
  'vshn-gitlab' (v1.13.3) requires 'puppetlabs-apt' (>=2.1.0 <3.0.0)
  'puppetlabs-postgresql' (v4.9.0) requires 'puppetlabs-apt' (>=1.8.0 <3.0.0)
  'puppetlabs-puppet_agent' (v1.4.1) requires 'puppetlabs-apt' (>= 2.0.1 < 3.0.0)
  'arioch-redis' (v3.0.0) requires 'puppetlabs-apt' (>= 2.3.0 <3.0.0)
  'razorsedge-vmwaretools' (v6.0.0) requires 'puppetlabs-apt' (>=2.0.0 <3.0.0)
Warning: Module 'puppetlabs-concat' (v3.0.0) fails to meet some dependencies:
  'davidb-dhcpd' (v0.4.1) requires 'puppetlabs-concat' (>= 1.1.0 < 2.0.0)
  'calmenergy-fail2ban' (v1.1.0) requires 'puppetlabs-concat' (>= 2.2.0 < 3.0.0)
  'jonnyx-logwatch' (v2.0.0) requires 'puppetlabs-concat' (>= 2.2.1 < 3.0.0)
  'pfmooney-mit_krb5' (v0.0.8) requires 'puppetlabs-concat' (>=1.0.0 <3.0.0)
  'jgazeley-motd' (v0.2.4) requires 'puppetlabs-concat' (>= 1.2.1 < 2.0.0)
  'jgazeley-nagios' (v0.1.0) requires 'puppetlabs-concat' (>= 4.0.0)
  'derdanne-nfs' (v2.0.3) requires 'puppetlabs-concat' (>= 1.1.2 < 3.0.0)
  'puppetlabs-postgresql' (v4.9.0) requires 'puppetlabs-concat' (>= 1.1.0 <3.0.0)
  'saz-ssh' (v3.0.1) requires 'puppetlabs-concat' (>= 1.2.5 < 3.0.0)
Warning: Missing dependency 'apache':
  'uob_team_wireless' (v0.1.0) requires 'apache' (>= 1.4.1)
Warning: Missing dependency 'darin-zypprepo':
  'razorsedge-vmwaretools' (v6.0.0) requires 'darin-zypprepo' (v1.x)
Warning: Missing dependency 'dns':
  'eduroam' (v0.1.0) requires 'dns' (>= 0.1.0)
  'Install a ResNet DNS server' (v0.1.0) requires 'dns' (>= 0.1.0)
Warning: Missing dependency 'eduroam':
  'uob_team_wireless' (v0.1.0) requires 'eduroam' (>= 0.1.0)
Warning: Missing dependency 'herculesteam-augeasproviders_core':
  'camptocamp-openldap' (v1.15.0) requires 'herculesteam-augeasproviders_core' (v2.x)
  'arioch-redis' (v3.0.0) requires 'herculesteam-augeasproviders_core' (>=2.1.0 < 3.0.0)
Warning: Missing dependency 'herculesteam-augeasproviders_shellvar':
  'camptocamp-openldap' (v1.15.0) requires 'herculesteam-augeasproviders_shellvar' (v2.x)
Warning: Missing dependency 'herculesteam-augeasproviders_sysctl':
  'arioch-redis' (v3.0.0) requires 'herculesteam-augeasproviders_sysctl' (>=2.1.0 < 3.0.0)
Warning: Missing dependency 'jfryman-selinux':
  'mariadb' (v0.1.0) requires 'jfryman-selinux' (>= 0.2.3)
  'uobnetops-sauron' (v0.1.0) requires 'jfryman-selinux' (>= 0.4.0)
  'nagios' (v0.1.0) requires 'jfryman-selinux' (>= 0.2.3)
Warning: Missing dependency 'jgazeley-syslog':
  'dhcp_new' (v0.1.0) requires 'jgazeley-syslog' (>= 0.1.0)
  'eduroam' (v0.1.0) requires 'jgazeley-syslog' (>= 0.1.0)
Warning: Missing dependency 'mariadb':
  'eduroam' (v0.1.0) requires 'mariadb' (>= 0.1.0)
  'uob_team_wireless' (v0.1.0) requires 'mariadb' (>= 0.1.0)
Warning: Missing dependency 'nagios':
  'uob_freeradius' (v0.1.0) requires 'nagios' (>= 0.1.0)
Warning: Missing dependency 'oracleinstantclient':
  'uob_resnet_vhosts' (v0.1.0) requires 'oracleinstantclient' (>= 0.1.0)
Warning: Missing dependency 'puppet-staging':
  'puppetlabs-mysql' (v3.11.0) requires 'puppet-staging' (>= 1.0.1 < 3.0.0)
Warning: Missing dependency 'puppetlabs-transition':
  'puppetlabs-puppet_agent' (v1.4.1) requires 'puppetlabs-transition' (>= 0.1.1 < 0.2.0)
Warning: Missing dependency 'samba':
  'uob_freeradius' (v0.1.0) requires 'samba' (>= 0.1.0)
Warning: Missing dependency 'selinux':
  'uob_team_wireless' (v0.1.0) requires 'selinux' (>= 0.2.3)
Warning: Missing dependency 'spuder-gitlab':
  'its_gitlab' (v0.1.0) requires 'spuder-gitlab' (>= 2.3.6)
Warning: Missing dependency 'stm-debconf':
  'saz-timezone' (v3.4.0) requires 'stm-debconf' (>= 1.0.0)
Warning: Missing dependency 'syslog':
  'dhcp' (v0.1.0) requires 'syslog' (>= 0.1.0)
Warning: Missing dependency 'this-sysctl':
  'uob_network' (v0.1.0) requires 'this-sysctl' (>= 1.0.2)
Warning: Missing dependency 'uob_apache':
  'uob_dns_rpz' (v0.1.0) requires 'uob_apache' (>= 0.1.0)
  'uob_resnet_vhosts' (v0.1.0) requires 'uob_apache' (>= 0.1.0)
  'uob_team_wireless' (v0.1.0) requires 'uob_apache' (>= 0.1.0)
  'uob_tftp' (v0.1.0) requires 'uob_apache' (>= 0.1.0)
Warning: Missing dependency 'vcsrepo':
  'uob_team_wireless' (v0.1.0) requires 'vcsrepo' (>= 1.2.0)
/etc/puppetlabs/code/environments/production/local_modules
├── Install a ResNet DNS server (v0.1.0)
├── build (v0.1.0)
├── consoles (v0.1.0)
├── cron (v0.1.0)
├── dhcp (v0.1.0)
├── dhcp_new (v0.1.0)
├── dns (v0.1.0)
├── eduroam (v0.1.0)
├── glue (v0.1.0)
├── its_gitlab (v0.1.0)
├── logchecker (???)
├── mariadb (v0.1.0)
├── nagios (v0.1.0)
├── rainbow (v0.1.0)
├── ssh (v0.1.0)
├── uob_dhcp (v0.1.0)
├── uob_dns (v0.1.0)
├── uob_dns_rpz (v0.1.0)
├── uob_firewall (v0.1.0)
├── uob_freeradius (v0.1.0)
├── uob_kerberos (v0.1.0)
├── uob_network (v0.1.0)
├── uob_networker (???)
├── uob_pe (???)
├── uob_postfix (v0.1.0)
├── uob_redis (???)
├── uob_resnet_vhosts (v0.1.0)
├── uob_rsyslog (???)
├── uob_team_wireless (v0.1.0)
├── uob_tftp (v0.1.0)
├── uob_www_wireless (???)
├── users (v0.1.0)
└── vmps (???)
/etc/puppetlabs/code/environments/production/modules
├── abrader-gms (v1.0.3)
├── alexharvey-disable_transparent_hugepage (v1.2.1)
├── alexharvey-firewall_multi (v1.5.0)
├── arioch-redis (v3.0.0)
├── brentschler-hddtemp (v1.0.3)
├── calmenergy-fail2ban (v1.1.0)
├── camptocamp-augeas (v1.6.0)
├── camptocamp-dell (v0.6.1)
├── camptocamp-kmod (v2.1.0)
├── camptocamp-openldap (v1.15.0)
├── camptocamp-systemd (v0.4.0)
├── ceh-freetds (v1.1.0)
├── dalen-dnsquery (v3.0.0)
├── dalen-puppetdbquery (v2.3.0)
├── danfoster-networker (v0.1.0)
├── davidb-dhcpd (v0.4.1)
├── derdanne-nfs (v2.0.3)
├── garethr-docker (v5.3.0)
├── hunner-wordpress (v1.0.0)
├── inkblot-ipcalc (v2.2.0)
├── jgazeley-django (v0.1.0)
├── jgazeley-ekahau_throughput_server (v0.1.2)
├── jgazeley-freeradius (v3.4.1)
├── jgazeley-galera_innoptimizer (v0.1.0)
├── jgazeley-iperf (v0.2.1)
├── jgazeley-lm_sensors (v0.3.0)
├── jgazeley-mod_auth_cas (v0.4.0)
├── jgazeley-motd (v0.2.4)
├── jgazeley-nagios (v0.1.0)
├── jgazeley-oracleinstantclient (v1.0.0)
├── jgazeley-percona_toolkit (v0.1.0)
├── jgazeley-rpmgpgkey (v0.1.0)
├── jgazeley-speedtest (v0.2.0)
├── jgazeley-sqlmap (v0.1.0)
├── jgazeley-winbind (v0.4.0)
├── jgazeley-yumupdate (v0.3.0)
├── jonnyx-logwatch (v2.0.0)
├── juniorsysadmin-irqbalance (v1.1.0)
├── leinaddm-htpasswd (v0.0.3)
├── maestrodev-wget (v1.7.3)
├── npwalker-pe_metric_curl_cron_jobs (v2.0.0)
├── pcfens-ca_cert (v1.6.1)
├── pfmooney-mit_krb5 (v0.0.8)
├── puppet-logrotate (v2.0.0)  invalid
├── puppet-selinux (v1.1.0)
├── puppetlabs-apache (v2.0.0)  invalid
├── puppetlabs-apt (v3.0.0)  invalid
├── puppetlabs-concat (v3.0.0)  invalid
├── puppetlabs-firewall (v1.9.0)
├── puppetlabs-gcc (v0.3.0)
├── puppetlabs-git (v0.5.0)
├── puppetlabs-inifile (v1.6.0)
├── puppetlabs-java (v1.6.0)
├── puppetlabs-mysql (v3.11.0)
├── puppetlabs-ntp (v6.2.0)
├── puppetlabs-pe_gem (v0.2.0)
├── puppetlabs-postgresql (v4.9.0)
├── puppetlabs-puppet_agent (v1.4.1)
├── puppetlabs-ruby (v0.6.0)
├── puppetlabs-stdlib (v4.17.0)
├── puppetlabs-tftp (v0.2.3)
├── puppetlabs-vcsrepo (v1.5.0)
├── puppetlabs-xinetd (v2.0.0)
├── razorsedge-network (v3.11.1)
├── razorsedge-vmwaretools (v6.0.0)
├── resnet-resnet_repo (v0.1.0)
├── saz-dnsmasq (v1.3.0)
├── saz-resolv_conf (v3.1.0)
├── saz-rsyslog (v5.0.0)
├── saz-ssh (v3.0.1)
├── saz-sudo (v4.1.0)
├── saz-timezone (v3.4.0)
├── stahnma-epel (v1.2.2)
├── stankevich-python (v1.18.2)
├── stm-haveged (v0.2.0)
├── thias-postfix (v0.3.4)
├── thias-sysctl (v1.0.6)
├── thrnio-ip (v1.0.0)
├── treydock-gpg_key (v0.0.4)
├── treydock-repo_centos (v4.0.0)
├── unibet-forge_server (v0.4.0)
├── uob-addressspace (v0.1.0)
├── uob-cacerts (v0.2.0)
├── uob-uob_classifier (v0.0.1)
├── uobnetops-sauron (v0.1.0)
├── vshn-gitlab (v1.13.3)
├── wolfspyre-vmware_puppetfact (v0.2.2)
├── yelp-netstdlib (v0.0.1)
├── zleslie-permute (v0.0.4)
└── zleslie-pxe (v0.2.0)
/etc/puppetlabs/code/environments/production/site
├── profile (???)
└── role (???)
/opt/puppetlabs/puppet/modules
├── puppetlabs-pe_accounts (v2017.1.0-rc0-2-gbfbde9a)
├── puppetlabs-pe_concat (v1.1.2-2-g11b9527)
├── puppetlabs-pe_console_prune (v0.1.1-9-gfc256c0)
├── puppetlabs-pe_hocon (v2016.2.0)
├── puppetlabs-pe_infrastructure (v2017.2.0)
├── puppetlabs-pe_inifile (v2016.5.0-9-gbef7515)
├── puppetlabs-pe_install (v2017.2.2-25-g75b8a66)
├── puppetlabs-pe_java_ks (v2016.4.0-2-g842e297)
├── puppetlabs-pe_manager (v2017.2.2-14-g44e4493)
├── puppetlabs-pe_nginx (v2017.1.0-4-gdad6201)
├── puppetlabs-pe_postgresql (v2016.5.0-4-g5613ac4)
├── puppetlabs-pe_puppet_authorization (v2016.2.0-rc1-2-gffbef87)
├── puppetlabs-pe_r10k (v2016.2.0-2-g5e70a18)
├── puppetlabs-pe_razor (v1.0.1-8-g8ce6a25)
├── puppetlabs-pe_repo (v2017.2.2-47-g6870b1b)
├── puppetlabs-pe_staging (v2015.3.0)
├── puppetlabs-pe_support_script (v2016.5.0-93-g61f8808)
└── puppetlabs-puppet_enterprise (v2017.2.2-38-g6fd4af6)
rnelson0 commented 6 years ago

@vchepkov Are you still seeing this? I do not see a problem now, running from HEAD (which should not have any changes affecting use of PMT):

[rnelson0@build03 puppet-generate-puppetfile:master]$ be generate-puppetfile -p Puppetfile

Installing modules. This may take a few minutes.

There was a problem with the module name 'puppet/r10k'.
  Check that modules exist as under the listed name, and/or your connectivity to the puppet forge.

Here is the PARTIAL Puppetfile that would have been generated.

Your Puppetfile has been generated. Copy and paste between the markers:

=======================================================================
forge 'https://forge.puppet.com'

# Modules discovered by generate-puppetfile
mod 'gentoo/portage',                  '2.3.0'
mod 'puppet/archive',                  '2.2.0'
mod 'puppet/extlib',                   '2.0.1'
mod 'puppet/make',                     '2.0.0'
mod 'puppet/puppetboard',              '4.0.0'
mod 'puppetlabs/apache',               '2.3.0'
mod 'puppetlabs/concat',               '4.1.1'
mod 'puppetlabs/gcc',                  '0.3.0'
mod 'puppetlabs/git',                  '0.5.0'
mod 'puppetlabs/hocon',                '1.0.0'
mod 'puppetlabs/inifile',              '2.1.1'
mod 'puppetlabs/java',                 '2.3.0'
mod 'puppetlabs/ntp',                  '7.0.0'
mod 'puppetlabs/puppet_authorization', '0.4.0'
mod 'puppetlabs/puppetserver_gem',     '1.0.0'
mod 'puppetlabs/ruby',                 '1.0.0'
mod 'puppetlabs/stdlib',               '4.24.0'
mod 'puppetlabs/vcsrepo',              '1.5.0'
mod 'stahnma/epel',                    '1.3.0'
mod 'stankevich/python',               '1.18.2'
mod 'theforeman/concat_native',        '1.5.0'
mod 'theforeman/puppet',               '8.1.0'
# Discovered elements from existing Puppetfile
#
# Forge modules
#
#
# Github modules
#
mod 'mcollective',
  :git    => 'https://github.com/vchepkov/puppet-mcollective.git',
  :branch => 'development'
mod 'python',
  :git    => 'https://github.com/vchepkov/puppet-python.git',
  :branch => 'setuptools34'
=======================================================================

@djjudas21 I don't have your full Puppetfile, so just chopping it down to the motd module appears to work fine:

[rnelson0@build03 puppet-generate-puppetfile:master]$ be generate-puppetfile -p Puppetfile

Installing modules. This may take a few minutes.

Your Puppetfile has been generated. Copy and paste between the markers:

=======================================================================
forge 'https://forge.puppet.com'

# Modules discovered by generate-puppetfile
mod 'jgazeley/motd',     '0.2.4'
mod 'puppetlabs/concat', '1.2.5'
mod 'puppetlabs/stdlib', '4.24.0'
=======================================================================

If neither of you can replicate, I'll close it out (maybe the forge gave the wrong results?). If you do, provide an updated Puppetfile and I'll re-investigate.

Thanks, and apologies for taking so long to respond on this!

djjudas21 commented 6 years ago

Thanks for your response - I'll test after Christmas :+1:

vchepkov commented 6 years ago

It was probably caused by some modules, which were since updated. I don't see it anymore

rnelson0 commented 6 years ago

@djjudas21 Have you had a chance to test this yet?

djjudas21 commented 6 years ago

I tested a newer version of the gem than I had previously, but haven't tested git head. I'll try that now...

djjudas21 commented 6 years ago

Oh I see gem 1.0.0 is already the latest. It seems to be doing something different now, and bombing out before it actually gets as far as updating versions.

[jg4461@it050546 resnet-puppet]$ generate-puppetfile -v
generate-puppetfile v1.0.0
[jg4461@it050546 resnet-puppet]$ generate-puppetfile -c -p Puppetfile

Installing modules. This may take a few minutes.

There was a problem with the module name 'abrader/gms'.
There was a problem with the module name 'alexharvey/disable_transparent_hugepage'.
There was a problem with the module name 'alexharvey/firewall_multi'.
There was a problem with the module name 'arioch/redis'.
There was a problem with the module name 'camptocamp/augeas'.
There was a problem with the module name 'camptocamp/dell'.
There was a problem with the module name 'camptocamp/kmod'.
There was a problem with the module name 'camptocamp/openldap'.
There was a problem with the module name 'camptocamp/systemd'.
There was a problem with the module name 'ceh/freetds'.
There was a problem with the module name 'dalen/dnsquery'.
There was a problem with the module name 'dalen/puppetdbquery'.
There was a problem with the module name 'darin/zypprepo'.
There was a problem with the module name 'davidb/dhcpd'.
There was a problem with the module name 'derdanne/nfs'.
There was a problem with the module name 'garethr/docker'.
There was a problem with the module name 'herculesteam/augeasproviders_core'.
There was a problem with the module name 'herculesteam/augeasproviders_shellvar'.
There was a problem with the module name 'herculesteam/augeasproviders_sysctl'.
There was a problem with the module name 'inkblot/ipcalc'.
There was a problem with the module name 'jgazeley/django'.
There was a problem with the module name 'jgazeley/ekahau_throughput_server'.
There was a problem with the module name 'jgazeley/freeradius'.
There was a problem with the module name 'jgazeley/galera_innoptimizer'.
There was a problem with the module name 'jgazeley/hddtemp'.
There was a problem with the module name 'jgazeley/iperf'.
There was a problem with the module name 'jgazeley/lm_sensors'.
There was a problem with the module name 'jgazeley/mod_auth_cas'.
There was a problem with the module name 'jgazeley/motd'.
There was a problem with the module name 'jgazeley/nagios'.
There was a problem with the module name 'jgazeley/networker'.
There was a problem with the module name 'jgazeley/oracleinstantclient'.
There was a problem with the module name 'jgazeley/percona_toolkit'.
There was a problem with the module name 'jgazeley/rpmgpgkey'.
There was a problem with the module name 'jgazeley/sqlmap'.
There was a problem with the module name 'jgazeley/winbind'.
There was a problem with the module name 'jgazeley/yumupdate'.
There was a problem with the module name 'jonnyx/logwatch'.
There was a problem with the module name 'juniorsysadmin/irqbalance'.
There was a problem with the module name 'leinaddm/htpasswd'.
There was a problem with the module name 'lwf/remote_file'.
There was a problem with the module name 'maestrodev/wget'.
There was a problem with the module name 'pcfens/ca_cert'.
There was a problem with the module name 'pfmooney/mit_krb5'.
There was a problem with the module name 'puppet/archive'.
There was a problem with the module name 'puppet/logrotate'.
There was a problem with the module name 'puppet/selinux'.
There was a problem with the module name 'puppet/staging'.
There was a problem with the module name 'puppetlabs/apache'.
There was a problem with the module name 'puppetlabs/apt'.
There was a problem with the module name 'puppetlabs/concat'.
There was a problem with the module name 'puppetlabs/firewall'.
There was a problem with the module name 'puppetlabs/gcc'.
There was a problem with the module name 'puppetlabs/git'.
There was a problem with the module name 'puppetlabs/inifile'.
There was a problem with the module name 'puppetlabs/java'.
There was a problem with the module name 'puppetlabs/mysql'.
There was a problem with the module name 'puppetlabs/ntp'.
There was a problem with the module name 'puppetlabs/pe_gem'.
There was a problem with the module name 'puppetlabs/postgresql'.
There was a problem with the module name 'puppetlabs/puppet_agent'.
There was a problem with the module name 'puppetlabs/ruby'.
There was a problem with the module name 'puppetlabs/stdlib'.
There was a problem with the module name 'puppetlabs/tftp'.
There was a problem with the module name 'puppetlabs/translate'.
There was a problem with the module name 'puppetlabs/vcsrepo'.
There was a problem with the module name 'puppetlabs/xinetd'.
There was a problem with the module name 'razorsedge/network'.
There was a problem with the module name 'razorsedge/vmwaretools'.
There was a problem with the module name 'saz/dnsmasq'.
There was a problem with the module name 'saz/resolv_conf'.
There was a problem with the module name 'saz/rsyslog'.
There was a problem with the module name 'saz/ssh'.
There was a problem with the module name 'saz/sudo'.
There was a problem with the module name 'saz/timezone'.
There was a problem with the module name 'stahnma/epel'.
There was a problem with the module name 'stankevich/python'.
There was a problem with the module name 'stm/debconf'.
There was a problem with the module name 'stm/haveged'.
There was a problem with the module name 'thias/postfix'.
There was a problem with the module name 'thias/sysctl'.
There was a problem with the module name 'thrnio/ip'.
There was a problem with the module name 'treydock/gpg_key'.
There was a problem with the module name 'treydock/repo_centos'.
There was a problem with the module name 'unibet/forge_server'.
There was a problem with the module name 'wolfspyre/vmware_puppetfact'.
There was a problem with the module name 'yelp/netstdlib'.
There was a problem with the module name 'zleslie/permute'.
There was a problem with the module name 'zleslie/pxe'.
  Check that modules exist as under the listed name, and/or your connectivity to the puppet forge.

Here is the PARTIAL Puppetfile that would have been generated.

Your Puppetfile has been generated. Copy and paste between the markers:

=======================================================================
forge 'https://forge.puppet.com'

# Modules discovered by generate-puppetfile
# Discovered elements from existing Puppetfile
mod 'npwalker-pe_metric_curl_cron_jobs', '4.6.0'
#897 addressspace
mod 'addressspace',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-addressspace.git',
  :tag => '0.1.0'
#898 cacerts
mod 'cacerts',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-cacerts.git',
  :tag => '0.2.1'
#900 fail2ban
mod 'fail2ban',
  :git => 'https://github.com/uobnetops/calmenergy-fail2ban.git',
  :ref => 'ed0233261c898eef72e0da91d5b1b3eb72c232be'
#904 gitlab
mod 'gitlab',
  :git => 'https://github.com/djjudas21/puppet-gitlab.git',
  :ref => '739fc01'
#906 sauron
mod 'sauron',
  :git => 'https://github.com/uobnetops/puppet-sauron.git',
  :ref => 'master'
#907 uob_classifier
mod 'uob_classifier',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-uob_classifier.git',
  :ref => '0.1.3'
#908 wordpress
mod 'wordpress',
  :git => 'https://github.com/uobnetops/puppet-wordpress.git',
  :ref => 'uob'
=======================================================================

And here's my Puppetfile, which was generated by a previous version, I think 0.1.0:

forge 'https://forge.puppet.com'

# Modules discovered by generate-puppetfile
mod 'abrader/gms', '1.0.3'
mod 'alexharvey/disable_transparent_hugepage', '1.2.1'
mod 'alexharvey/firewall_multi', '1.10.0'
mod 'arioch/redis', '3.1.0'
mod 'camptocamp/augeas', '1.6.0'
mod 'camptocamp/dell', '1.0.0'
mod 'camptocamp/kmod', '2.2.0'
mod 'camptocamp/openldap', '1.16.1'
mod 'camptocamp/systemd', '1.1.0'
mod 'ceh/freetds', '1.1.0'
mod 'dalen/dnsquery', '3.0.0'
mod 'dalen/puppetdbquery', '3.0.1'
mod 'darin/zypprepo', '1.0.2'
mod 'davidb/dhcpd', '0.4.1'
mod 'derdanne/nfs', '2.0.5'
mod 'garethr/docker', '5.3.0'
mod 'herculesteam/augeasproviders_core', '2.1.4'
mod 'herculesteam/augeasproviders_shellvar', '2.2.2'
mod 'herculesteam/augeasproviders_sysctl', '2.2.0'
mod 'inkblot/ipcalc', '2.2.0'
mod 'jgazeley/django', '0.1.0'
mod 'jgazeley/ekahau_throughput_server', '0.1.2'
mod 'jgazeley/freeradius', '3.5.0'
mod 'jgazeley/galera_innoptimizer', '0.1.0'
mod 'jgazeley/hddtemp', '1.0.3'
mod 'jgazeley/iperf', '0.2.1'
mod 'jgazeley/lm_sensors', '0.3.0'
mod 'jgazeley/mod_auth_cas', '0.4.1'
mod 'jgazeley/motd', '0.2.4'
mod 'jgazeley/nagios', '0.3.0'
mod 'jgazeley/networker', '0.1.0'
mod 'jgazeley/oracleinstantclient', '1.0.0'
mod 'jgazeley/percona_toolkit', '0.1.0'
mod 'jgazeley/rpmgpgkey', '0.1.0'
mod 'jgazeley/sqlmap', '0.1.0'
mod 'jgazeley/winbind', '0.4.1'
mod 'jgazeley/yumupdate', '0.4.0'
mod 'jonnyx/logwatch', '2.0.0'
mod 'juniorsysadmin/irqbalance', '1.1.0'
mod 'leinaddm/htpasswd', '0.0.3'
mod 'lwf/remote_file', '1.1.3'
mod 'maestrodev/wget', '1.7.3'
mod 'npwalker-pe_metric_curl_cron_jobs', '4.6.0'
mod 'pcfens/ca_cert', '1.7.0'
mod 'pfmooney/mit_krb5', '0.0.8'
mod 'puppet/archive', '2.2.0'
mod 'puppet/logrotate', '3.1.0'
mod 'puppet/selinux', '1.4.0'
mod 'puppet/staging', '2.2.0'
mod 'puppetlabs/apache', '2.3.0'
mod 'puppetlabs/apt', '3.0.0'
mod 'puppetlabs/apt', '3.0.0'
mod 'puppetlabs/concat', '4.1.1'
mod 'puppetlabs/firewall', '1.10.0'
mod 'puppetlabs/gcc', '0.3.0'
mod 'puppetlabs/git', '0.5.0'
mod 'puppetlabs/inifile', '2.0.0'
mod 'puppetlabs/java', '2.2.0'
mod 'puppetlabs/mysql', '5.1.0'
mod 'puppetlabs/ntp', '7.0.0'
mod 'puppetlabs/pe_gem', '0.2.0'
mod 'puppetlabs/postgresql', '5.2.1'
mod 'puppetlabs/puppet_agent', '1.4.1'
mod 'puppetlabs/ruby', '1.0.0'
mod 'puppetlabs/stdlib', '4.23.0'
mod 'puppetlabs/tftp', '0.2.3'
mod 'puppetlabs/translate', '1.1.0'
mod 'puppetlabs/vcsrepo', '2.2.0'
mod 'puppetlabs/xinetd', '3.0.0'
mod 'razorsedge/network', '3.11.1'
mod 'razorsedge/vmwaretools', '6.1.0'
mod 'saz/dnsmasq', '1.3.1'
mod 'saz/resolv_conf', '3.3.0'
mod 'saz/rsyslog', '5.0.0'
mod 'saz/ssh', '3.0.1'
mod 'saz/sudo', '4.2.0'
mod 'saz/timezone', '4.0.1'
mod 'stahnma/epel', '1.3.0'
mod 'stankevich/python', '1.18.2'
mod 'stm/debconf', '2.0.0'
mod 'stm/haveged', '0.2.0'
mod 'thias/postfix', '0.3.4'
mod 'thias/sysctl', '1.0.6'
mod 'thrnio/ip', '1.0.0'
mod 'treydock/gpg_key', '0.0.4'
mod 'treydock/repo_centos', '4.0.0'
mod 'unibet/forge_server', '0.4.0'
mod 'wolfspyre/vmware_puppetfact', '0.2.2'
mod 'yelp/netstdlib', '0.0.1'
mod 'zleslie/permute', '0.0.4'
mod 'zleslie/pxe', '0.2.0'
# Discovered elements from existing Puppetfile
#897 addressspace
mod 'addressspace',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-addressspace.git',
  :tag => '0.1.0'
#898 cacerts
mod 'cacerts',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-cacerts.git',
  :tag => '0.2.1'
#900 fail2ban
mod 'fail2ban',
  :git => 'https://github.com/uobnetops/calmenergy-fail2ban.git',
  :ref => 'ed0233261c898eef72e0da91d5b1b3eb72c232be'
#904 gitlab
mod 'gitlab',
  :git => 'https://github.com/djjudas21/puppet-gitlab.git',
  :ref => '739fc01'
#906 sauron
mod 'sauron',
  :git => 'https://github.com/uobnetops/puppet-sauron.git',
  :ref => 'master'
#907 uob_classifier
mod 'uob_classifier',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-uob_classifier.git',
  :ref => '0.1.3'
#908 wordpress
mod 'wordpress',
  :git => 'https://github.com/uobnetops/puppet-wordpress.git',
  :ref => 'uob'
rnelson0 commented 6 years ago

I cannot reproduce the errors you receive exactly:

[rnelson0@build03 puppet-generate-puppetfile:master±]$ be generate-puppetfile -v -c -p Puppetfile.55
generate-puppetfile v1.0.1.rc0
[rnelson0@build03 puppet-generate-puppetfile:master±]$ be generate-puppetfile -c -p Puppetfile.55

Installing modules. This may take a few minutes.

There was a problem with the module name 'jgazeley/nagios'.
There was a problem with the module name 'puppetlabs/mysql'.
  Check that modules exist as under the listed name, and/or your connectivity to the puppet forge.

Here is the PARTIAL Puppetfile that would have been generated.

Your Puppetfile has been generated. Copy and paste between the markers:

=======================================================================
forge 'https://forge.puppet.com'

# Modules discovered by generate-puppetfile
mod 'abrader/gms',                             '1.0.3'
mod 'alexharvey/disable_transparent_hugepage', '1.2.1'
mod 'alexharvey/firewall_multi',               '1.10.0'
mod 'arioch/redis',                            '3.1.0'
mod 'camptocamp/augeas',                       '1.6.0'
mod 'camptocamp/dell',                         '1.0.0'
mod 'camptocamp/kmod',                         '2.2.0'
mod 'camptocamp/openldap',                     '1.16.1'
mod 'camptocamp/systemd',                      '1.1.1'
mod 'ceh/freetds',                             '1.1.0'
mod 'dalen/dnsquery',                          '3.0.0'
mod 'dalen/puppetdbquery',                     '3.0.1'
mod 'darin/zypprepo',                          '1.0.2'
mod 'davidb/dhcpd',                            '0.4.1'
mod 'derdanne/nfs',                            '2.0.6'
mod 'garethr/docker',                          '5.3.0'
mod 'herculesteam/augeasproviders_core',       '2.1.4'
mod 'herculesteam/augeasproviders_shellvar',   '2.2.2'
mod 'herculesteam/augeasproviders_sysctl',     '2.2.0'
mod 'inkblot/ipcalc',                          '2.2.0'
mod 'jgazeley/django',                         '0.1.0'
mod 'jgazeley/ekahau_throughput_server',       '0.1.2'
mod 'jgazeley/freeradius',                     '3.5.0'
mod 'jgazeley/galera_innoptimizer',            '0.1.0'
mod 'jgazeley/hddtemp',                        '1.0.3'
mod 'jgazeley/iperf',                          '0.2.1'
mod 'jgazeley/lm_sensors',                     '0.3.0'
mod 'jgazeley/mod_auth_cas',                   '0.4.0'
mod 'jgazeley/motd',                           '0.2.4'
mod 'jgazeley/networker',                      '0.1.0'
mod 'jgazeley/oracleinstantclient',            '1.0.0'
mod 'jgazeley/percona_toolkit',                '0.1.0'
mod 'jgazeley/rpmgpgkey',                      '0.1.0'
mod 'jgazeley/sqlmap',                         '0.1.0'
mod 'jgazeley/winbind',                        '0.4.1'
mod 'jgazeley/yumupdate',                      '0.4.0'
mod 'jonnyx/logwatch',                         '1.0.2'
mod 'juniorsysadmin/irqbalance',               '1.1.0'
mod 'leinaddm/htpasswd',                       '0.0.3'
mod 'lwf/remote_file',                         '1.1.3'
mod 'maestrodev/wget',                         '1.7.3'
mod 'pcfens/ca_cert',                          '1.7.0'
mod 'pfmooney/mit_krb5',                       '0.0.8'
mod 'puppet/archive',                          '2.2.0'
mod 'puppet/logrotate',                        '3.2.0'
mod 'puppet/selinux',                          '1.5.1'
mod 'puppet/staging',                          '3.1.0'
mod 'puppetlabs/apache',                       '1.11.1'
mod 'puppetlabs/apt',                          '2.4.0'
mod 'puppetlabs/concat',                       '1.2.5'
mod 'puppetlabs/firewall',                     '1.11.0'
mod 'puppetlabs/gcc',                          '0.3.0'
mod 'puppetlabs/git',                          '0.5.0'
mod 'puppetlabs/inifile',                      '2.1.1'
mod 'puppetlabs/java',                         '2.3.0'
mod 'puppetlabs/ntp',                          '7.0.0'
mod 'puppetlabs/pe_gem',                       '0.2.0'
mod 'puppetlabs/postgresql',                   '5.2.1'
mod 'puppetlabs/puppet_agent',                 '1.5.0'
mod 'puppetlabs/ruby',                         '1.0.0'
mod 'puppetlabs/stdlib',                       '4.24.0'
mod 'puppetlabs/tftp',                         '1.0.0'
mod 'puppetlabs/transition',                   '0.1.1'
mod 'puppetlabs/translate',                    '1.1.0'
mod 'puppetlabs/vcsrepo',                      '2.2.0'
mod 'puppetlabs/xinetd',                       '3.0.0'
mod 'razorsedge/network',                      '3.11.1'
mod 'razorsedge/vmwaretools',                  '6.1.0'
mod 'saz/dnsmasq',                             '1.4.0'
mod 'saz/resolv_conf',                         '3.3.0'
mod 'saz/rsyslog',                             '5.0.0'
mod 'saz/ssh',                                 '3.0.1'
mod 'saz/sudo',                                '5.0.0'
mod 'saz/timezone',                            '4.1.1'
mod 'stahnma/epel',                            '1.3.0'
mod 'stankevich/python',                       '1.18.2'
mod 'stm/debconf',                             '2.0.0'
mod 'stm/haveged',                             '0.2.0'
mod 'thias/postfix',                           '0.3.4'
mod 'thias/sysctl',                            '1.0.6'
mod 'thrnio/ip',                               '1.0.1'
mod 'treydock/gpg_key',                        '0.0.4'
mod 'treydock/repo_centos',                    '4.0.0'
mod 'unibet/forge_server',                     '0.4.0'
mod 'wolfspyre/vmware_puppetfact',             '0.2.2'
mod 'yelp/netstdlib',                          '0.0.1'
mod 'zleslie/permute',                         '0.0.4'
mod 'zleslie/pxe',                             '0.2.0'
# Discovered elements from existing Puppetfile
mod 'npwalker-pe_metric_curl_cron_jobs', '4.6.0'
#897 addressspace
mod 'addressspace',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-addressspace.git',
  :tag => '0.1.0'
#898 cacerts
mod 'cacerts',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-cacerts.git',
  :tag => '0.2.1'
#900 fail2ban
mod 'fail2ban',
  :git => 'https://github.com/uobnetops/calmenergy-fail2ban.git',
  :ref => 'ed0233261c898eef72e0da91d5b1b3eb72c232be'
#904 gitlab
mod 'gitlab',
  :git => 'https://github.com/djjudas21/puppet-gitlab.git',
  :ref => '739fc01'
#906 sauron
mod 'sauron',
  :git => 'https://github.com/uobnetops/puppet-sauron.git',
  :ref => 'master'
#907 uob_classifier
mod 'uob_classifier',
  :git => 'https://git.services.bristol.ac.uk/resnet/puppet-uob_classifier.git',
  :ref => '0.1.3'
#908 wordpress
mod 'wordpress',
  :git => 'https://github.com/uobnetops/puppet-wordpress.git',
  :ref => 'uob'
=======================================================================

Both puppetlabs/mysql and jgazeley/nagios exist. I wonder if this might represent transient issues with connectivity to the forge, or response issues internal to the forge. It takes a while, but I will run it again and see if I get different results.

rnelson0 commented 6 years ago

I strongly feel this is an issue with the forge. I know received:

Installing modules. This may take a few minutes.

There was a problem with the module name 'jgazeley/django'.
There was a problem with the module name 'jgazeley/nagios'.
There was a problem with the module name 'puppetlabs/mysql'.
There was a problem with the module name 'puppetlabs/pe_gem'.
There was a problem with the module name 'puppetlabs/puppet_agent'.
There was a problem with the module name 'yelp/netstdlib'.
  Check that modules exist as under the listed name, and/or your connectivity to the puppet forge.

Clearly they exist, but PMT is not able to download them at the time. I will try and talk to Puppet and see if they have an idea. The only way I see this being related to generate-puppetfile is if there's an API threshold that such a large Puppetfile crosses, maybe slowing it down would help.

rnelson0 commented 6 years ago

I opened https://tickets.puppetlabs.com/browse/FORGE-408 on this issue. Note that jgazeley/nagios SHOULD fail, as it wants concat >=4.0.0 and other modules want <=2.0.0. The intermittent issues with the other modules are what really concerns me.

djjudas21 commented 6 years ago

Thanks, I'm watching that issue now