add token credentials to pull private git repos

[mitchdubdub]

We have an internal github that is setup as private, so we have to use http token credentials to pull the interal github puppet repos for testing. It would be nice if the puppet-litmus spec_helper_acceptance.rb had a way to add the token to pull the repos.

[trevor-vaughan]

Wouldn't it be better to have all of this abstracted instead of leaving it in the user's spec_helper_acceptance.rb?

[tphoney]

Does this feel like it is an additional step, and therefore should be a rake task ? Are we reading these repo's from the fixtures file ?

[trevor-vaughan]

You should totally use the .fixtures.yml if present. It contains dependencies that may not be operationally required and you don't want to do any dependency resolution since many modules have dependencies that aren't needed for the system under test.

[Dan33l]

You should totally use the .fixtures.yml if present.

Does litmus use .fixtures.yml to install modules on SUTs ?

[tphoney]

litmus does use .fixtures.yml. and it uses the rake task spec_prep to get the associated modules. https://github.com/puppetlabs/puppetlabs_spec_helper#using-fixtures

[Dan33l]

@mitchdubdub this issue is it yet relevant with the answer from @trevor-vaughan ?

[mitchdubdub]

From reading the notes from @tphoney maybe the puppetlabs_spec_helper is where this needs to be requested. but just to clarify, we have the .fixtures.yml file setup to use a token to access our private repos. How doe we add a token to be passed by litmus to pull the internal repos? our .fixtures.yml has this: local_private_repo: https://$TOKEN@private_github.com/local_org/local_priv_repo.git It needs the ability to have $TOKEN added in a variable to pull from the local private github

[tphoney]

@mitchdubdub so in the metadata file of the module under test, are the dependencies listed in there? https://github.com/puppetlabs/puppet_litmus/wiki/Architecture-of-puppet-litmus#install-module

[mitchdubdub]

yes, we have the dependencies; but they are the forge repos that we are using. In some cases, our internal modules require other internal modules. Those internal modules are not in the list of dependencies. I could add those.

[Dan33l]

@tphoney what happens if modules listed in metadata.json are not on the forge when the rake task -> pdk -> bolt is used?

[ekohl]

Wouldn't it be better to always use a forge (or git checkouts), even if it's a private one? Puppet supports using a private forge and beaker exposes that option.

[tphoney]

it would be a standard error from puppet module install if that module is not available to that system. having a private forge makes sense, thought i am not familiar with how that works with the puppet agent, i found this https://stackoverflow.com/questions/17571472/how-to-install-a-puppet-file-from-a-local-repo-with-puppet-module-install.

[ekohl]

https://github.com/unibet/puppet-forge-server worked well for me. https://github.com/puppetlabs/beaker-puppet/commit/6c87f62050c617fcf64ff7de228e2e1488ca4ee1 is how I used it with beaker when the forge was way too slow (timing out all the time). It can act as a cache but also store modules locally.

[tphoney]

so specifically https://github.com/puppetlabs/beaker-puppet/blob/6c87f62050c617fcf64ff7de228e2e1488ca4ee1/lib/beaker-puppet/install_utils/module_utils.rb#L77-L84

would it then be a case of adding a similar helper function into rake tasks like https://github.com/puppetlabs/puppet_litmus/blob/master/lib/puppet_litmus/rake_tasks.rb#L159 or as a more universal approach i would suggest leveraging bolt more. ala bundle exec bolt command run "puppet module install --module_repository http://dev-forge.example.com puppetlabs-apache " -i inventory.yaml --nodes '*'

[mitchdubdub]

i'd perfer the second approach

[tphoney]

There is another possible way, https://puppet.com/docs/bolt/latest/installing_tasks_from_the_forge.html#task-8928. yeah i know its another way. but it works well with the litmus workflow. I have been speaking with a contributor about maybe adding this as a litmus rake task.

[tphoney]

Hey ! i have "the fix" maybe :)

192

This rake task allows a user to build all the modules in a directory copy the resultant tar files to the SUT, and force install them. By default it will look in the spec/fixtures/modules. folder. It will accept a folder path as a parameter too. (it ignores symlinked folders)

bundle exec rake litmus:install_modules_from_directory
# or 
bundle exec rake 'litmus:install_modules_from_directory[~/.puppetlabs/bolt/modules]'

Can we close this ticket, and open a new ticket if you have feedback with the new functionality

[pmcmaw]

@mitchdubdub @ekohl @Dan33l @trevor-vaughan

Hey :-) Im just having a quick skim through opened issues here and I am wondering inline with TPs above comment, would yous be happy with me closing this issue?

[jordanbreen28]

Closing in line with above comments.