Config file #44 - Githubissues

alikins commented 6 years ago

ala #44

undecided:

if I'll keep the dep on yamlloader or just implement something similar
if we should support multiple config files (ie, /etc/ansible/galaxy.yml and/or ~/.ansible/galaxy.yml) [probably not short term]
if we support multiple config file formats [hopefully not]
if we need write/save support for the config file [likely yes, but not soon, way past current sprint]

likely to change:

config file format data structure

SUMMARY

Add a config file at ~/.ansible/galaxy.yml (or wherever ANSIBLE_GALAXY_CONFIG env var points to)

Load the galaxy.yml into an OrderedDict via new dep 'yamlloader'
update main/cli/etc to load the new config file on CLI.run().
Use the config file settings for the galaxy server url and 'ignore_certs' settings.
Remove some unused default/runtime options from ansible_galaxy.config.
Add an example galaxy.yml to config/galaxy.yml
Add a new GalaxyConfigError to raise on config errors (currently only on parse/format errors, FileNotFoundError or OSError or logged and default builtin config is used)

An example config file pointing to a galaxy server on localhost:

server:
  ignore_certs: false
  url: http://localhost:8000
content_path: ~/.ansible/content
options:
  role_skeleton_ignore:
    - ^.git$
    - ^.*/.git_keep$
  role_skeleton_path: null
  verbosity: 0
version: 1

Config file is a yaml dict (OrderedDict to preserve option ordering when saved).

The top level keys are:

servers
content_roots
options

ISSUE TYPE

Feature Pull Request
Bugfix Pull Request

GALAXY CLI VERSION

Ansible Galaxy CLI, version 0.1.0
Linux, newswoop, 4.16.6-202.fc27.x86_64, #1 SMP Wed May 2 00:09:32 UTC 2018, x86_64
2.7.14 (default, Mar 14 2018, 13:36:31) 
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] /home/adrian/venvs/galaxy-py27/bin/python
No config file found; using defaults

ADDITIONAL INFORMATION

chouseknecht commented 6 years ago

Config

Location

ANSIBLE_GALAXY_CONFIG=~/galaxy.yml

Test Case:

Command

ansible-galaxy-cli install chouseknecht.minishift-up-role

Expected Results

The role chouseknecht.minishift-up-role would be installed from galaxy-qa.ansible.com

Actual Results

Received the following traceback:

Traceback (most recent call last):
  File "/Users/chouseknecht/.pyenv/versions/venv27/bin/ansible-galaxy-cli", line 11, in <module>
    load_entry_point('ansible-galaxy-cli', 'console_scripts', 'ansible-galaxy-cli')()
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy_cli/main.py", line 34, in main
    exit_code = cli.run()
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy_cli/cli/galaxy.py", line 226, in run
    self.execute()
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy_cli/cli/__init__.py", line 145, in execute
    fn()
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy_cli/cli/galaxy.py", line 514, in execute_install
    installed = content.install(force_overwrite=force_overwrite)
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy/flat_rest_api/content.py", line 455, in install
    content_archive = fetcher.fetch()
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy/fetch/galaxy_url.py", line 116, in fetch
    self.display_callback("failed to download the file: %s" % str(e))
AttributeError: 'GalaxyUrlFetch' object has no attribute 'display_callback'

The following is from the .log file:

[2018-05-24 05:52:07,320 77905 D] ansible_galaxy.fetch.galaxy_url fetch:108 - content_spec=chouseknecht.minishift-up-role
[2018-05-24 05:52:07,320 77905 D] ansible_galaxy.fetch.galaxy_url fetch:109 - download_url=https://github.com/chouseknecht/minishift-up-role/archive/1.0.12.tar.gz
[2018-05-24 05:52:11,865 77905 E] ansible_galaxy.download fetch_url:35 - HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy/download.py", line 21, in fetch_url
    url_file = open_url(archive_url, validate_certs=validate_certs)
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy/flat_rest_api/urls.py", line 923, in open_url
    r = urllib_request.urlopen(*urlopen_args)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/Users/chouseknecht/.pyenv/versions/2.7.14/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
[2018-05-24 05:52:11,867 77905 E] ansible_galaxy.fetch.galaxy_url fetch:115 - failed to download the file: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy/fetch/galaxy_url.py", line 113, in fetch
    validate_certs=self.validate_certs)
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy/download.py", line 36, in fetch_url
    raise exceptions.GalaxyDownloadError("failed to download the file: %s" % str(e))
GalaxyDownloadError: failed to download the file: HTTP Error 404: Not Found

chouseknecht commented 6 years ago

Config

No config file

Test Case:

Command

ansible-galaxy-cli install geerlingguy.logstash

Expected Results

Default configuration would be used, and the install would work.

Actual Results

Received the following traceback:

Traceback (most recent call last):
  File "/Users/chouseknecht/.pyenv/versions/venv27/bin/ansible-galaxy", line 11, in <module>
    load_entry_point('ansible-galaxy-cli', 'console_scripts', 'ansible-galaxy')()
  File "/Users/chouseknecht/.pyenv/versions/venv27/lib/python2.7/site-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/chouseknecht/.pyenv/versions/venv27/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2690, in load_entry_point
    raise ImportError("Entry point %r not found" % ((group, name),))
ImportError: Entry point ('console_scripts', 'ansible-galaxy') not found

chouseknecht commented 6 years ago

@alikins

How does the user configure logging? It's not clear from the example config files, nor the code. This is something we'll need to document for sure.

chouseknecht commented 6 years ago

You may want to consider using ruamel.yaml. It was used in Ansible Container, because it preserves comments, and handles ordering.

Not saying it's the right thing, just that it might be worth a look.

alikins commented 6 years ago

@chouseknecht For https://github.com/ansible/galaxy-cli/pull/50#issuecomment-391658643 ('ansible-galaxy-cli install chouseknecht.minishift-up-role'), it looks like it is an issue with the 'v1.0.0' style version tags. It ends up trying to download:

https://github.com/chouseknecht/minishift-up-role/archive/1.0.12.tar.gz

instead of:

https://github.com/chouseknecht/minishift-up-role/archive/v1.0.12.tar.gz

and then cascades errors.

master fails the same way, so the 'no leading v in archive version url' thing is a bug, just not (directly) related to the config code.

alikins commented 6 years ago

@chouseknecht https://github.com/ansible/galaxy-cli/pull/50#issuecomment-391660231

Recent versions don't install a 'ansible-galaxy' script/entry point.

Looks like the command was 'ansible-galaxy' not 'ansible-galaxy-cli' based on the '/bin/ansible-galaxy' in the bottom frame in the traceback:

  File "/Users/chouseknecht/.pyenv/versions/venv27/bin/ansible-galaxy", line 11, in <module>
    load_entry_point('ansible-galaxy-cli', 'console_scripts', 'ansible-galaxy')()

ansible-galaxy from an old install will fail exactly as described. ansible-galaxy-cli should work.

probably worth while deleting '/Users/chouseknecht/.pyenv/versions/venv27/bin/ansible-galaxy' from that venv

alikins commented 6 years ago

@chouseknecht re https://github.com/ansible/galaxy-cli/pull/50#issuecomment-391661441

For configuring logging, I tend to lean towards using a separate config file for detailed logging config (ala 'galaxy-logging.yml' as per https://github.com/ansible/galaxy-cli/pull/51). But possibly with some high level options in the app config file ('galaxy.yml') for things like enabling debug level logging.

Primarily because logging config can be pretty verbose. Though, there shouldn't need to be any detailed logging configuration in the default 'galaxy.yml' config file since the app has baked in logging config defaults.

Options:

include any (non-default) detailed logging config in the app config file ('galaxy.yml') under a top level 'logging' key. (this assumes a yaml or json config file, I don't think it would be workable for ini style[1])
use a separate logging config file ('galaxy-logging.yml' as per https://github.com/ansible/galaxy-cli/pull/51)

[1] at least not without kluges like a config item value that is a quoted string containing a logging config dict in yaml,json,or the weirdish 'fileConfig' format

alikins commented 6 years ago

@chouseknecht re https://github.com/ansible/galaxy-cli/pull/50#issuecomment-391661850

I would love to use ruamel.yaml if that is an option. It certainly seems to fix about 90% of pyyaml weirdness.

alikins commented 6 years ago

@chouseknecht re https://github.com/ansible/galaxy-cli/pull/50#issuecomment-391658643

https://github.com/ansible/galaxy-cli/pull/54 is pr for some better error handling on 404.

Still need to fix the root bug which is related to it not using the original version tag in the URL

chouseknecht commented 6 years ago

@alikins

Went back and re-tested the scenario where galaxy.yml is missing. Still getting an error.

Here's the exact command:

ansible-galaxy-cli install geerlingguy.logstash

Below is the error displayed on the command line. It looks different than the one I reported recently. Maybe copy-pated the wrong thing last night.

Traceback (most recent call last):
  File "/Users/chouseknecht/.pyenv/versions/venv27/bin/ansible-galaxy-cli", line 11, in <module>
    load_entry_point('ansible-galaxy-cli', 'console_scripts', 'ansible-galaxy-cli')()
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy_cli/main.py", line 34, in main
    exit_code = cli.run()
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy_cli/cli/galaxy.py", line 216, in run
    galaxy_context = self._get_galaxy_context(self.options, self.config)
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy_cli/cli/galaxy.py", line 187, in _get_galaxy_context
    server = config.server.copy()
AttributeError: 'list' object has no attribute 'copy'

Here's the traceback found in the log file:

[2018-05-24 17:51:07,084 02711 D] ansible_galaxy_cli.main main:21 - args: ['/Users/chouseknecht/.pyenv/versions/venv27/bin/ansible-galaxy-cli', 'install', 'geerlingguy.logstash']
[2018-05-24 17:51:07,085 02711 D] ansible_galaxy_cli.cli run:154 - self.args: ['geerlingguy.logstash']
[2018-05-24 17:51:07,085 02711 I] ansible_galaxy_cli.cli run:157 - Using /Users/chouseknecht/.ansible/galaxy.yml as config file
[2018-05-24 17:51:07,085 02711 E] ansible_galaxy.config.config_file load:15 - [Errno 2] No such file or directory: '/Users/chouseknecht/.ansible/galaxy.yml'
Traceback (most recent call last):
  File "/Users/chouseknecht/projects/galaxy-cli/ansible_galaxy/config/config_file.py", line 12, in load
    with open(full_file_path, 'r') as config_file_stream:
IOError: [Errno 2] No such file or directory: '/Users/chouseknecht/.ansible/galaxy.yml'
[2018-05-24 17:51:07,085 02711 E] ansible_galaxy.config.config_file load:16 - Unable to load config file (/Users/chouseknecht/.ansible/galaxy.yml): [Errno 2] No such file or directory: '/Users/chouseknecht/.ansible/galaxy.yml'
[2018-05-24 17:51:07,085 02711 W] ansible_galaxy.config.config_file load:22 - No config data was loaded from file (/Users/chouseknecht/.ansible/galaxy.yml), returning None instead
[2018-05-24 17:51:07,085 02711 D] ansible_galaxy.config.config load:42 - config_data: OrderedDict([('server', [{'url': 'https://galaxy-qa.ansible.com', 'validate_certs': True}]), ('content_path', '~/.ansible/content'), ('options', {'role_skeleton_path': None, 'role_skeleton_ignore': ['^.git$', '^.*/.git_keep$']}), ('version', 1)])
[2018-05-24 17:51:07,086 02711 D] ansible_galaxy_cli.cli.galaxy run:209 - {
    "server": [
        {
            "url": "https://galaxy-qa.ansible.com",
            "validate_certs": true
        }
    ],
    "content_path": "~/.ansible/content",
    "options": {
        "role_skeleton_path": null,
        "role_skeleton_ignore": [
            "^.git$",
            "^.*/.git_keep$"
        ]
    }
}

alikins commented 6 years ago

@chouse https://github.com/ansible/galaxy-cli/pull/50/commits/cce70c94f814fda7549984a0a1b6bb6a25087d6e should fix https://github.com/ansible/galaxy-cli/pull/50#issuecomment-391879092

ansible / mazer

Config file #44 #50

SUMMARY

ISSUE TYPE

GALAXY CLI VERSION

ADDITIONAL INFORMATION

Config

Location

Contents

Test Case:

Command

Expected Results

Actual Results

Config

Test Case:

Command

Expected Results

Actual Results