crasher: AttributeError: 'CommentedSeq' object has no attribute 'get'

[markstos]

Summary

Running ansible-lint 6.15 can result in:

AttributeError: 'CommentedSeq' object has no attribute 'get'

Issue Type

• Bug Report

OS / ENVIRONMENT

ansible-lint 6.17.2 using ansible-core:2.15.1 ansible-compat:4.1.2 ruamel-yaml:0.17.32 ruamel-yaml-clib:0.2.7

- ansible installation method: OS
- ansible-lint installation method: pipx

##### STEPS TO REPRODUCE

I don't have a reduced-test case, but perhaps one is not necessary. From the stack trace, it should be possible to see where the  problem is triggered and improve the error handling there.

```console (paste below)
Traceback (most recent call last):
  File "/home/mark/.local/bin/ansible-lint", line 8, in <module>
    sys.exit(_run_cli_entrypoint())
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/__main__.py", line 317, in _run_cli_entrypoint
    sys.exit(main(sys.argv))
             ^^^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/__main__.py", line 268, in main
    result = _get_matches(rules, options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/runner.py", line 555, in _get_matches
    matches.extend(runner.run())
                   ^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/runner.py", line 155, in run
    matches = self._run()
              ^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/runner.py", line 211, in _run
    if isinstance(lintable.data, States) and lintable.exc:
                  ^^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/file_utils.py", line 399, in data
    self.state = append_skipped_rules(self.state, self)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 113, in append_skipped_rules
    yaml_skip = _append_skipped_rules(pyyaml_data, lintable)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 202, in _append_skipped_rules
    for ruamel_task, pyyaml_task in zip(ruamel_tasks, pyyaml_tasks):
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 251, in _get_tasks_from_blocks
    yield from get_nested_tasks(task)
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 240, in get_nested_tasks
    if not task or not is_nested_task(task):
                       ^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 316, in is_nested_task
    return any(task.get(key) for key in NESTED_TASK_KEYS)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mark/.local/pipx/venvs/ansible-lint/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 316, in <genexpr>
    return any(task.get(key) for key in NESTED_TASK_KEYS)
               ^^^^^^^^
AttributeError: 'CommentedSeq' object has no attribute 'get'

Desired Behavior

If this state indicates a linting problem or error with my playbook, the output should explain that.

Otherwise, perhaps this is a bug in Ansible lint for some edge case it should handle.

I don't understand the error enough to guess what in my files might be triggering this.

Actual Behavior

Simply running ansible-lint

Here's the only clue I have to what is triggering it: I took a playbook that triggered it, moved it to /tmp and re-ran ansible-lint againt the same file stored in /tmp... and it didn't trigger it. So it seems perhaps something in th environment is triggering the issue, and not the playbook itself.

I tried using strace to find extra files being loaded, but I could not find a smoking gun.

[tonykay]

I see this too:

Initially triggered inside neovim using the Ansible VSCode plugin via LSP but can recreate from the command line:

ansible-lint --offline --nocolor -f codeclimate "/Users/ankay/Dropbox/repos/collections/community/showroom/roles/info/tasks/main.yml"

Error message:

... File "/opt/homebrew/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 240, in get_nested_tasks if not task or not is_nested_task(task): ^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 316, in is_nested_task return any(task.get(key) for key in NESTED_TASK_KEYS) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 316, in <genexpr> return any(task.get(key) for key in NESTED_TASK_KEYS) ^^^^^^^^ AttributeError: 'CommentedSeq' object has no attribute 'get' Versions:

ansible-lint 6.17.2 using ansible-core:2.15.2 ansible-compat:4.1.5 ruamel-yaml:0.17.21 ruamel-yaml-clib:None
❯ ansible --version
ansible [core 2.15.2]
  config file = None
  configured module search path = ['/Users/ankay/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/ankay/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.11.4 (main, Jul 25 2023, 17:36:13) [Clang 14.0.3 (clang-1403.0.22.14.1)] (/opt/homebrew/opt/python@3.11/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

[tonykay]

I see if I pip uninstall ansible-lint and then install it again I get this version issue output

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
aider 0.3.0 requires requests==2.30.0, but you have requests 2.31.0 which is incompatible.

[ssbarnea]

Closing due to not including reproducing data.

[matthewkeil]

I also get this error. This should not have been closed @ssbarnea . It happens errantly and rerunning ansible-lint sometimes fixes the error. What format would you like the "reproducing data" in that can get this handled?

[ssbarnea]

TBH, I do not really believe in "sometimes", I need data to reproduce the issue and this kind of issue in particular should be reproducible.

[markstos]

Because there's a full stack trace here, it's not necessary to have reproducible data to make progress on the issue.

What can be done is update the code in ansiblelint/skip_utils.py around line line 316 to validate that task has of a type that has get attribute. Even better to follow the stacktrace further up the stack and throw a more useful error.

This is typing issue and failure to validate input somewhere. If the new error message is clear enough, the next time this is triggered, it may be clearer if there was something wrong with the YAML which the linter should handle more gracefully, or some other type of bug in the linter.

[leonlarin]

I'm able to reproduce with below data in ansible-error-test/tasks/main.yml :

---
- name: Test | Include vars1
  ansible.builtin.include_vars: vars1.yml
  tags: [test]

---
- name: Test | Include vars2
  ansible.builtin.include_vars: vars2.yml
  tags: [test]

In my case it was the superfluous yaml doc start --- in the middle of the file.

ansible-lint 6.22.1 using ansible-core:2.15.4 ansible-compat:4.1.10 ruamel-yaml:0.18.5 ruamel-yaml-clib:0.2.7

[thatmatthewxu]

I'm able to reproduce with below data in ansible-error-test/tasks/main.yml :

---
- name: Test | Include vars1
  ansible.builtin.include_vars: vars1.yml
  tags: [test]

---
- name: Test | Include vars2
  ansible.builtin.include_vars: vars2.yml
  tags: [test]

In my case it was the superfluous yaml doc start --- in the middle of the file.

ansible-lint 6.22.1 using ansible-core:2.15.4 ansible-compat:4.1.10 ruamel-yaml:0.18.5 ruamel-yaml-clib:0.2.7

This happened to me as well. After I removed the --- in the middle, the error was gone.

[markfaine]

Happening to me now and I have no multi document yaml files. I've tried several versions of ansible-lint and they are all affected.

Traceback (most recent call last):
  File "/usr/local/bin/ansible-lint", line 8, in <module>
    sys.exit(_run_cli_entrypoint())
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/__main__.py", line 402, in _run_cli_entrypoint
    sys.exit(main(sys.argv))
             ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/__main__.py", line 362, in main
    result = get_matches(rules, options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/runner.py", line 660, in get_matches
    matches.extend(runner.run())
                   ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/runner.py", line 159, in run
    matches = self._run()
              ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/runner.py", line 215, in _run
    if isinstance(lintable.data, States) and lintable.exc:
                  ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/file_utils.py", line 417, in data
    self.state = append_skipped_rules(self.state, self)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 113, in append_skipped_rules
    yaml_skip = _append_skipped_rules(pyyaml_data, lintable)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 202, in _append_skipped_rules
    for ruamel_task, pyyaml_task in zip(ruamel_tasks, pyyaml_tasks):
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 251, in _get_tasks_from_blocks
    yield from get_nested_tasks(task)
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 240, in get_nested_tasks
    if not task or not is_nested_task(task):
                       ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 316, in is_nested_task
    return any(task.get(key) for key in NESTED_TASK_KEYS)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ansiblelint/skip_utils.py", line 316, in <genexpr>
    return any(task.get(key) for key in NESTED_TASK_KEYS)
               ^^^^^^^^
AttributeError: 'CommentedSeq' object has no attribute 'get'

[FlippingBinary]

@ssbarnea Does Ansible even support multiple documents in a single yaml file? It's not something I usually see, so I would expect it to be an error. If you need a reproducible example, you can use the simple example in my repository, but it's basically just a task file that contains two tasks represented as two yaml documents in the same file.

First, ansible-lint is pushing through the multiple documents with yaml.load_all(file_text) when it encounters the ComposerError in load_data():

https://github.com/ansible/ansible-lint/blob/b7f69e36ae2ab3aa2161faee9cdda1f7730fd70c/src/ansiblelint/skip_utils.py#L137-L141

Later, it tries to call the get attribute as a method on a CommentedSeq object that doesn't have one, but that can be avoided by expanding the escape hatch in is_nested_task() to confirm that the task has a get attribute in this line:

https://github.com/ansible/ansible-lint/blob/b7f69e36ae2ab3aa2161faee9cdda1f7730fd70c/src/ansiblelint/skip_utils.py#L311

After that, it encounters the same problem expecting the ruamel_task to have a get attribute that's callable in _append_skipped_rules(), but that crash can be avoided by testing for a get attribute in this line:

https://github.com/ansible/ansible-lint/blob/b7f69e36ae2ab3aa2161faee9cdda1f7730fd70c/src/ansiblelint/skip_utils.py#L211

When those two lines are modified, ansible-lint does not crash. It does print out some stack traces, but it also prints out the location of the error followed by the code report and a summary at the end.

Here is an excerpt showing how it indicates the line containing the error:

Syntax Error while loading YAML.
  but found another document

The error appears to be in '/home/user/projects/ansible-lint-crasher/roles/common/tasks/main.yml': line 6, column 1, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

---
^ here
, file was ignored.

That message seems to imply that it's an error to find another document. Maybe the load_data function could reject the file when it encounters a ComposerError instead of loading it as multiple documents? Or it could extract the first document for analysis and append the list of linting errors with the fact that the file contains a second document?

Either way, would you mind reopening this issue? It is a crash that still exists and is easily reproduced.

[markstos]

It is a crash that still exists and is easily reproduced.

And it looks like at least 7 people have reported running into it. I agree, a crasher with a clear and simple reproduction is worth re-opening.