Syntax error when using heredoc

[jaredbrogan]

The issue I'm dealing with is when using a bash heredoc within an Azure Devops pipeline yaml. Yamllint produces a error syntax error: could not find expected ':' (syntax) result, which unfortunately cannot be disabled/suppressed to a warning like the other rules.

For added context, the heredoc contains a graphql body. I think my only path forward is to prevent yamllint from scanning this particular yaml.

Here is an example of the bash step used within the ADO pipeline yaml.

jobs:
- job: "Example_Pipeline"
  steps:
  - bash: |
      read -r -d '' query_payload <<-EOM
{"query":"{
  pages {
    list(orderBy: ID) {
      id
      path
      title
    }
  }
}
"}
EOM
    displayName: 'Query pages'

The error is referencing the last line of the heredoc, "}.

---

I'm doubting this suggestion would resolve the issue at hand, but this does seem somewhat related and a good idea.

• 517

[andrewimeson]

Your example is not valid (i.e. parseable) yaml. It's possible that whatever library Azure DevOps uses is able to parse it, but neither yq (Go) or PyYAML are able to parse it.

 cat << 'EOF' | python3 -c 'import yaml,sys;yaml.safe_load(sys.stdin)'
jobs:
- job: "Example_Pipeline"
  steps:
  - bash: |
      read -r -d '' query_payload <<-EOM
{"query":"{
  pages {
    list(orderBy: ID) {
      id
      path
      title
    }
  }
}
"}
EOM
    displayName: 'Query pages'

Results in

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/__init__.py", line 125, in safe_load
    return load(stream, SafeLoader)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
                         ^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/parser.py", line 403, in parse_indentless_sequence_entry
    if self.check_token(BlockEntryToken):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/scanner.py", line 115, in check_token
    while self.need_more_tokens():
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/scanner.py", line 152, in need_more_tokens
    self.stale_possible_simple_keys()
  File "/Users/aimeson/Documents/Code/personal/yamllint/.venv/lib/python3.11/site-packages/yaml/scanner.py", line 291, in stale_possible_simple_keys
    raise ScannerError("while scanning a simple key", key.mark,
yaml.scanner.ScannerError: while scanning a simple key
  in "<stdin>", line 6, column 1
could not find expected ':'
  in "<stdin>", line 15, column 2

[andrewimeson]

Here's a reformatted YAML document that is valid and I believe has the structure you need

---
jobs:
  - job: Example_Pipeline
    steps:
      - bash: |
          read -r -d '' query_payload <<-EOM
          {"query":"{
            pages {
              list(orderBy: ID) {
                id
                path
                title
              }
            }
          }
          "}
          EOM
        displayName: Query pages

[jaredbrogan]

@andrewimeson I figured out the issue and it is indeed simply a YAML limitation.

I was under the impression that I had been able to do this in the past, but apparently that has either changed or (more than likely) I just misremembered. Your suggestion is technically accurate for YAML, but when ran in the pipeline it will result in a bash syntax error due to the indentation (probably an issue on Azure's end).

I solved my dilemma by storing the payloads within .graphql files and then read from them directly, removing the need for a heredoc. Although heredocs are convenient, they don't work for every situation, this being one of them.

Thanks for offering assistance, I appreciate it. 🙂

[andrewimeson]

@jaredbrogan I am perplexed by why it didn't work in Azure, because when I parse the value with yq and eval it in Bash it does work.

$ eval "$(yq -r '.jobs.[0].steps.[0].bash' < test.yaml)"
$ echo "$query_payload"
{"query":"{
  pages {
    list(orderBy: ID) {
      id
      path
      title
    }
  }
}
"}

Either way, glad you got something working. Happy to help! 👍🏻