Improve the raw tag to make it easier to escape curly brackets from Ansible playbooks

[n0one42]

Describe the issue

Like I already wrote in this post: https://github.com/kestra-io/kestra/issues/4093 which was instantly closed again... without even waiting for an answer.

@gregoir does it works now for you?

id: ubuntu-server-orchestrator
namespace: homelab.dev
description: Has to be executed at least once after a clone!

labels:
  project: Ubuntu-Servers
  step: init

variables:
  hosts: ["xxx"]

tasks:
  - id: ubuntu-server-orchestrator
    type: io.kestra.plugin.core.flow.WorkingDirectory
    inputFiles:
      inventory.ini: |
        {% for host in vars.hosts %}
        {{ host }} ansible_port=xxx
        {% endfor %}
      playbook.yml: |
        ---
        - name: Configure Ubuntu 24.04 server with new hostname and machine-id
          hosts: all
          become: yes
          vars:
            new_hostname: "new-server-name"

          tasks:
            - name: Set new hostname
              hostname:
                name: {% raw %}{{ new_hostname }}{% endraw %}

            - name: Update /etc/hosts file
              lineinfile:
                path: /etc/hosts
                regexp: '^127\.0\.1\.1'
                line: {% raw %}127.0.1.1 {{ new_hostname }}{% endraw %}

            - name: Remove existing machine-id files
              file:
                path: {% raw %}{{ item }}{% endraw %}
                state: absent
              loop:
                - /etc/machine-id
                - /var/lib/dbus/machine-id

            - name: Generate new machine-id in /etc
              command: dbus-uuidgen --ensure=/etc/machine-id

            - name: Generate new machine-id in /var/lib/dbus
              command: dbus-uuidgen --ensure

            - name: Restart systemd-hostnamed
              systemd:
                name: systemd-hostnamed
                state: restarted

    tasks:
      - id: execute_playbooks
        type: io.kestra.plugin.ansible.cli.AnsibleCLI
        namespaceFiles:
          enabled: true
        docker:
          image: cytopia/ansible:latest-tools
          pullPolicy: IF_NOT_PRESENT
          user: "1000"
        commands:
          - ansible-playbook -i inventory.ini playbook.yml --extra-vars "@common_vars.yml" --private-key xxx
        env:
          ANSIBLE_HOST_KEY_CHECKING: "False"
          ANSIBLE_REMOTE_USER: "xxx"

Does not matter if like:

 {% raw %}{{ new_hostname }}{% endraw %}
 {% raw %}"{{ new_hostname }}"{% endraw %}
"{% raw %}{{ new_hostname }}{% endraw %}"

Error:

2024-07-03 09:30:03.449
Failed preExecuteTasks on WorkingDirectory: Unable to find `new_hostname` used in the expression `---
- name: Configure Ubuntu 24.04 server with new hostname and machine-id
  hosts: all
  become: yes
  vars:
    new_hostname: "new-server-name"  # Replace with your desired hostname

  tasks:
    - name: Set new hostname
      hostname:
        name: "{{ new_hostname }}"

    - name: Update /etc/hosts file
      lineinfile:
        path: /etc/hosts
        regexp: '^127\.0\.1\.1'
        line: 127.0.1.1 {{ new_hostname }}

    - name: Remove existing machine-id files
      file:
        path: {{ item }}
        state: absent
      loop:
        - /etc/machine-id
        - /var/lib/dbus/machine-id

    - name: Generate new machine-id in /etc
      command: dbus-uuidgen --ensure=/etc/machine-id

    - name: Generate new machine-id in /var/lib/dbus
      command: dbus-uuidgen --ensure

    - name: Restart systemd-hostnamed
      systemd:
        name: systemd-hostnamed
        state: restarted
` at line 11

@loicmathieu

Environment

• Kestra Version: 0.17.8
• Operating System (OS/Docker/Kubernetes): docker

[n0one42]

@anna-geller As a hint because you changed the name: It has not only to do with ansible playbooks but also with docker compose files. This is why I actually upload the docker compose files first to the remote server.

[anna-geller]

The example was for Ansible, can you share other examples where you see that issue?

[n0one42]

@anna-geller It does not matter if its an ansible, an docker compose or what ever. It has to do with the interpretation of yml files by itself. More over with the io.kestra.plugin.core.flow.WorkingDirectory

id: yaml_test
namespace: homelab.dev
description: Testing yml files

labels:
  project: Test

variables:
  host: "xxx"
  ssh_port: xxx
  primary_user_name: "xxx"

tasks:
  - id: wdir
    type: io.kestra.plugin.core.flow.WorkingDirectory
    inputFiles:
      myfile.yml: |
        Do not interpret it... {% raw %}{{ item }}{% endraw %}
    outputFiles:
      - "*"
    tasks:
      - id: showContent
        type: io.kestra.plugin.scripts.shell.Commands
        commands:
          - ls {{ workingDir }}
          - echo "Task is needed here... so placeholdertask"

  - id: upload
    type: io.kestra.plugin.fs.sftp.Upload
    host: "{{ vars.host }}"
    username: "{{ vars.primary_user_name }}"
    keyfile: "{{ read('shared/secrets/xxx') }}"
    port: "{{ vars.ssh_port }}"
    from: "{{ outputs.wdir.outputFiles['myfile.yml'] }}"
    to: "myfile.yml"

Results:

      myfile.yml: |
        Do not interpret it... {% raw %}{{ item }}{% endraw %}

Failed preExecuteTasks on WorkingDirectory: Unable to find `item` used in the expression `Do not interpret it... {{ item }}
` at line 1

      myfile.yml: |
        {% raw %}Do not interpret it... {{ item }}{% endraw %}

Failed preExecuteTasks on WorkingDirectory: Unable to find `item` used in the expression `Do not interpret it... {{ item }}
` at line 1

[anna-geller]

yes, I'm just asking for more examples, this will help us identify the right solution

[anna-geller]

hey @n0one42, I talked to the team and there seems to be a misunderstanding about when the fix will be available. Your first issue https://github.com/kestra-io/kestra/issues/4093 was closed after the fix was done. However, the fix hasn't been released yet. It will be released by the latest in Kestra 0.18 planned for the 6th of August.

[n0one42]

@anna-geller but he wrote: So the issue was that we render twice the file content, I fixed it, it will be backported for our next bugfix release on 0.17 so the v0.17 was already out... Anyway. I think, this could be a pretty cool software but atm. for me and the companies I support, it is not worth it yet the hassle. There are to many issues. Lets hope for the best.

[loicmathieu]

It has been backported to 0.17.11.

[anna-geller]

@n0one42 sorry to hear, I respect your decision though. If you want to discuss your use cases with a real human, feel free to talk to our product expert @Ben8t - I'm sure that frustrations with expressions are solvable

[Ben8t]

Hello @n0one42 , would be happy to talk about your case over call - here is my schedule slots