Volume Disconnection after deploying grafana-backup-tool

[Kalougear]

Hello there!

I'm encountering a persistent volume disconnection issue after running the grafana-backup-tool within a Docker container deployed using Ansible. Restarting the container doesn't resolve the problem, confirming it's not a transient issue.

Steps to Reproduce:

• Set up a Debian server with Docker.
• Deploy Grafana container with persistent volumes mounted using an Ansible playbook.
• Deploy backup tool with Ansible playbook (using VENV)

Grafana Ansible playbook

  hosts: master
  become: true
  tasks:
    - name: Ensure Grafana Data Directory Exists
      file:
        path: "{{ general_cont_path }}/grafana/data"
        state: directory
        mode: "0755"

    - name: Ensure Grafana backups Directory Exists
      file:
        path: "{{ general_cont_path }}/grafana/backups"
        state: directory
        mode: "0755"

    - name: Ensure Grafana vend_dir directory Exists
      file:
        path: "{{ venv_dir }}"
        state: directory
        mode: "0755"

    - name: Create Docker Compose File for Grafana
      copy:
        content: |
          version: '3.8'
          services:
            grafana:
              container_name: grafana
              image: grafana/grafana:latest
              ports:
                - "3000:3000"
              volumes:
                - /etc/localtime:/etc/localtime:ro
                - grafana_data:/var/lib/grafana
                - grafana_venv_dir:/opt/grafana_venv
                - grafana_backup:/var/lib/grafana/backups # Corrected line
              restart: unless-stopped
              networks:
                - home_automation_network
          volumes:
            grafana_data:
              name: grafana_data
              driver: local
              driver_opts:
                type: none
                device: "{{ general_cont_path }}/grafana/data"
                o: bind
            grafana_venv_dir:
              name: grafana_venv_dir
              driver: local
              driver_opts:
                type: none
                device: "{{ venv_dir }}"
                o: bind
            grafana_backup:
              name: grafana_backup
              driver: local
              driver_opts:
                type: none
                device: "{{ general_cont_path }}/grafana/backups"
                o: bind    
          networks:
            home_automation_network:
              external: true
        dest: "{{ composes_path }}/grafana-compose.yml"
        mode: "0644"

    - name: Deploy Grafana Container Using Docker Compose
      community.general.docker_compose:
        project_src: "{{ composes_path }}"
        files:
          - grafana-compose.yml
        state: present
        pull: yes
        restarted: yes

• All global vars are correct,The volumes are beeing created and verified with Inspect command.

Script Ansible playbook

  hosts: master
  become: true

  vars:
    backup_days: "1,2,3" # Monday, Tuesday, Wednesday
    backup_hour: 4
    backup_minute: 0
    backup_retention_days: 14

  tasks:
    - name: Ensure rsync is installed
      apt:
        name: rsync
        state: present
      become: true
    # Use docker_container module instead of docker exec commands
    - name: Create virtual environment within Grafana container
      docker_container:
        name: grafana
        state: started
        command: bash -c "python3 -m venv {{ venv_dir }}"
        env:
          VIRTUAL_ENV: "{{ venv_dir }}"

    - name: Install grafana-backup-tool within virtual environment
      docker_container:
        name: grafana
        state: started
        command: bash -c "source {{ venv_dir }}/bin/activate && pip install grafana-backup-tool"
        env:
          VIRTUAL_ENV: "{{ venv_dir }}"

    - name: Backup Grafana configuration on specific days
      cron:
        name: "Grafana Backup - Multiple Days"
        job: "grafana-backup-tool --config /var/lib/grafana/grafana.ini --token {{ grafana_token }} --backup /var/lib/grafana/backups{{ lookup('pipe', 'date +%Y-%m-%d-%H-%M-%S') }}.tar.gz"
        hour: "{{ backup_hour }}"
        minute: "{{ backup_minute }}"
        weekday: "{{ backup_days }}" # Using the variable directly
        user: root

    - name: Execute backup tool within container
      docker_container:
        name: grafana
        state: started
        command: /bin/bash -c "source {{ venv_dir }}/bin/activate && grafana-backup-tool --config /var/lib/grafana/grafana.ini --token {{ grafana_token }} --backup /var/lib/grafana/backups{{ lookup('pipe', 'date +%Y-%m-%d-%H-%M-%S') }}.tar.gz"
        env:
          VIRTUAL_ENV: "{{ venv_dir }}"

    - name: List backup files in the container
      ansible.builtin.shell: |
        docker exec grafana ls -lah /var/lib/grafana/backups
      register: backup_files
      ignore_errors: yes

    - name: Show backup files
      ansible.builtin.debug:
        msg: "{{ backup_files.stdout_lines }}"

    - name: Set proper ownership and permissions on the backup directory
      file:
        path: "{{ grafana_backup_directory }}"
        owner: root
        group: root
        mode: "0640"
        recurse: yes
      become: true

    - name: Copy backup from container to host
      command: rsync --archive --compress --delay-updates "{{ general_cont_path }}/grafana/backups" /home/sktech/backups/grafana
      become: yes

    - name: Clean up old backups (optional)
      find:
        paths: "{{ grafana_backup_directory }}/archive"
        patterns: "*.tar.gz"
        age: "{{ backup_retention_days | default(30) }}"
# Make sure to replace placeholders like `venv_dir`, `grafana_backup_directory`, and `grafana_token` with your actual values.
# Adjust the backup schedule, retention days, and other settings as needed.
# Consider using Ansible Vault to securely store sensitive information like the `grafana_token`.
# Key Improvements:
#
# - Uses `docker_container` module for more structured and reliable container interaction.
# - Consolidated virtual environment creation and activation into a single step.
# - Clarified variable usage and provided notes for placeholder replacement.
# - Emphasized security considerations in the comments.
#
# Remember to adapt this script to your specific environment and security requirements. If you have any further questions, feel free to ask!

Expected Behavior:

The grafana-backup-tool should run successfully (whichs runs) within the container but without impacting volume persistence. Volumes should remain attached even after container restarts.After this backup is finished no files are created anywhere which is expected Cause of the volume disconection when the script tryed to backup i assume.

Actual Behavior:

Volumes become disconnected after running the script and persist even after various container restarts, indicating a deeper issue.

Troubleshooting:

• I've isolated the problem to the script section executing the grafana-backup-tool.

  docker_container:
    name: grafana
    state: started
    command: /bin/bash -c "source {{ venv_dir }}/bin/activate && grafana-backup-tool --config /var/lib/grafana/grafana.ini --token {{ grafana_token }} --backup /var/lib/grafana/backups{{ lookup('pipe', 'date +%Y-%m-%d-%H-%M-%S') }}.tar.gz"
    env:
      VIRTUAL_ENV: "{{ venv_dir }}"

• Commenting out this section prevents volume disconnection, confirming the tool's involvement.

• I haven't found similar issues reported online or in the tool's documentation.

• I've tested volume persistence across various container restart methods (restart, stop/start, exit/start).

• The issue is narrowed down to the specific code block provided earlier.

Request:

I'm seeking assistance in pinpointing the root cause of the volume disconnection and finding a solution to ensure persistent backups within the container. Any insights, suggestions, or workarounds related to the grafana-backup-tool or potential volume manipulation within the container would be greatly appreciated.

[acjohnson]

This is not an issue in any way specific to grafana-backup-tool. My guess is you are misunderstanding the proper use of the docker_container ansible module, which is not something we are able to help with, other than point you to the upstream documentation for that module.

[Kalougear]

It seems I caused some confusion about my setup. While the issue I'm facing might not be directly related to the grafana-backup-tool, it appears backups aren't working at all,the fact that i was lucky with the ansible playbook to pinpoint the issue can help you for sure, regardless of whether I use manual methods or the tool with an Ansible playbook. I've tried various approaches and installations, but the data remains unbacked up and the volumes keep breaking.