search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.83k stars 1.95k forks source link

template rebound as mount does not propagate changes from re-rendered template #23691

Closed EugenKon closed 2 months ago

EugenKon commented 2 months ago

Nomad version

Nomad v1.8.2
BuildDate 2024-07-16T08:50:09Z
Revision 7f0822c1e4f25907d9f60e2d595411950dd1bd28

Operating system and Environment details

6.8.0-1012-aws #13-Ubuntu SMP Mon Jul 15 13:40:27 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Issue

image image image

Reproduction steps

  1. Configure job on which nginx depends
  2. start cluster without that job
  3. start job
  4. Nginx template is rerendered and nginx is reloaded

Expected Result

configuration should contain correct configuration, eg. configured upstream

Actual Result

generated configuration does not have expected upstream, but the service is up and healthy. The suspicious part is that I do not see that template was rerendered. When earlier it was change_mode="signal" I believe I saw messages like: tempalte was rerendered.

Job file (if appropriate)

      driver = "docker"

      config {
        force_pull = true
        image = "xxxx-nginx:${TAG}"
        ports = ["http", "https", "status"]

        volumes = [
          "custom/nginx.conf:/tmp/nginx/sites-enabled/site",
        ]
      }

      template {
        source        = "nginx.conf.tpl"
        destination   = "custom/nginx.conf"
        change_mode   = "script"
        change_script {
          command = "local/reload_nginx.sh"
        }
      }

      template {
        destination = "local/reload_nginx.sh"
        perms       = "0750"
        data        = <<-EOH
          #!/bin/bash
          set -ex;
          nginx -s reload

          if [ $? -eq 0 ]; then
            echo "Nginx reloaded successfully."
            exit 0
          else
            echo "Failed to reload Nginx."
            exit 1
          fi
        EOH
      }

Temporary workaround

docker stop wi-nginx-task-xxx

jrasell commented 2 months ago

Hi @EugenKon and thanks for raising this issue. Is the issue that the included template is not being rendered as you expect (it's a little hard to understand)? If so, can you provide any logs to help look at what the client does as well as a reproduction job specification? The consul-template service function has a large number of options which may be needed depending on your setup.

The screenshots show the reload script exited with code 0, indicating success and that NGINX has an acceptable configuration. That show the previous template was successfully rendered prior to executing it.

EugenKon commented 2 months ago

@jrasell It seems I found the problem. The file on the host system is updated, but it is not visible from the container.

image less /data/nomad/client/alloc/d27f63e2-35da-917d-862d-a1850fae97db/wi-nginx-task/custom/walk-inside.com.nginx.conf image docker exec -it wi-nginx-task-d27f63e2-35da-917d-862d-a1850fae97db less /tmp/nginx/sites-enabled/walk-inside.com image

So I just checked if I manually change the configuration file for Nginx on a host system, then this new content is not visible from the Nginx container.

EugenKon commented 2 months ago

Probably the problem is rprivate option used to mount the nginx configuration into the container.

docker --version
Docker version 27.1.1, build 6312585
EugenKon commented 2 months ago

This looks like the old bug in Docker https://github.com/docker/for-win/issues/5530, but it was fixed. Probably it could be related to Ubuntu 24.04 )

Linux ip-172-31-0-179 6.8.0-1012-aws #13-Ubuntu SMP Mon Jul 15 13:40:27 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
tgross commented 2 months ago

@EugenKon looks like you've got it figured out. Just FYI, instead of volumes you can use the mount block to have fine-grained control over things like mount propagation.

EugenKon commented 1 month ago

@tgross Finally I managed to minimize the experiment and show that this is a Nomad bug which is critical and easy to fix: Just do not recreate the file. It should be rewritten instead.

From below you can see that the rendered template is mounted into container. The inode for both files are the same. Changes to the file from inside container are visible on the host and changes to the file from the host are visible from inside container. But when Nomad notices changes it rerenders template and creates the new file on the host system. But this new file is not mounted into container. Notice that inode on the host system is changed. This resulting to have two different files:

  1. Nomad continue a rendering changes into the different file which is not visible from container
  2. Container stays to refer to the original file when it was run.

From CONTAINER

6581921960ca:/opt# stat zz
  File: zz
  Size: 37          Blocks: 8          IO Block: 4096   regular file
Device: 10301h/66305d   Inode: 3825587     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-08-09 15:50:56.926060691 +0000
Modify: 2024-08-09 15:50:54.300050975 +0000
Change: 2024-08-09 15:50:54.303050986 +0000
6581921960ca:/opt# cat zz
upstream server 172.31.10.241:24826;
6581921960ca:/opt# date; cat zz; date; echo "INSIDE" >> zz; cat zz; date;
Fri Aug  9 15:58:10 UTC 2024
upstream server 172.31.10.241:24826;
HOST
Fri Aug  9 15:58:10 UTC 2024
upstream server 172.31.10.241:24826;
HOST
INSIDE
Fri Aug  9 15:58:10 UTC 2024
6581921960ca:/opt# stat zz
  File: zz
  Size: 49          Blocks: 8          IO Block: 4096   regular file
Device: 10301h/66305d   Inode: 3825587     Links: 0
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-08-09 15:58:12.065860201 +0000
Modify: 2024-08-09 15:58:10.296851206 +0000
Change: 2024-08-09 15:58:13.464867313 +0000
6581921960ca:/opt# cat zz
upstream server 172.31.10.241:24826;
HOST
INSIDE

From HOST

root@ip-172-31-10-241:/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom# stat zz
  File: zz
  Size: 37          Blocks: 8          IO Block: 4096   regular file
Device: 259,1   Inode: 3825587     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-08-09 15:50:56.926060691 +0000
Modify: 2024-08-09 15:50:54.300050975 +0000
Change: 2024-08-09 15:50:54.303050986 +0000
 Birth: 2024-08-09 15:50:54.300050975 +0000
root@ip-172-31-10-241:/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom# cat zz
upstream server 172.31.10.241:24826;
root@ip-172-31-10-241:/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom# date; echo "HOST" >> zz; cat zz; date;
Fri Aug  9 15:58:09 UTC 2024
upstream server 172.31.10.241:24826;
HOST
Fri Aug  9 15:58:09 UTC 2024
root@ip-172-31-10-241:/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom# date; cat zz; date;
Fri Aug  9 15:58:12 UTC 2024
upstream server 172.31.10.241:24826;
HOST
INSIDE
Fri Aug  9 15:58:12 UTC 2024
root@ip-172-31-10-241:/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom# stat zz
  File: zz
  Size: 37          Blocks: 8          IO Block: 4096   regular file
Device: 259,1   Inode: 3825588     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-08-09 15:58:23.468928499 +0000
Modify: 2024-08-09 15:58:13.461867298 +0000
Change: 2024-08-09 15:58:13.464867313 +0000
 Birth: 2024-08-09 15:58:13.461867298 +0000
root@ip-172-31-10-241:/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom# cat zz
upstream server 172.31.10.241:24826;
Nomad logs ``` Aug 09 15:58:11 ip-172-31-10-241 nomad[145250]: client.alloc_runner.task_runner.task_hook.script_checks: tasklet executing: alloc_id=435b023c-9801-37f8-2247-4068f4714d1f task=postgres-task Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.460Z [DEBUG] agent: (runner) received template "78729eba1e2b937878cf101798d434d8" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.460Z [DEBUG] agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.460Z [DEBUG] agent: (runner) checking template 4437a110d252be45402c6d98dcfe607e Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) received template "78729eba1e2b937878cf101798d434d8" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.460Z [DEBUG] agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 4437a110d252be45402c6d98dcfe607e Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) rendering "/ops/shared/config/walk-inside.com.nginx.conf.tpl" => "/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom/walk-inside.com.nginx.conf" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) rendering "/ops/shared/config/walk-inside.com.nginx.conf.tpl" => "/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom/walk-inside.com.nginx.conf" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 2422dc5704082bcefb01d65b4465c7fb Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) checking template 2422dc5704082bcefb01d65b4465c7fb Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) enabling global quiescence for "78729eba1e2b937878cf101798d434d8" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) received template "4437a110d252be45402c6d98dcfe607e" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.461Z [DEBUG] agent: (runner) checking template 4437a110d252be45402c6d98dcfe607e Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.462Z [DEBUG] agent: (runner) rendering "(dynamic)" => "/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom/zz" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) enabling global quiescence for "78729eba1e2b937878cf101798d434d8" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) received template "4437a110d252be45402c6d98dcfe607e" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 4437a110d252be45402c6d98dcfe607e Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) rendering "(dynamic)" => "/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom/zz" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.465Z [INFO] agent: (runner) rendered "(dynamic)" => "/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom/zz" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.465Z [DEBUG] agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) rendered "(dynamic)" => "/data/nomad/client/alloc/b80c8922-4e84-32fb-68ab-fbf6c56c31ff/test/custom/zz" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.465Z [DEBUG] agent: (runner) checking template 2422dc5704082bcefb01d65b4465c7fb Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 2422dc5704082bcefb01d65b4465c7fb Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.466Z [DEBUG] agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.466Z [DEBUG] agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.466Z [DEBUG] agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.466Z [DEBUG] agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.466Z [DEBUG] agent: (runner) enabling global quiescence for "4437a110d252be45402c6d98dcfe607e" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) enabling global quiescence for "4437a110d252be45402c6d98dcfe607e" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.501Z [DEBUG] agent: (runner) received template "78729eba1e2b937878cf101798d434d8" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.501Z [DEBUG] agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.501Z [DEBUG] agent: (runner) checking template d8e8fca2dc0f896fd7cb4cb0031ba249 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) received template "78729eba1e2b937878cf101798d434d8" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template d8e8fca2dc0f896fd7cb4cb0031ba249 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.501Z [DEBUG] agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.502Z [DEBUG] agent: (runner) rendering "/ops/shared/config/walk-inside.com.nginx.conf.tpl" => "/data/nomad/client/alloc/90e27a8b-e7fa-4b7e-a7dc-b63beea6f44a/wi-nginx-task/custom/walk-inside.com.nginx.conf" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) rendering "/ops/shared/config/walk-inside.com.nginx.conf.tpl" => "/data/nomad/client/alloc/90e27a8b-e7fa-4b7e-a7dc-b63beea6f44a/wi-nginx-task/custom/walk-inside.com.nginx.conf" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 57e42e80aaac3ca747ab6a62a191595a Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.502Z [DEBUG] agent: (runner) checking template 57e42e80aaac3ca747ab6a62a191595a Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.502Z [DEBUG] agent: (runner) checking template d2fff3cdb405c48dc33350313fe7ca26 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template d2fff3cdb405c48dc33350313fe7ca26 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) enabling global quiescence for "78729eba1e2b937878cf101798d434d8" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) received template "d8e8fca2dc0f896fd7cb4cb0031ba249" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) checking template d8e8fca2dc0f896fd7cb4cb0031ba249 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) rendering "(dynamic)" => "/data/nomad/client/alloc/90e27a8b-e7fa-4b7e-a7dc-b63beea6f44a/wi-nginx-task/custom/test/xx" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) enabling global quiescence for "78729eba1e2b937878cf101798d434d8" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.503Z [DEBUG] agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) received template "d8e8fca2dc0f896fd7cb4cb0031ba249" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template d8e8fca2dc0f896fd7cb4cb0031ba249 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) rendering "(dynamic)" => "/data/nomad/client/alloc/90e27a8b-e7fa-4b7e-a7dc-b63beea6f44a/wi-nginx-task/custom/test/xx" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) checking template 57e42e80aaac3ca747ab6a62a191595a Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 57e42e80aaac3ca747ab6a62a191595a Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) checking template d2fff3cdb405c48dc33350313fe7ca26 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template d2fff3cdb405c48dc33350313fe7ca26 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) enabling global quiescence for "d8e8fca2dc0f896fd7cb4cb0031ba249" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) received template "d2fff3cdb405c48dc33350313fe7ca26" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.504Z [DEBUG] agent: (runner) checking template d8e8fca2dc0f896fd7cb4cb0031ba249 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) enabling global quiescence for "d8e8fca2dc0f896fd7cb4cb0031ba249" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.505Z [DEBUG] agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) received template "d2fff3cdb405c48dc33350313fe7ca26" from quiescence Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) initiating run Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template d8e8fca2dc0f896fd7cb4cb0031ba249 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 78729eba1e2b937878cf101798d434d8 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.505Z [DEBUG] agent: (runner) checking template 57e42e80aaac3ca747ab6a62a191595a Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template 57e42e80aaac3ca747ab6a62a191595a Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.506Z [DEBUG] agent: (runner) checking template d2fff3cdb405c48dc33350313fe7ca26 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) checking template d2fff3cdb405c48dc33350313fe7ca26 Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.506Z [DEBUG] agent: (runner) rendering "(dynamic)" => "/data/nomad/client/alloc/90e27a8b-e7fa-4b7e-a7dc-b63beea6f44a/wi-nginx-task/local/env" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) rendering "(dynamic)" => "/data/nomad/client/alloc/90e27a8b-e7fa-4b7e-a7dc-b63beea6f44a/wi-nginx-task/local/env" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.506Z [DEBUG] agent: (runner) diffing and updating dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.506Z [DEBUG] agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.506Z [DEBUG] agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.506Z [DEBUG] agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.506Z [DEBUG] agent: (runner) enabling global quiescence for "d2fff3cdb405c48dc33350313fe7ca26" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) health.service(wi-api|passing) is still needed Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) watching 1 dependencies Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) all templates rendered Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: agent: (runner) enabling global quiescence for "d2fff3cdb405c48dc33350313fe7ca26" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: 2024-08-09T15:58:13.645Z [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="212.267µs" Aug 09 15:58:13 ip-172-31-10-241 nomad[145250]: http: request complete: method=GET path=/v1/agent/health?type=client duration="212.267µs" ```
EugenKon commented 1 month ago

As temporary workaround we should not mount a rendered template file into container, instead we should mount folder into container and render a template into that folder.

tgross commented 1 month ago

Finally I managed to minimize the experiment and show that this is a Nomad bug which is critical and easy to fix: Just do not recreate the file. It should be rewritten instead.

The file write is happening via the embedded consul-template. What you're seeing is not a bug but intentional, because it's the only way to ensure that a write is atomic. See https://github.com/hashicorp/consul-template/issues/1410 for another example of someone reporting this non-bug behavior.

Your workaround described in https://github.com/hashicorp/nomad/issues/23691#issuecomment-2278753245 is almost certainly how you should be handling this kind of issue.