search

edgexfoundry / edgex-compose

EdgeX Foundry Docker Compose release compose files and tools for building EdgeX compose files
Apache License 2.0
81 stars 115 forks source link

edgex-security-proxy-setup service fails to start after Ubuntu reboot #434

Closed ethan256 closed 4 months ago

ethan256 commented 4 months ago

🐞 Bug Report

Affected Services [REQUIRED]

The issue is located in: When Ubuntu reboots, the proxy-setup service fails to start. This in turn causes the nginx container to fail to start properly. ### Is this a regression? Yes, the previous version in which this bug was not present was: .... ### Description and Minimal Reproduction [**REQUIRED**] ```shell git clone git@github.com:edgexfoundry/edgex-compose.git git checkout tags/v3.1.0 -b release/3.1.0 cd compose-builder && make gen mqtt-broker ds-opcua ds-modbus ds-mqtt app-mqtt app-metric-process arm64 docker compose up -d # wait service running and check docker ps # reboot system sudo reboot ``` ## πŸ”₯ Exception or Error
image

🌍 Your Environment

Deployment Environment: os： ubuntu 20.04 TLS arch: arm64 docker info: image

EdgeX Version [REQUIRED]: 3.1.0

Anything else relevant?

cloudxxx8 commented 4 months ago

Please provide the logs of the two services which cannot be restarted

ethan256 commented 4 months ago

Please provide the logs of the two services which cannot be restarted

edgex-nginx service logs:

Awaiting proxy-setup to finish before starting NGINX
Fri Feb 23 01:49:01 UTC 2024 Executing waitFor with waiting on tcp://edgex-security-proxy-setup:54325
level=INFO ts=2024-02-23T01:49:01.342201226Z app=security-bootstrapper source=config.go:731 msg="Loading configuration file from /edgex-init/res/configuration.yaml"
level=INFO ts=2024-02-23T01:49:01.344280683Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Host' by environment variable: STAGEGATE_REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2024-02-23T01:49:01.344418341Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Port' by environment variable: STAGEGATE_REGISTRY_PORT=8500"
level=INFO ts=2024-02-23T01:49:01.344467046Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/WaitFor/Timeout' by environment variable: STAGEGATE_WAITFOR_TIMEOUT=60s"
level=INFO ts=2024-02-23T01:49:01.344513127Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/StartPort' by environment variable: STAGEGATE_BOOTSTRAPPER_STARTPORT=54321"
level=INFO ts=2024-02-23T01:49:01.344554541Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Port' by environment variable: STAGEGATE_DATABASE_PORT=6379"
level=INFO ts=2024-02-23T01:49:01.344623662Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ReadyPort' by environment variable: STAGEGATE_REGISTRY_READYPORT=54324"
level=INFO ts=2024-02-23T01:49:01.344670909Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Host' by environment variable: STAGEGATE_SECRETSTORESETUP_HOST=edgex-security-secretstore-setup"
level=INFO ts=2024-02-23T01:49:01.344723989Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Ready/ToRunPort' by environment variable: STAGEGATE_READY_TORUNPORT=54329"
level=INFO ts=2024-02-23T01:49:01.344769194Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/ReadyPort' by environment variable: STAGEGATE_DATABASE_READYPORT=6379"
level=INFO ts=2024-02-23T01:49:01.344821108Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Tokens/ReadyPort' by environment variable: STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT=54322"
level=INFO ts=2024-02-23T01:49:01.34486223Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Host' by environment variable: STAGEGATE_DATABASE_HOST=edgex-redis"
level=INFO ts=2024-02-23T01:49:01.344901311Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/Host' by environment variable: STAGEGATE_BOOTSTRAPPER_HOST=edgex-security-bootstrapper"
level=INFO ts=2024-02-23T01:49:01.344939517Z app=security-bootstrapper source=config.go:251 msg="Private configuration loaded from file with 12 overrides applied"
level=INFO ts=2024-02-23T01:49:01.350045415Z app=security-bootstrapper source=command.go:119 msg="Security bootstrapper running waitFor"
level=INFO ts=2024-02-23T01:49:01.350433308Z app=security-bootstrapper source=command.go:144 msg="Waiting for: [tcp://edgex-security-proxy-setup:54325] with timeout: [1m0s]"
level=INFO ts=2024-02-23T01:49:01.449468529Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:02.456471353Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:03.463994142Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:04.468941707Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:05.474056884Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:06.479768187Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:07.486644623Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:08.49341202Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:09.498822328Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:10.504067268Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:11.510870192Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:12.519488044Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:13.529842702Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:14.540498462Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:15.55025519Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:16.559884797Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:17.570410678Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:18.580510498Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:19.590705141Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:20.600919655Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:21.612936887Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:22.623356213Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:23.63351001Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:24.639188492Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:25.644532532Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:26.649923232Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:27.654913202Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:28.660167403Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:29.665085328Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:30.670475721Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:31.676308832Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:32.680741442Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:33.685844549Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:34.690853449Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:35.695888802Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:36.704094417Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.9:54325: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:37.71159707Z app=security-bootstrapper source=command.go:214 msg="Connected to tcp://edgex-security-proxy-setup:54325"
Fri Feb 23 01:49:37 UTC 2024 Starting NGINX ...
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/edgex-default.conf.template to /etc/nginx/conf.d/edgex-default.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/edgex-custom-rewrites.inc.template to /etc/nginx/conf.d/edgex-custom-rewrites.inc
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/generated-routes.inc.template to /etc/nginx/conf.d/generated-routes.inc
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
Awaiting proxy-setup to finish before starting NGINX
Fri Feb 23 01:50:59 UTC 2024 Executing waitFor with waiting on tcp://edgex-security-proxy-setup:54325
level=INFO ts=2024-02-23T01:50:59.291407057Z app=security-bootstrapper source=config.go:731 msg="Loading configuration file from /edgex-init/res/configuration.yaml"
level=INFO ts=2024-02-23T01:50:59.318500346Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/ReadyPort' by environment variable: STAGEGATE_DATABASE_READYPORT=6379"
level=INFO ts=2024-02-23T01:50:59.318724916Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Host' by environment variable: STAGEGATE_REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2024-02-23T01:50:59.32415279Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Tokens/ReadyPort' by environment variable: STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT=54322"
level=INFO ts=2024-02-23T01:50:59.324342361Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/WaitFor/Timeout' by environment variable: STAGEGATE_WAITFOR_TIMEOUT=60s"
level=INFO ts=2024-02-23T01:50:59.324397191Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Ready/ToRunPort' by environment variable: STAGEGATE_READY_TORUNPORT=54329"
level=INFO ts=2024-02-23T01:50:59.324441813Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/StartPort' by environment variable: STAGEGATE_BOOTSTRAPPER_STARTPORT=54321"
level=INFO ts=2024-02-23T01:50:59.324485561Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Port' by environment variable: STAGEGATE_REGISTRY_PORT=8500"
level=INFO ts=2024-02-23T01:50:59.324526975Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ReadyPort' by environment variable: STAGEGATE_REGISTRY_READYPORT=54324"
level=INFO ts=2024-02-23T01:50:59.324567222Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Host' by environment variable: STAGEGATE_SECRETSTORESETUP_HOST=edgex-security-secretstore-setup"
level=INFO ts=2024-02-23T01:50:59.324652675Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Host' by environment variable: STAGEGATE_DATABASE_HOST=edgex-redis"
level=INFO ts=2024-02-23T01:50:59.324701381Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Port' by environment variable: STAGEGATE_DATABASE_PORT=6379"
level=INFO ts=2024-02-23T01:50:59.32474367Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/Host' by environment variable: STAGEGATE_BOOTSTRAPPER_HOST=edgex-security-bootstrapper"
level=INFO ts=2024-02-23T01:50:59.324784501Z app=security-bootstrapper source=config.go:251 msg="Private configuration loaded from file with 12 overrides applied"
level=INFO ts=2024-02-23T01:50:59.328801961Z app=security-bootstrapper source=command.go:119 msg="Security bootstrapper running waitFor"
level=INFO ts=2024-02-23T01:50:59.330818712Z app=security-bootstrapper source=command.go:144 msg="Waiting for: [tcp://edgex-security-proxy-setup:54325] with timeout: [1m0s]"
level=INFO ts=2024-02-23T01:50:59.353991368Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:00.391767544Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:01.414730593Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:02.441794763Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:03.460643814Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:04.470707532Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:05.485093775Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:06.502201601Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:07.523237351Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:08.545860087Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:09.565701827Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:10.576197393Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:11.589803976Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:12.609795291Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:13.624630131Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:14.638646314Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:15.649267664Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:16.659442792Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:17.672530907Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:18.681571521Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:19.698592518Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:20.708930386Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:21.719456951Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:22.73200814Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:23.746251767Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:24.770453364Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:25.787381242Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:26.796360818Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:27.812421708Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:28.818717783Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:29.839333101Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:30.860659917Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:31.880972878Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:32.902017378Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:33.922497162Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:34.94767724Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:35.964589661Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:36.984808003Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:38.005675597Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:39.026550524Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:40.047774595Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:41.068441493Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:42.098818376Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:43.112866682Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:44.134104003Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:45.154824522Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:46.175700199Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:47.195163421Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:48.214220543Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:49.235309873Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:50.251291227Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:51.271686391Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:52.294543737Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:53.314684209Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:54.334295422Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:55.345747555Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:56.366152135Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:57.379353493Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=INFO ts=2024-02-23T01:51:58.399738908Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-proxy-setup on 127.0.0.11:53: server misbehaving. Sleeping 1s"
level=ERROR ts=2024-02-23T01:51:59.330478431Z app=security-bootstrapper source=init.go:90 msg="Timeout after 1m0s waiting on dependencies to become available: [tcp://edgex-security-proxy-setup:54325]"

edgex-security-proxy-setup service logs:

Awaiting ReadyToRun signal prior to starting running proxy-setup tasks
Fri Feb 23 09:49:00 CST 2024 Executing waitFor for edgex-security-proxy-setup with waiting on   tcp://edgex-security-bootstrapper:54329
level=INFO ts=2024-02-23T01:49:00.393658922Z app=security-bootstrapper source=config.go:731 msg="Loading configuration file from /edgex-init/res/configuration.yaml"
level=INFO ts=2024-02-23T01:49:00.399420155Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/Host' by environment variable: STAGEGATE_BOOTSTRAPPER_HOST=edgex-security-bootstrapper"
level=INFO ts=2024-02-23T01:49:00.399646474Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Host' by environment variable: STAGEGATE_DATABASE_HOST=edgex-redis"
level=INFO ts=2024-02-23T01:49:00.39971822Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Tokens/ReadyPort' by environment variable: STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT=54322"
level=INFO ts=2024-02-23T01:49:00.399764301Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/StartPort' by environment variable: STAGEGATE_BOOTSTRAPPER_STARTPORT=54321"
level=INFO ts=2024-02-23T01:49:00.399817381Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/WaitFor/Timeout' by environment variable: STAGEGATE_WAITFOR_TIMEOUT=60s"
level=INFO ts=2024-02-23T01:49:00.399908083Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Host' by environment variable: STAGEGATE_REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2024-02-23T01:49:00.39995708Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/ReadyPort' by environment variable: STAGEGATE_DATABASE_READYPORT=6379"
level=INFO ts=2024-02-23T01:49:00.399998203Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
level=INFO ts=2024-02-23T01:49:00.400042242Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Ready/ToRunPort' by environment variable: STAGEGATE_READY_TORUNPORT=54329"
level=INFO ts=2024-02-23T01:49:00.400082489Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ReadyPort' by environment variable: STAGEGATE_REGISTRY_READYPORT=54324"
level=INFO ts=2024-02-23T01:49:00.400122445Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Host' by environment variable: STAGEGATE_SECRETSTORESETUP_HOST=edgex-security-secretstore-setup"
level=INFO ts=2024-02-23T01:49:00.400165318Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Port' by environment variable: STAGEGATE_REGISTRY_PORT=8500"
level=INFO ts=2024-02-23T01:49:00.400207607Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Port' by environment variable: STAGEGATE_DATABASE_PORT=6379"
level=INFO ts=2024-02-23T01:49:00.400249313Z app=security-bootstrapper source=config.go:251 msg="Private configuration loaded from file with 13 overrides applied"
level=INFO ts=2024-02-23T01:49:00.40194321Z app=security-bootstrapper source=command.go:119 msg="Security bootstrapper running waitFor"
level=INFO ts=2024-02-23T01:49:00.409201184Z app=security-bootstrapper source=command.go:144 msg="Waiting for: [tcp://edgex-security-bootstrapper:54329] with timeout: [1m0s]"
level=INFO ts=2024-02-23T01:49:00.458686714Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:01.462165757Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:02.468519953Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:03.475671472Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:04.48181684Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:05.486206686Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:06.492541534Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:07.498904084Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:08.507260968Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:09.512427458Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:10.518100247Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:11.523532713Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:12.528867472Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:13.538608793Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:14.550255284Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:15.560283246Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:16.567367049Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:17.577345214Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:18.588015791Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:19.599124753Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:20.613789245Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:21.623825015Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:22.634046895Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:23.644170943Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:24.654058502Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:25.660498558Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:26.665610734Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:27.671387573Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:28.676511989Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:29.681698523Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:30.688567284Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:31.695067688Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:32.699145361Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:33.704612154Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:34.707958364Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:35.712252929Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:36.728750121Z app=security-bootstrapper source=command.go:214 msg="Connected to tcp://edgex-security-bootstrapper:54329"
Fri Feb 23 09:49:36 CST 2024 Starting edgex-security-proxy-setup ...
Default TLS certificate created.  Recommend replace with your own.
Fri Feb 23 09:49:37 CST 2024 CORS settings dump ...
EDGEX_SERVICE_CORSCONFIGURATION_CORSALLOWCREDENTIALS='false'
EDGEX_SERVICE_CORSCONFIGURATION_CORSALLOWEDHEADERS='Authorization, Accept, Accept-Language, Content-Language, Content-Type, X-Correlation-ID'
EDGEX_SERVICE_CORSCONFIGURATION_CORSALLOWEDMETHODS='GET, POST, PUT, PATCH, DELETE'
EDGEX_SERVICE_CORSCONFIGURATION_CORSALLOWEDORIGIN='https://localhost'
EDGEX_SERVICE_CORSCONFIGURATION_CORSEXPOSEHEADERS='Cache-Control, Content-Language, Content-Length, Content-Type, Expires, Last-Modified, Pragma, X-Correlation-ID'
EDGEX_SERVICE_CORSCONFIGURATION_CORSMAXAGE='3600'
EDGEX_SERVICE_CORSCONFIGURATION_ENABLECORS='false'
Fri Feb 23 09:49:37 CST 2024 Generating default NGINX config ...
device-modbus.http://edgex-device-modbus:59901 device-modbus edgex-device-modbus 59901
device-mqtt.http://edgex-device-mqtt:59982 device-mqtt edgex-device-mqtt 59982
Fri Feb 23 09:49:37 CST 2024 Signalling reverse proxy to start (loop forever) ...
level=INFO ts=2024-02-23T01:49:37.375746103Z app=security-bootstrapper source=config.go:731 msg="Loading configuration file from /edgex-init/res/configuration.yaml"
level=INFO ts=2024-02-23T01:49:37.378283449Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Host' by environment variable: STAGEGATE_SECRETSTORESETUP_HOST=edgex-security-secretstore-setup"
level=INFO ts=2024-02-23T01:49:37.378490519Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Port' by environment variable: STAGEGATE_REGISTRY_PORT=8500"
level=INFO ts=2024-02-23T01:49:37.378550599Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Host' by environment variable: STAGEGATE_DATABASE_HOST=edgex-redis"
level=INFO ts=2024-02-23T01:49:37.378599888Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Host' by environment variable: STAGEGATE_REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2024-02-23T01:49:37.378750379Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/ReadyPort' by environment variable: STAGEGATE_DATABASE_READYPORT=6379"
level=INFO ts=2024-02-23T01:49:37.378867621Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Tokens/ReadyPort' by environment variable: STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT=54322"
level=INFO ts=2024-02-23T01:49:37.378969115Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/WaitFor/Timeout' by environment variable: STAGEGATE_WAITFOR_TIMEOUT=60s"
level=INFO ts=2024-02-23T01:49:37.379035903Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
level=INFO ts=2024-02-23T01:49:37.379102107Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Ready/ToRunPort' by environment variable: STAGEGATE_READY_TORUNPORT=54329"
level=INFO ts=2024-02-23T01:49:37.379146438Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/StartPort' by environment variable: STAGEGATE_BOOTSTRAPPER_STARTPORT=54321"
level=INFO ts=2024-02-23T01:49:37.379198643Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ReadyPort' by environment variable: STAGEGATE_REGISTRY_READYPORT=54324"
level=INFO ts=2024-02-23T01:49:37.37925289Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Port' by environment variable: STAGEGATE_DATABASE_PORT=6379"
level=INFO ts=2024-02-23T01:49:37.379313552Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/Host' by environment variable: STAGEGATE_BOOTSTRAPPER_HOST=edgex-security-bootstrapper"
level=INFO ts=2024-02-23T01:49:37.379366633Z app=security-bootstrapper source=config.go:251 msg="Private configuration loaded from file with 13 overrides applied"
level=INFO ts=2024-02-23T01:49:37.38383936Z app=security-bootstrapper source=command.go:81 msg="Security bootstrapper running listenTcp"
level=INFO ts=2024-02-23T01:49:37.38721957Z app=security-bootstrapper source=listener.go:58 msg="Security bootstrapper starts listening on tcp://edgex-security-proxy-setup:54325"
level=INFO ts=2024-02-23T01:49:37.710131826Z app=security-bootstrapper source=listener.go:67 msg="Accepted connection on edgex-security-proxy-setup:54325"
ethan256 commented 4 months ago

Please provide the logs of the two services which cannot be restarted

edgex-security-proxy-setup state： image

ethan256 commented 4 months ago

/etc/docker/daemon.json:

{
  "live-restore": true
}
cloudxxx8 commented 4 months ago

another service which cannot be started normally is common-config-bootstraper please check its log

ethan256 commented 4 months ago

another service which cannot be started normally is common-config-bootstraper please check its log

edgex-core-common-config-bootstrapper logs:

Script for waiting on security bootstrapping ready-to-run
Fri Feb 23 09:49:01 CST 2024 Executing waitFor with /entrypoint.sh /core-common-config-bootstrapper -cp=consul.http://edgex-core-consul:8500 waiting on tcp://edgex-security-bootstrapper:54329
level=INFO ts=2024-02-23T01:49:01.982867893Z app=security-bootstrapper source=config.go:731 msg="Loading configuration file from /edgex-init/res/configuration.yaml"
level=INFO ts=2024-02-23T01:49:01.987555566Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Ready/ToRunPort' by environment variable: STAGEGATE_READY_TORUNPORT=54329"
level=INFO ts=2024-02-23T01:49:01.988103575Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/ReadyPort' by environment variable: STAGEGATE_DATABASE_READYPORT=6379"
level=INFO ts=2024-02-23T01:49:01.988466386Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ReadyPort' by environment variable: STAGEGATE_REGISTRY_READYPORT=54324"
level=INFO ts=2024-02-23T01:49:01.988561172Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Host' by environment variable: STAGEGATE_REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2024-02-23T01:49:01.988609002Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Host' by environment variable: STAGEGATE_SECRETSTORESETUP_HOST=edgex-security-secretstore-setup"
level=INFO ts=2024-02-23T01:49:01.988809657Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Tokens/ReadyPort' by environment variable: STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT=54322"
level=INFO ts=2024-02-23T01:49:01.988871486Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/StartPort' by environment variable: STAGEGATE_BOOTSTRAPPER_STARTPORT=54321"
level=INFO ts=2024-02-23T01:49:01.988915233Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/WaitFor/Timeout' by environment variable: STAGEGATE_WAITFOR_TIMEOUT=60s"
level=INFO ts=2024-02-23T01:49:01.988969188Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Port' by environment variable: STAGEGATE_REGISTRY_PORT=8500"
level=INFO ts=2024-02-23T01:49:01.989011478Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Host' by environment variable: STAGEGATE_DATABASE_HOST=edgex-redis"
level=INFO ts=2024-02-23T01:49:01.989066016Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/Host' by environment variable: STAGEGATE_BOOTSTRAPPER_HOST=edgex-security-bootstrapper"
level=INFO ts=2024-02-23T01:49:01.989110347Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Port' by environment variable: STAGEGATE_DATABASE_PORT=6379"
level=INFO ts=2024-02-23T01:49:01.989234006Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
level=INFO ts=2024-02-23T01:49:01.989344249Z app=security-bootstrapper source=config.go:251 msg="Private configuration loaded from file with 13 overrides applied"
level=INFO ts=2024-02-23T01:49:01.991858554Z app=security-bootstrapper source=command.go:119 msg="Security bootstrapper running waitFor"
level=INFO ts=2024-02-23T01:49:01.992948446Z app=security-bootstrapper source=command.go:144 msg="Waiting for: [tcp://edgex-security-bootstrapper:54329] with timeout: [1m0s]"
level=INFO ts=2024-02-23T01:49:02.012440218Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:03.020568929Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:04.036292466Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:05.048121735Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:06.054719361Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:07.0632092Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:08.073709163Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:09.081855682Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:10.089548725Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:11.095110393Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:12.100912667Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:13.107597259Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:14.117822881Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:15.128128244Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:16.140573264Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:17.15109848Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:18.164327694Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:19.171180418Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:20.182452577Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:21.18770389Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:22.200941587Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:23.213166384Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:24.22205938Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:25.228463024Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:26.236972327Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:27.243640155Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:28.251702061Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:29.256457248Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:30.262311072Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:31.267795665Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:32.275867471Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:33.280574812Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:34.28773025Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:35.293632385Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:36.302560456Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp 172.18.0.3:54329: connect: connection refused. Sleeping 1s"
level=INFO ts=2024-02-23T01:49:37.306734398Z app=security-bootstrapper source=command.go:214 msg="Connected to tcp://edgex-security-bootstrapper:54329"
Fri Feb 23 09:49:37 CST 2024 Starting /entrypoint.sh /core-common-config-bootstrapper -cp=consul.http://edgex-core-consul:8500 ...
level=INFO ts=2024-02-23T01:49:37.75458378Z app=core-common-config-bootstrapper source=main.go:63 msg="Core Common Config is starting"
level=INFO ts=2024-02-23T01:49:37.755783624Z app=core-common-config-bootstrapper source=secret.go:69 msg="Creating SecretClient"
level=INFO ts=2024-02-23T01:49:37.757463521Z app=core-common-config-bootstrapper source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
level=INFO ts=2024-02-23T01:49:37.75797274Z app=core-common-config-bootstrapper source=secret.go:172 msg="SecretStore information created with 1 overrides applied"
level=INFO ts=2024-02-23T01:49:37.75810894Z app=core-common-config-bootstrapper source=secret.go:79 msg="Reading secret store configuration and authentication token"
level=INFO ts=2024-02-23T01:49:37.758539122Z app=core-common-config-bootstrapper source=secret.go:216 msg="load token from file"
level=INFO ts=2024-02-23T01:49:37.759246371Z app=core-common-config-bootstrapper source=secret.go:101 msg="Attempting to create secret client"
level=INFO ts=2024-02-23T01:49:37.803535585Z app=core-common-config-bootstrapper source=secret.go:112 msg="Created SecretClient"
level=INFO ts=2024-02-23T01:49:37.804153006Z app=core-common-config-bootstrapper source=secret.go:117 msg="SecretsFile not set, skipping seeding of service secrets."
level=INFO ts=2024-02-23T01:49:37.80456948Z app=core-common-config-bootstrapper source=main.go:75 msg="Secret Provider created"
level=INFO ts=2024-02-23T01:49:37.806253753Z app=core-common-config-bootstrapper source=secrets.go:277 msg="kick off token renewal with interval: 30m0s"
level=INFO ts=2024-02-23T01:49:37.882797582Z app=core-common-config-bootstrapper source=secrets.go:149 msg="successfully generated Consul token for service core-common-config-bootstrapper"
level=INFO ts=2024-02-23T01:49:37.883496665Z app=core-common-config-bootstrapper source=main.go:84 msg="Got Config Provider Access Token with length 36"
level=INFO ts=2024-02-23T01:49:37.883939388Z app=core-common-config-bootstrapper source=config.go:718 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/core-common-config-bootstrapper"
level=INFO ts=2024-02-23T01:49:37.893994857Z app=core-common-config-bootstrapper source=main.go:122 msg="Pushing common configuration. It doesn't exists or overwrite flag is set"
level=INFO ts=2024-02-23T01:49:37.96240285Z app=core-common-config-bootstrapper source=main.go:167 msg="Using common configuration from res/configuration.yaml"
level=INFO ts=2024-02-23T01:49:37.97175078Z app=core-common-config-bootstrapper source=variables.go:462 msg="Variables override of 'all-services/MessageBus/Host' by environment variable: ALL_SERVICES_MESSAGEBUS_HOST=edgex-redis"
level=INFO ts=2024-02-23T01:49:37.971903312Z app=core-common-config-bootstrapper source=variables.go:462 msg="Variables override of 'all-services/Database/Host' by environment variable: ALL_SERVICES_DATABASE_HOST=edgex-redis"
level=INFO ts=2024-02-23T01:49:37.97199839Z app=core-common-config-bootstrapper source=variables.go:462 msg="Variables override of 'all-services/Registry/Host' by environment variable: ALL_SERVICES_REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2024-02-23T01:49:37.972077427Z app=core-common-config-bootstrapper source=variables.go:462 msg="Variables override of 'app-services/Clients/core-metadata/Host' by environment variable: APP_SERVICES_CLIENTS_CORE_METADATA_HOST=edgex-core-metadata"
level=INFO ts=2024-02-23T01:49:37.972141298Z app=core-common-config-bootstrapper source=variables.go:462 msg="Variables override of 'device-services/Clients/core-metadata/Host' by environment variable: DEVICE_SERVICES_CLIENTS_CORE_METADATA_HOST=edgex-core-metadata"
level=INFO ts=2024-02-23T01:49:37.972199919Z app=core-common-config-bootstrapper source=main.go:250 msg="Common configuration loaded from file with 5 overrides applied"
level=INFO ts=2024-02-23T01:49:38.754410106Z app=core-common-config-bootstrapper source=main.go:213 msg="Common configuration has been pushed to into Configuration Provider with overrides applied"
level=INFO ts=2024-02-23T01:49:38.754651008Z app=core-common-config-bootstrapper source=main.go:134 msg="Core Common Config exiting"

docker state

image

cloudxxx8 commented 4 months ago

@ethan256 I just figured out the issue. All your logs look normal, becuase they were the log before the reboot. I tried to add restart: always into the service of security-proxy-setup, and it works fine. Please try to use this workaround for now.

@lenny-intel I noticed that we don't define restart: always in security-proxy-setup, and it blocks the starting up of nginx https://github.com/edgexfoundry/edgex-compose/blob/55680017ff1b7c9db23deb52a21b27b28a7812ac/docker-compose.yml#L917 Do you know the reason or just a missing? The same situation for edgex-core-common-config-bootstrapper, but it doesn't impact anything as far as I know.

ethan256 commented 4 months ago

@ethan256 I just figured out the issue. All your logs look normal, becuase they were the log before the reboot. I tried to add restart: always into the service of security-proxy-setup, and it works fine. Please try to use this workaround for now.

@lenny-intel I noticed that we don't define restart: always in security-proxy-setup, and it blocks the starting up of nginx https://github.com/edgexfoundry/edgex-compose/blob/55680017ff1b7c9db23deb52a21b27b28a7812ac/docker-compose.yml#L917 Do you know the reason or just a missing? The same situation for edgex-core-common-config-bootstrapper, but it doesn't impact anything as far as I know.

Thank you. That's fine.

lenny-goodell commented 4 months ago

@lenny-intel I noticed that we don't define restart: always in security-proxy-setup, and it blocks the starting up of nginx https://github.com/edgexfoundry/edgex-compose/blob/55680017ff1b7c9db23deb52a21b27b28a7812ac/docker-compose.yml#L917 Do you know the reason or just a missing? The same situation for edgex-core-common-config-bootstrapper, but it doesn't impact anything as far as I know.

@cloudxxx8 , not sure on security-proxy-setup, but looking at the logs on a successful start-up, it seems that it should have restart: always since as you pointed out it blocks the starting up of nginx. I think in the past before security-bootstrapper, it was a one-shot service.

level=INFO ts=2024-02-26T20:37:30.697041104Z app=security-bootstrapper source=listener.go:67 msg="Accepted connection on edgex-security-proxy-setup:54325"

edgex-core-common-config-bootstrapper is a one shot service. We have entrypoint script that keeps the container up after the service exits. If this ever exits, then the service would continue to start/exit/restart continuously if we used restart: always

cloudxxx8 commented 4 months ago

@cherrycl please help add restart: always in security-proxy-setup for both main and napa branches to resolve this issue