Closed washeeeq closed 23 hours ago
robholland
commented
3 days ago The hooks are now removed on main (replaced with a single job), that should improve things for you once that is released. The config issue will be unrelated, please can you check the configmap and see what line 30 of the config we generated there looks like?
washeeeq
commented
1 day ago Yes,
but that is a bit unclear to me, I tried to use busybox image to connect to the container but I see no config/docker.yaml there so probably you take config_template.yaml and somehow transform it to docker.yaml. So here is a configmap from the cluster:
apiVersion: v1
data:
config_template.yaml: |-
log:
stdout: true
level: "debug,info"
persistence:
defaultStore: default
visibilityStore: es-visibility
numHistoryShards: 512
datastores:
default:
sql:
pluginName: "postgres12"
driverName: "postgres12"
databaseName: "temporal"
connectAddr: "10.100.100.3:5432"
connectProtocol: "tcp"
user: temporal_app
password: "{{ .Env.TEMPORAL_STORE_PASSWORD }}"
maxConnLifetime: 1h
maxConns: 20
secretName: ""
visibility:
sql:
pluginName: "postgres12"
driverName: "postgres12"
databaseName: "temporal_visibility"
connectAddr: "10.100.100.3:5432"
connectProtocol: "tcp"
user: "temporal_visibility_app"
password: "{{ .Env.TEMPORAL_VISIBILITY_STORE_PASSWORD }}"
maxConnLifetime: 1h
maxConns: 20
secretName: ""
es-visibility:
elasticsearch:
version: "v7"
url:
scheme: "http"
host: "elasticsearch-master:9200"
username: ""
password: ""
logLevel: "error"
indices:
visibility: "temporal_visibility_v1_dev"
global:
membership:
name: temporal
maxJoinDuration: 30s
broadcastAddress: {{ default .Env.POD_IP "0.0.0.0" }}
pprof:
port: 7936
metrics:
tags:
type: frontend
prometheus:
timerType: histogram
listenAddress: "0.0.0.0:9090"
services:
frontend:
rpc:
grpcPort: 7233
membershipPort: 6933
bindOnIP: "0.0.0.0"
history:
rpc:
grpcPort: 7234
membershipPort: 6934
bindOnIP: "0.0.0.0"
matching:
rpc:
grpcPort: 7235
membershipPort: 6935
bindOnIP: "0.0.0.0"
worker:
rpc:
grpcPort: 7239
membershipPort: 6939
bindOnIP: "0.0.0.0"
clusterMetadata:
enableGlobalDomain: false
failoverVersionIncrement: 10
masterClusterName: "active"
currentClusterName: "active"
clusterInformation:
active:
enabled: true
initialFailoverVersion: 1
rpcName: "temporal-frontend"
rpcAddress: "127.0.0.1:7233"
dcRedirectionPolicy:
policy: "noop"
toDC: ""
archival:
status: "disabled"
publicClient:
hostPort: "temporal-frontend:7233"
dynamicConfigClient:
filepath: "/etc/temporal/dynamic_config/dynamic_config.yaml"
pollInterval: "10s"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"v1","data":{"config_template.yaml":"log:\n stdout: true\n
level: \"debug,info\"\n\npersistence:\n defaultStore: default\n
visibilityStore: es-visibility\n numHistoryShards: 512\n
datastores:\n default:\n sql:\n pluginName:
\"postgres12\"\n driverName: \"postgres12\"\n databaseName:
\"temporal\"\n connectAddr: \"10.100.100.3:5432\"\n
connectProtocol: \"tcp\"\n user: temporal_app\n password:
\"{{ .Env.TEMPORAL_STORE_PASSWORD }}\"\n maxConnLifetime:
1h\n maxConns: 20\n secretName: \"\"\n
tls:\n enabled: false\n visibility:\n sql:\n
pluginName: \"postgres12\"\n driverName: \"postgres12\"\n
databaseName: \"temporal_visibility\"\n connectAddr:
\"10.100.100.3:5432\"\n connectProtocol: \"tcp\"\n user:
\"temporal_visibility_app\"\n password: \"{{
.Env.TEMPORAL_VISIBILITY_STORE_PASSWORD }}\"\n maxConnLifetime:
1h\n maxConns: 20\n secretName: \"\"\n
tls:\n enabled: false\n es-visibility:\n
elasticsearch:\n version: \"v7\"\n
url:\n scheme: \"http\"\n host:
\"elasticsearch-master:9200\"\n username: \"\"\n
password: \"\"\n logLevel: \"error\"\n
indices:\n visibility:
\"temporal_visibility_v1_dev\"\n\nglobal:\n membership:\n name:
temporal\n maxJoinDuration: 30s\n broadcastAddress: {{ default
.Env.POD_IP \"0.0.0.0\" }}\n\n pprof:\n port: 7936\n\n metrics:\n
tags:\n type: frontend\n prometheus:\n timerType:
histogram\n listenAddress: \"0.0.0.0:9090\"\n\nservices:\n
frontend:\n rpc:\n grpcPort: 7233\n membershipPort:
6933\n bindOnIP: \"0.0.0.0\"\n\n history:\n rpc:\n grpcPort:
7234\n membershipPort: 6934\n bindOnIP: \"0.0.0.0\"\n\n
matching:\n rpc:\n grpcPort: 7235\n membershipPort:
6935\n bindOnIP: \"0.0.0.0\"\n\n worker:\n rpc:\n grpcPort:
7239\n membershipPort: 6939\n bindOnIP:
\"0.0.0.0\"\nclusterMetadata:\n enableGlobalDomain: false\n
failoverVersionIncrement: 10\n masterClusterName: \"active\"\n
currentClusterName: \"active\"\n clusterInformation:\n active:\n
enabled: true\n initialFailoverVersion: 1\n rpcName:
\"temporal-frontend\"\n rpcAddress:
\"127.0.0.1:7233\"\ndcRedirectionPolicy:\n policy: \"noop\"\n toDC:
\"\"\narchival:\n status: \"disabled\"\n\npublicClient:\n hostPort:
\"temporal-frontend:7233\"\n\ndynamicConfigClient:\n filepath:
\"/etc/temporal/dynamic_config/dynamic_config.yaml\"\n pollInterval:
\"10s\""},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"temporal","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"temporal","app.kubernetes.io/part-of":"temporal","app.kubernetes.io/version":"1.24.2","argocd.argoproj.io/instance":"temporal","helm.sh/chart":"temporal-0.43.0"},"name":"temporal-frontend-config","namespace":"temporal"}}
creationTimestamp: '2024-07-06T16:45:27Z'
labels:
app.kubernetes.io/instance: temporal
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: temporal
app.kubernetes.io/part-of: temporal
app.kubernetes.io/version: 1.24.2
argocd.argoproj.io/instance: temporal
helm.sh/chart: temporal-0.43.0
name: temporal-frontend-config
namespace: temporal
resourceVersion: '68519726'
uid: ba3537f0-b0fe-4492-8588-3498b2a8a0f8I think the problem is with this line:
password: "{{ .Env.TEMPORAL_VISIBILITY_STORE_PASSWORD }}"
claude gave a hint to do it this way:
password: "{{ {{ .Env.TEMPORAL_VISIBILITY_STORE_PASSWORD }} }}"
It seems our password contains special characters
robholland
commented
23 hours ago Ok, this is a known issue, this is fixed in the latest helm chart release.
robholland
commented
23 hours ago Closing now, but please open a new issue if you are still having configmap issues with latest release.
What are you really trying to do?
Describe the bug
Using GCP Postgres as default store and as visibility store. We are using a chart as a subchart:
Than adding a values to modify the deployment:
Now with such a config, temporal hangs on bringing up for pods: frontend, history, worker and one more.
After further investigation I found out that a batch job to create index (es-index-setup) is not starting. Probably wrong weight used: "helm.sh/hook-weight": "0"
If I add external: true the script is triggered but this hinders than initial deployment.
After the elastic is initilazed, frontend pod is starting but very quickly ends with:
Minimal Reproduction
Install ArgoCD on the Kubernetes cluster
create chart with a subchart of temporal
create application file for ArgoCD which triggers the deployment