Open maximemf opened 11 months ago
Try setting the env HELMFILE_USE_CONTEXT_NAMESPACE=true
and see what you get. The readme has a few details and/or look at the src to see how it changes the output.
From what I understand, it's only useful if the namespace is set in the Application, to kinda ignore it and use the context instead, but that's not even our case. We tried setting this variable nonetheless, but it didn't change anything. I guess we'll have to debug the script manually to find where the issue could come from 😕
Ok, given that we’re templating everything entirely and then argo applies the values in a single operation it posses a challenge (to do something like post-processing) because the context is lost at that point.
I think helmfile has an ability to do a post render hook on a per release basis where hypothetically you could find all namespaced assets and try to inject the ns if not present.
I've done some research and I understand better the origin of these errors, with helm template not using the namespace from the context for some obscure reasons that seem to be a great source of debate in the community (for instance https://github.com/helm/helm/issues/3553).
But after some additional investigations, I found out that Helmfile already implements a "secret feature" to get around this issue, which is the forceNamespace parameter, to be defined for each release. This parameter makes Helmfile complete the manifests generated by helmfile template with the namespace from the configuration (see https://github.com/roboll/helmfile/issues/326#issuecomment-805222638 🙏).
So our errors have been fixed, thank you for your reactivity, we'll keep using your great plugin, and come back to you if we face any further issue 😄
Wow great research and references! Thanks for sharing your findings.
Hello,
Thank you for sharing issue with namespaces - really helped me solving the same issue this issue creator stumbled on. But after that I do have another issue:
When I want to create / sync for the first time, namespaces which are provided in helmfile
namespace: test forceNamespace: test
are not being created and argo-cd throws an error that "Sync Failed", because "Nnamespaces "test" not found"
Am I missing some additional flag or env added to deployment? I am currently using only HELMFILE_GLOBAL_OPTIONS="-e test" as it is environment based deployment.
Thank you.
Does this help that? https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#create-namespace
Yes, that helps. Thank you!
So just to be clear.. It is not enough to simply specify namespaces in helmfile and additional stuff (as mentioned in your answer) needs to be deployed. Can this be added to helmfile deployment or has to be done separately, I mean before deployment?
I should note that my helmfile contains multiple namespaces that needs to be created.
That's another issue we also faced after using the forceNamespace parameter of Helmfile, and we were not able to fix it with the CreateNamespace option of Argo.
https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/#create-namespace
Note that the namespace to be created must be informed in the spec.destination.namespace field of the Application resource.
As we have a multi-namespaces Application, we shouldn't set the spec.destination.namespace
field, and so the CreateNamespace option is not working.
We didn't work more on this yet, but I guess we'll have to craft something as a workaround, like preInstall Helmfile hooks to create the namespaces, but if anyone have a better idea I'll take it 🙂
What have I did to go as a workaround:
{{- if .Values.namespaces.test.enabled }}
apiVersion: v1
kind: Namespace
metadata:
name: {{ .Values.namespaces.test.name }}
{{- end }}
and then added in values files values/keys respectively. Not the cleanest workaround, but it works. Although, another issue appears. For example my full helmfile deployment contains prometheus-stack deployment and for some reason argo-CD does not install CRDs, even if I set in prometheus values file installCRDs: true
If any one was looking at this and still needed ideas. I tried using the HELMFILE_INIT_SCRIPT_FILE
, just created a basic bash script. It can create namespaces on remote clusters or just locally in the cluster. I use environment variables so this determines which releases you want to target and grabs the namespace to create and the hooks to run on each release.
It is a little ugly, but it runs quickly, and efficiently.
create-ns.sh
#!/bin/bash
function KUBE_CONF {
KUBE_CONFIG_PATH="/tmp/kube.yaml"
# Get config
SECRETCONFIGJSON=$(kubectl get secret "server-${ENVIRONMENT}" -n argocd -o jsonpath='{.data.config}' | base64 --decode | jq .)
CLIENTCERT=$(echo ${SECRETCONFIGJSON} | jq '.tlsClientConfig.certData')
CLIENTKEY=$(echo ${SECRETCONFIGJSON} | jq '.tlsClientConfig.keyData')
CA=$(echo ${SECRETCONFIGJSON} | jq '.tlsClientConfig.caData')
# Get Server
SECRETSERVER=$(kubectl get secret "server-${ENVIRONMENT}" -n argocd -o jsonpath='{.data.server}' | base64 --decode)
CONFIG="---
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ${CA}
server: ${SECRETSERVER}
name: default
contexts:
- context:
cluster: default
user: default
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
user:
client-certificate-data: ${CLIENTCERT}
client-key-data: ${CLIENTKEY}
"
# Create temporary kubeconfig
echo "$CONFIG" | yq eval-all '.' > /tmp/kube.yaml
export KUBECONFIG="${KUBE_CONFIG_PATH}"
}
function RUN_HELM_HOOKS {
HELMFILE_LOCATION="${PWD}/helmfile.yaml"
# Start helmfile hooks
HELMFILE_RELEASES=$(helmfile -f ${HELMFILE_LOCATION} list ${HELMFILE_GLOBAL_OPTIONS} --output json | jq -r '.[] | select(.enabled) | "\(.name)=\(.namespace)"')
for r in $HELMFILE_RELEASES; do
NAME=$(echo "$r" | grep -oE "^[^=]+")
NS=$(echo "$r" | grep -oE "[^=]+$")
# Check NS
kubectl get ns "$NS" &>/dev/null
if [[ $? -ne 0 ]]; then
kubectl create ns "$NS"
else
continue
fi
# Check Hooks
HELMFILE_HOOKS=$(rname="$NAME" yq eval '.releases[] | select(.name == env(rname)) | .hooks.[].command' ${HELMFILE_LOCATION})
if [ -z "$HELMFILE_HOOKS" ]; then
continue
fi
for h in $HELMFILE_HOOKS; do
h=$(basename "$h")
if [ $h == "Your hook name 1" ]; then
# Run your hook
fi
if [ $h == "Your hook name 2" ]; then
# Run the second hook
fi
done
done
}
function CLEAN_UP {
unset KUBECONFIG
rm -f $KUBE_CONFIG_PATH
}
# Get the environment based on what you set "-e" to
ENVIRONMENT=$(echo "$HELMFILE_GLOBAL_OPTIONS" | grep -oP '\-e \K.*')
# IF it is a local deployment in the cluster, only run helm hooks. The pod knows the cluster config
if [[ "${ENVIRONMENT}" == *"Your local environment variable"* ]]; then
RUN_HELM_HOOKS
else
KUBE_CONF
RUN_HELM_HOOKS
fi
CLEAN_UP
Just mount this in the helmfile plugin pod, and set the parameters in the app:
- name: HELMFILE_INIT_SCRIPT_FILE
value: '/tmp/scripts/create-ns.sh'
Hello,
We're just starting to use your plugin, and we're facing an issue. Maybe we're missing something.
We have a Helmfile configuration with many Helm releases to be installed in different namespaces. The namespaces are defined in the Helmfile configuration for each release.
But when we want to synchronize the Argo Application related to this Helmfile (with no namespace defined in the Application), we have errors like this for each resource:
After some investigations, these errors only happen for Helm charts that don't define the namespace metadata field in the charts templates. There is no error when the field that corresponds to the namespace in the metadata section is defined in the templates, in this case the right namespace defined in the Helmfile configuration is taken into account. But most of the Helm charts are not defining this value by default in the templates...
It seems to not be able to retrieve or to apply the namespace from the context, that we define in the Helmfile configuration, if the namespace field is not already present in the chart.
Do you have any idea about what could be the issue?