welcome[bot]
commented
5 months ago Thank you for opening your first issue here! 🛠
Closed nuhaj closed 3 months ago
welcome[bot]
commented
5 months ago Thank you for opening your first issue here! 🛠
runllm[bot]
commented
5 months ago The issue you're experiencing seems related to the configuration and namespace handling of PodTemplates in Flyte. According to this discussion, PodTemplates need to be present in the namespace where the execution is triggered. Additionally, another discussion mentions that PodTemplates are namespace-specific and should be configured accordingly. Lastly, this issue highlights that the default-pod-template-name should be correctly set in the FlytePropeller configuration.
Would you like more details on how to configure the namespaces and PodTemplates correctly?
Sources:
This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.
Mention @runllm to ask follow-up questions. See runllm.com for more!
davidmirror-ops
commented
5 months ago @nuhaj what version of Flyte are you using? @hamersaw is this the PR that freed the watch from having to have the default Pod template set? https://github.com/flyteorg/flytepropeller/pull/614
nuhaj
commented
5 months ago flyte-binary-release:v1.12.0
hamersaw
commented
5 months ago @nuhaj what version of Flyte are you using? @hamersaw is this the PR that freed the watch from having to have the default Pod template set? https://github.com/flyteorg/flytepropeller/pull/614
Yes, this looks like it.
nuhaj
commented
4 months ago Any updates here? In a similar issue we can not use clusterResourceTemplate to define namespace for flyte-binary
davidmirror-ops
commented
4 months ago @nuhaj I'll work on reproducing this behavior, but I'm curious, what's the content of your test_template.yaml file?
nuhaj
commented
4 months ago I posted a trimmed version above that I tested just to see if the error would resolve, I expect it to ignore the container since name: noob and its expecting name to be default or primary.
Below is the default pod for all namespaces that I initially planned to set. The intent was a default pod with persistent volume claim. The definition below works when defined in the task decorator as a compile-time pod template but not as a default pod
apiVersion: v1
kind: PodTemplate
metadata:
name: flyte-workflow-base
namespace: flyte
template:
metadata:
name: flyte-workflow-base
spec:
initContainers:
- name: init
image: alpine
volumeMounts:
- name: shared-data
mountPath: /data
containers:
- name: primary
volumeMounts:
- name: shared-data
mountPath: /data
volumes:
- name: shared-data
ephemeral:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5GiThanks for sharing.
My confusion comes from what I see Flyte expects in a template for the ClusterResources controller (see example).
The PodTemplate is a different resource and I don't see how any PodTemplate definition would be able to create the project-domain namespaces.
In that sense, what the clusterResourceTemplates section should contain is a namespace spec, even the PodTemplate can be part of that section I guess, but the sole PodTemplate won't create the namespaces, unless I'm missing something. Does that make sense?
nuhaj
commented
4 months ago Yes they are separate issues, I was attempting to create a default podtemplate with pvc and separately also override the default namespace of {{project}}-{{domain}}. For this open report we can focus on the default podtemplate resource.
davidmirror-ops
commented
4 months ago @nuhaj regarding the PodTemplate behavior I just had success using the flyte-binary v1.12.0 with the following config:
In the Helm values:
inline:
plugins:
k8s:
inject-finalizer: true
default-pod-template-name: "flyte-workflow-base"The PodTemplate:
apiVersion: v1
kind: PodTemplate
metadata:
name: flyte-workflow-base
namespace: flyte
template:
metadata:
name: flyte-workflow-base
spec:
initContainers:
- name: init
image: alpine
volumeMounts:
- name: shared-data
mountPath: /data
containers:
- name: default
image: rwgrim/docker-noop
volumeMounts:
- name: shared-data
mountPath: /data
terminationMessagePath: "/dev/foo"
hostNetwork: false
volumes:
- name: shared-data
ephemeral:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5GiAnd running a simple workflow:
import typing
from flytekit import task, workflow
@task
def say_hello(name: str) -> str:
return f"hello {name}!"
@task
def greeting_length(greeting: str) -> int:
@workflow
def wf(name: str = "union") -> typing.Tuple[str, int]:
greeting = say_hello(name=name)
greeting_len = greeting_length(greeting=greeting)
return greeting, greeting_len
if __name__ == "__main__":
print(f"Running wf() { wf(name='passengers') }")I get the following Pod spec:
Notice here that I used default as the container name, which should instruct flytepropeller to use this spec as a base for all the containers in the Pod, not only the primary (in which case, you'd use primary as the container name)
In regards to the namespace issue which I wasn't able to reproduce: without even using anything in the clusterResourceTemplates section and creating a new project:
flytectl create project --name projectx --id projectx
The cluster resources controller creates the namespaces:
k get ns
NAME STATUS AGE
default Active 63d
flyte Active 62d
flytesnacks-development Active 61d
flytesnacks-production Active 61d
flytesnacks-staging Active 61d
kube-node-lease Active 63d
kube-public Active 63d
kube-system Active 63d
projectx-development Active 6h47m
projectx-production Active 6h47m
projectx-staging Active 6h47mLet me know if you have additional questions.
nuhaj
commented
4 months ago @davidmirror-ops for the namespace we are over riding the default namespace {{ project }}-{{ domain }} in cluster resource template. Instead of projectx-development we would get say flyte-projectx-development pyflyte run still tries to register the workflow to projectx-development. Would a default pod template be used here to over ride the namespace again?
davidmirror-ops
commented
4 months ago Would a default pod template be used here to over ride the namespace again?
Maybe that would work but I don't think it's a very maintainable workaround.
So, is flyte-projectx a project in your environment? Otherwise flyte will still run your workflows on the project-domain namespace.
If this is an existing namespace, you can instruct Flyte to run your executions on a particular namespace:
configuration:
inline:
namespace_mapping:
template: "my_namespace"You can create new projects using flytectl create project --name <PROJECT_NAME> --id <PROJECT_NAME>
davidmirror-ops
commented
3 months ago @nuhaj is this still an issue in your environment?
nuhaj
commented
3 months ago @davidmirror-ops I was away. Trying this now. Did you also have a section inside clusterResourceTemplates.inline?
clusterResourceTemplates = {
"flyte-workflow-base.yaml" = ...@nuhaj No, I wasn't setting anything under that section
nuhaj
commented
3 months ago @davidmirror-ops The init container section of the pod describe output does not appear for me even with a new project and workflow deployment . I do see the pod-template in the config but I don't see the yaml contents of pod-template extrapolated anywhere
100-inline-config.yaml: |
k8s:
default-pod-template-name: pod-template
inject-finalizer: trueHow does the helm chart know where "flyte-workflow-base" is ? defined the way you have it there is not context on path
inline:
plugins:
k8s:
inject-finalizer: true
default-pod-template-name: "flyte-workflow-base"The init container section of the pod describe output does not appear for me even with a new project and workflow deployment
You mean an init container as part of the flyte-binary pod or the execution one?
How does the helm chart know where "flyte-workflow-base" is ?
The logic is not applied by the Helm chart. This field ends up on a configmap that then propeller picks up.
When you define this global Pod template as part of the K8s plugin config, propeller starts a watch looking first for that template in the namespace where the task is being executed, otherwise, it looks up on the flyte namespace (see docs)
nuhaj
commented
3 months ago @davidmirror-ops we managed to get the default pod template working by
metadata:
name: flyte-workflow-base
namespace: flyteThank you for your help, the pod description and template set us in the right direction to debug
davidmirror-ops
commented
3 months ago @nuhaj great, hopefully, we'll improve the PodTemplates docs soon to cover some of the gaps. Any other questions please let us know. Thanks!
Describe the bug
when configuring a basic default pod template the name spaces are not created. Projects are created without error but there are no traces of the namespace for the project. When deploying a workflow the propeller error is
failed to create workflow in propeller namespaces "test-development" not found…Removing the default pod, the namespaces are created
Expected behavior
I expect the namespaces to be created or an error when creation failed. I expect the default pod template to be used as either primary, default containers or ignored.
Additional context to reproduce
Trimming down the default template to the most basic skeleton for testing. test_template.yaml
Screenshots
No response
Are you sure this issue hasn't been raised already?
Have you read the Code of Conduct?