search

Panfactum / stack

The Panfactum Stack
https://panfactum.com
Other
16 stars 5 forks source link

[feature]: config.linkerd.io/shutdown-grace-period should be set based on terminationGracePeriodSeconds #164

Closed wesbragagt closed 1 month ago

wesbragagt commented 1 month ago

Prior Search

What new functionality would you like to see?

Our team has noticed in our Argo workflows that the linkerd-proxy side car would fail to exit successfully which I filed this issue https://github.com/Panfactum/stack/issues/148.

After reviewing the recommendations, I applied the strict request/limits to match. Our team continued to report the same issue through out the week where the main workflow would exit 0 however linkerd-proxy would remain running and eventually get killed with exit 1 causing the whole workflow to fail.

After further reviewing this issue in the linkerd github page along with how to pass this annotation to the workflow pods, we were able to successfully fix the issue.

extra_pod_annotations = {
  "config.linkerd.io/shutdown-grace-period": "30s"
}

I suggest adding this annotation to the wf_spec module in order to avoid these issues.

How would you use this new functionality?

I would just use the module defaults.

fullykubed commented 1 month ago

@wesbragagt Linkerd sidecars do automatically shutdown when the other containers in the pod have exited.

However, they won't shutdown if there are open connections as this would caused dropped data. The fact that config.linkerd.io/shutdown-grace-period (which does a forced termination, even if there are open connections) resolves the issue for you leads me to believe that your main containers are not properly closing their connections before shutdown.

We have not noticed this issue in vanilla Argo; however, it does not rule out the possibility that there is a bug there. That said, before I look into that, can you confirm that your containers are properly closing any persistent connections (databases, etc.) during their shutdown process and not just leaving them open.

fullykubed commented 1 month ago

I am also a little confused as the default grace period is 120s (reference). I would not imagine that changing the value from 120s to 30s would have that large of an impact.

fullykubed commented 1 month ago

@wesbragagt Can you share what the terminationGracePeriodSeconds of the pods where you are experiencing this issue is?

It is possible that what is happening is that you are exceeding the terminationGracePeriodSeconds because Linkerd is still running and thus getting a SIGKILL which causes an exit 1. By setting config.linkerd.io/shutdown-grace-period < terminationGracePeriodSeconds, you are avoiding that possibility.

fullykubed commented 1 month ago

To clarify (as I imagine this is a bit confusing), connections are a logical entity and are not closed until the appropriate "closed" packets are sent. This is not done immediately when a container / process exits but tends to happen at some arbitrary later point unless you explicitly provide the close logic in your application code. More information can be found here: https://unix.stackexchange.com/questions/386536/when-how-does-linux-decides-to-close-a-socket-on-application-kill.

As a TCP proxy, Linkerd will think the connection is still open until it sees these "close" packets, even if the main container has shutdown. Even though that might seem silly, its actually an important resiliency safeguard that ensures the packets you send will make it to their destination. Its possible the packets are still stuck in a buffer somewhere before reaching the linkerd proxy, even if the main container has exited. The only way linkerd will know there are no more packets coming is if it sees these "close" packets.

In other words, when your process is done executing, can you confirm that you have some code that goes connection.close() (or some equivalent) for every connection that you have open? If not, Linkerd is doing the correct thing by staying alive and waiting for additional packets to eventually/potentially arrive.

fullykubed commented 1 month ago

Even more good information here: https://stackoverflow.com/questions/4438146/where-does-linux-kernel-do-process-and-tcp-connections-cleanup-after-process-die

wesbragagt commented 1 month ago

@fullykubed Attaching one of the pod's manifest that did not exit successfully 

Name:                 reconciliation-cdmmapping-gpwsz-entry-3274882298
Namespace:            implentio
Priority:             0
Priority Class Name:  default
Service Account:      reconciliation-cdmmapping-23a87cb67861bab5
Node:                 ip-10-0-202-45.us-west-2.compute.internal/10.0.202.45
Start Time:           Tue, 15 Oct 2024 10:08:45 -0700
Labels:               id=reconciliation-cdmmapping-4313c52255bf7ae9
                      linkerd.io/control-plane-ns=linkerd
                      linkerd.io/workload-ns=implentio
                      panfactum.com/environment=production
                      panfactum.com/local=false
                      panfactum.com/module=wf_spec
                      panfactum.com/prevent-lifetime-eviction=true
                      panfactum.com/region=us-west-2
                      panfactum.com/root-module=implentio_workflows
                      panfactum.com/scheduler=true
                      panfactum.com/stack-commit=c61f7564067d148447fb8cfb1c8d8e2b5a91de4d
                      panfactum.com/stack-version=edge.24-09-04
                      panfactum.com/workload=reconciliation-cdmmapping
                      workflows.argoproj.io/completed=false
                      workflows.argoproj.io/workflow=reconciliation-cdmmapping-gpwsz
Annotations:          karpenter.sh/do-not-disrupt: true
                      kubectl.kubernetes.io/default-container: main
                      linkerd.io/created-by: linkerd/proxy-injector edge-24.5.1
                      linkerd.io/inject: enabled
                      linkerd.io/proxy-version: edge-24.5.1
                      linkerd.io/trust-root-sha256: 7ffb0b03cd0909363d64ff5fd8f5b19f37994ffdfad7e6552b73a113bf97711b
                      workflows.argoproj.io/node-id: reconciliation-cdmmapping-gpwsz-3274882298
                      workflows.argoproj.io/node-name: reconciliation-cdmmapping-gpwsz(0)
Status:               Succeeded
IP:                   10.0.246.100
IPs:
  IP:           10.0.246.100
Controlled By:  Workflow/reconciliation-cdmmapping-gpwsz
Init Containers:
  linkerd-init:
    Container ID:    containerd://86bdc6a2981b5a0f7b65781aeaa455d668f689e4ef9dc48fc92f56307b906183
    Image:           590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy-init:v2.4.0
    Image ID:        590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy-init@sha256:5bd804267a4e0b585c5e6e1e1cbf5d91887ed73be84e35fe784df2331b6e9c61
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Args:
      --incoming-proxy-port
      4143
      --outgoing-proxy-port
      4140
      --proxy-uid
      2102
      --inbound-ports-to-ignore
      4190,4191,4567,4568
      --outbound-ports-to-ignore
      4567,4568
      --log-format
      json
      --log-level
      warn
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 15 Oct 2024 10:08:46 -0700
      Finished:     Tue, 15 Oct 2024 10:08:46 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  10Mi
    Requests:
      cpu:     10m
      memory:  10Mi
    Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-west-2
      AWS_REGION:                   us-west-2
      AWS_ROLE_ARN:                 arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /run from linkerd-proxy-init-xtables-lock (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzxb2 (ro)
  linkerd-proxy:
    Container ID:    containerd://12d97e558dc8475fc6dc377e9a6bfb791dfb538b1ca64ff8205cc2ffc300d872
    Image:           590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy:edge-24.5.1
    Image ID:        590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy@sha256:6ecc3ede913be8014a3f93c34bf6a2e6fbd1f4009f3d39d134b925d609529402
    Ports:           4143/TCP, 4191/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    State:           Terminated
      Reason:        Error
      Exit Code:     137
      Started:       Tue, 15 Oct 2024 10:08:47 -0700
      Finished:      Tue, 15 Oct 2024 10:11:22 -0700
    Ready:           False
    Restart Count:   0
    Limits:
      memory:  200Mi
    Requests:
      memory:   10Mi
    Liveness:   http-get http://:4191/live delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:4191/ready delay=2s timeout=1s period=10s #success=1 #failure=3
    Startup:    http-get http://:4191/ready delay=0s timeout=1s period=1s #success=1 #failure=120
    Environment:
      _pod_name:                                                 reconciliation-cdmmapping-gpwsz-entry-3274882298 (v1:metadata.name)
      _pod_ns:                                                   implentio (v1:metadata.namespace)
      _pod_nodeName:                                              (v1:spec.nodeName)
      LINKERD2_PROXY_LOG:                                        warn,linkerd=warn,linkerd2_proxy=warn
      LINKERD2_PROXY_LOG_FORMAT:                                 json
      LINKERD2_PROXY_DESTINATION_SVC_ADDR:                       linkerd-dst-headless.linkerd.svc.cluster.local.:8086
      LINKERD2_PROXY_DESTINATION_PROFILE_NETWORKS:               10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16,fd00::/8
      LINKERD2_PROXY_POLICY_SVC_ADDR:                            linkerd-policy.linkerd.svc.cluster.local.:8090
      LINKERD2_PROXY_POLICY_WORKLOAD:                            {"ns":"$(_pod_ns)", "pod":"$(_pod_name)"}

      LINKERD2_PROXY_INBOUND_DEFAULT_POLICY:                     all-unauthenticated
      LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS:                    10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16,fd00::/8
      LINKERD2_PROXY_CONTROL_STREAM_INITIAL_TIMEOUT:             3s
      LINKERD2_PROXY_CONTROL_STREAM_IDLE_TIMEOUT:                5m
      LINKERD2_PROXY_CONTROL_STREAM_LIFETIME:                    1h
      LINKERD2_PROXY_INBOUND_CONNECT_TIMEOUT:                    100ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_TIMEOUT:                   1000ms
      LINKERD2_PROXY_OUTBOUND_DISCOVERY_IDLE_TIMEOUT:            5s
      LINKERD2_PROXY_INBOUND_DISCOVERY_IDLE_TIMEOUT:             90s
      LINKERD2_PROXY_CONTROL_LISTEN_ADDR:                        [::]:4190
      LINKERD2_PROXY_ADMIN_LISTEN_ADDR:                          [::]:4191
      LINKERD2_PROXY_OUTBOUND_LISTEN_ADDR:                       127.0.0.1:4140
      LINKERD2_PROXY_OUTBOUND_LISTEN_ADDRS:                      127.0.0.1:4140,[::1]:4140
      LINKERD2_PROXY_INBOUND_LISTEN_ADDR:                        [::]:4143
      LINKERD2_PROXY_INBOUND_IPS:                                 (v1:status.podIPs)
      LINKERD2_PROXY_INBOUND_PORTS:                              
      LINKERD2_PROXY_DESTINATION_PROFILE_SUFFIXES:               svc.cluster.local.
      LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE:                   10000ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE:                 10000ms
      LINKERD2_PROXY_INBOUND_SERVER_HTTP2_KEEP_ALIVE_INTERVAL:   10s
      LINKERD2_PROXY_INBOUND_SERVER_HTTP2_KEEP_ALIVE_TIMEOUT:    3s
      LINKERD2_PROXY_OUTBOUND_SERVER_HTTP2_KEEP_ALIVE_INTERVAL:  10s
      LINKERD2_PROXY_OUTBOUND_SERVER_HTTP2_KEEP_ALIVE_TIMEOUT:   3s
      LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION:   25,587,3306,4444,5432,6379,9300,11211
      LINKERD2_PROXY_DESTINATION_CONTEXT:                        {"ns":"$(_pod_ns)", "nodeName":"$(_pod_nodeName)", "pod":"$(_pod_name)"}

      _pod_sa:                                                    (v1:spec.serviceAccountName)
      _l5d_ns:                                                   linkerd
      _l5d_trustdomain:                                          cluster.local
      LINKERD2_PROXY_IDENTITY_DIR:                               /var/run/linkerd/identity/end-entity
      LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS:                     -----BEGIN CERTIFICATE-----
                                                                 MIIC2TCCAn+gAwIBAgIUJ/mP5KMLYTB3q4fS7ZODFDhtC/4wCgYIKoZIzj0EAwIw
                                                                 ZTESMBAGA1UEChMJcGFuZmFjdHVtMRQwEgYDVQQLEwtlbmdpbmVlcmluZzE5MDcG
                                                                 A1UEAxMwaHR0cDovL3ZhdWx0LWFjdGl2ZS52YXVsdC5zdmMuY2x1c3Rlci5sb2Nh
                                                                 bDo4MjAwMB4XDTI0MDYyMDIwMzYwMFoXDTM0MDYxODIwMzYzMFowZTESMBAGA1UE
                                                                 ChMJcGFuZmFjdHVtMRQwEgYDVQQLEwtlbmdpbmVlcmluZzE5MDcGA1UEAxMwaHR0
                                                                 cDovL3ZhdWx0LWFjdGl2ZS52YXVsdC5zdmMuY2x1c3Rlci5sb2NhbDo4MjAwMFkw
                                                                 EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE/aK5bxPHsw1RpK2PYtNSXp2E4eSIetEE
                                                                 Ad4hgXiGeWnwW2UF1FsHA3mbO1n8oCXh5JeA28BrB0XqYHCepwA+cqOCAQswggEH
                                                                 MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR2dGag
                                                                 Sinb//5+toDC27lqur4C2jAfBgNVHSMEGDAWgBR2dGagSinb//5+toDC27lqur4C
                                                                 2jBWBggrBgEFBQcBAQRKMEgwRgYIKwYBBQUHMAKGOmh0dHA6Ly92YXVsdC1hY3Rp
                                                                 dmUudmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODIwMC92MS9wa2kvY2EwTAYDVR0f
                                                                 BEUwQzBBoD+gPYY7aHR0cDovL3ZhdWx0LWFjdGl2ZS52YXVsdC5zdmMuY2x1c3Rl
                                                                 ci5sb2NhbDo4MjAwL3YxL3BraS9jcmwwCgYIKoZIzj0EAwIDSAAwRQIgapJxAslc
                                                                 qgsVWb4k1dDzDzaiT5XPTAOqO9iFawc1kCcCIQDETlt99S2A/UaP6H7SSBulPMRX
                                                                 UEXOrR9AWkahiJeUQw==
                                                                 -----END CERTIFICATE-----

      LINKERD2_PROXY_IDENTITY_TOKEN_FILE:                        /var/run/secrets/tokens/linkerd-identity-token
      LINKERD2_PROXY_IDENTITY_SVC_ADDR:                          linkerd-identity-headless.linkerd.svc.cluster.local.:8080
      LINKERD2_PROXY_IDENTITY_LOCAL_NAME:                        $(_pod_sa).$(_pod_ns).serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_IDENTITY_SVC_NAME:                          linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_DESTINATION_SVC_NAME:                       linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_POLICY_SVC_NAME:                            linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      AWS_STS_REGIONAL_ENDPOINTS:                                regional
      AWS_DEFAULT_REGION:                                        us-west-2
      AWS_REGION:                                                us-west-2
      AWS_ROLE_ARN:                                              arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
      AWS_WEB_IDENTITY_TOKEN_FILE:                               /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/linkerd/identity/end-entity from linkerd-identity-end-entity (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzxb2 (ro)
      /var/run/secrets/tokens from linkerd-identity-token (rw)
  init:
    Container ID:  containerd://11af64a98e663c3e8f63e5d8f08a111884c306373088ae83ad367b14e1983d25
    Image:         590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5
    Image ID:      590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec@sha256:32a568bd1ecb2691a61aa4a646d90b08fe5c4606a2d5cbf264565b1ced98f12b
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      init
      --loglevel
      info
      --log-format
      json
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 15 Oct 2024 10:08:50 -0700
      Finished:     Tue, 15 Oct 2024 10:08:50 -0700
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  70Mi
    Requests:
      cpu:     10m
      memory:  50Mi
    Environment:
      ARGO_POD_NAME:                      reconciliation-cdmmapping-gpwsz-entry-3274882298 (v1:metadata.name)
      ARGO_POD_UID:                        (v1:metadata.uid)
      GODEBUG:                            x509ignoreCN=0
      ARGO_WORKFLOW_NAME:                 reconciliation-cdmmapping-gpwsz
      ARGO_WORKFLOW_UID:                  c0fb22f9-9441-4c03-982e-ef5f73df29b2
      ARGO_CONTAINER_NAME:                init
      ARGO_TEMPLATE:                      {"name":"entry","inputs":{"parameters":[{"name":"image_version","default":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","value":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","description":""},{"name":"jobName","default":"cdmMappingJob","value":"cdmMappingJob","enum":["healthCheckJob","SyntheticDataCreationJob","cdmMappingJob","ServiceChargeReconciliationJob"],"description":""},{"name":"brandId","default":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","description":""},{"name":"invoiceId","default":"","value":"","description":""},{"name":"ingestionId","default":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","description":""},{"name":"source","default":"NETSUITE_SALESORDER_VW","value":"NETSUITE_SALESORDER_VW","description":""}]},"outputs":{},"affinity":{"nodeAffinity":{}},"metadata":{},"container":{"name":"","image":"730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","command":["/bin/sh","start_your_engine.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_SERVICE_ACCOUNT","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.serviceAccountName"}}},{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"NODE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}},{"name":"CONTAINER_CPU_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.cpu","divisor":"0"}}},{"name":"CONTAINER_MEMORY_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.memory","divisor":"0"}}},{"name":"CONTAINER_MEMORY_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.memory","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.ephemeral-storage","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.ephemeral-storage","divisor":"0"}}},{"name":"DB_HOST","value":"pg-88cc-pooler-rw.implentio"},{"name":"DB_NAME","value":"app"},{"name":"DB_PORT","value":"5432"},{"name":"JAVA_OPTS","value":"-Xms4g -Xmx4g"},{"name":"SNOWFLAKE_ACCOUNT","value":"izb31483.prod3.us-west-2.aws"},{"name":"SNOWFLAKE_DATABASE","value":"PROD"},{"name":"SNOWFLAKE_SCHEMA","value":"public"},{"name":"SNOWFLAKE_WAREHOUSE","value":"WH_APPLICATION_XSM"},{"name":"SPRING_BATCH_JOB_NAME","value":"cdmMappingJob"},{"name":"SPRING_DATASOURCE_POSTGRES_PASSWORD","value":"${DB_PASSWORD}"},{"name":"SPRING_DATASOURCE_POSTGRES_URL","value":"jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=cdm"},{"name":"SPRING_DATASOURCE_POSTGRES_USERNAME","value":"${DB_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_ACCOUNT","value":"${SNOWFLAKE_ACCOUNT}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_DATABASE","value":"${SNOWFLAKE_DATABASE}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_PASSWORD","value":"${SNOWFLAKE_PASSWORD}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_SCHEMA","value":"${SNOWFLAKE_SCHEMA}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_USERNAME","value":"${SNOWFLAKE_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_WAREHOUSE","value":"${SNOWFLAKE_WAREHOUSE}"},{"name":"brandId","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464"},{"name":"ingestionId","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1"},{"name":"invoiceId"},{"name":"source","value":"NETSUITE_SALESORDER_VW"},{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_PASSWORD","optional":false}}},{"name":"DB_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_USER","optional":false}}},{"name":"SNOWFLAKE_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_PASSWORD","optional":false}}},{"name":"SNOWFLAKE_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_USER","optional":false}}}],"resources":{"limits":{"memory":"5000Mi"},"requests":{"cpu":"1","memory":"5000Mi"}},"volumeMounts":[{"name":"podinfo","mountPath":"/etc/podinfo"}],"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true,"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":false}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"s3.amazonaws.com","bucket":"argo-5f172fe57e0cb410","region":"us-west-2","key":"reconciliation-cdmmapping-gpwsz/reconciliation-cdmmapping-gpwsz-entry-3274882298"}},"tolerations":[{"key":"spot","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"burstable","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"arm64","operator":"Equal","value":"true","effect":"NoSchedule"}],"schedulerName":"panfactum","serviceAccountName":"reconciliation-cdmmapping-23a87cb67861bab5"}
      ARGO_NODE_ID:                       reconciliation-cdmmapping-gpwsz-3274882298
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      2024-10-16T17:08:45Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
      AWS_STS_REGIONAL_ENDPOINTS:         regional
      AWS_DEFAULT_REGION:                 us-west-2
      AWS_REGION:                         us-west-2
      AWS_ROLE_ARN:                       arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
      AWS_WEB_IDENTITY_TOKEN_FILE:        /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzxb2 (ro)
Containers:
  wait:
    Container ID:  containerd://0910a93cea4e3206e6269718decc725fbfe0fa83a8ca32f51362d1d1c1c2ef84
    Image:         590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5
    Image ID:      590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec@sha256:32a568bd1ecb2691a61aa4a646d90b08fe5c4606a2d5cbf264565b1ced98f12b
    Port:          <none>
    Host Port:     <none>
    Command:
      argoexec
      wait
      --loglevel
      info
      --log-format
      json
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 15 Oct 2024 10:08:52 -0700
      Finished:     Tue, 15 Oct 2024 10:10:51 -0700
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  70Mi
    Requests:
      cpu:     10m
      memory:  50Mi
    Environment:
      ARGO_POD_NAME:                      reconciliation-cdmmapping-gpwsz-entry-3274882298 (v1:metadata.name)
      ARGO_POD_UID:                        (v1:metadata.uid)
      GODEBUG:                            x509ignoreCN=0
      ARGO_WORKFLOW_NAME:                 reconciliation-cdmmapping-gpwsz
      ARGO_WORKFLOW_UID:                  c0fb22f9-9441-4c03-982e-ef5f73df29b2
      ARGO_CONTAINER_NAME:                wait
      ARGO_TEMPLATE:                      {"name":"entry","inputs":{"parameters":[{"name":"image_version","default":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","value":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","description":""},{"name":"jobName","default":"cdmMappingJob","value":"cdmMappingJob","enum":["healthCheckJob","SyntheticDataCreationJob","cdmMappingJob","ServiceChargeReconciliationJob"],"description":""},{"name":"brandId","default":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","description":""},{"name":"invoiceId","default":"","value":"","description":""},{"name":"ingestionId","default":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","description":""},{"name":"source","default":"NETSUITE_SALESORDER_VW","value":"NETSUITE_SALESORDER_VW","description":""}]},"outputs":{},"affinity":{"nodeAffinity":{}},"metadata":{},"container":{"name":"","image":"730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","command":["/bin/sh","start_your_engine.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_SERVICE_ACCOUNT","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.serviceAccountName"}}},{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"NODE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}},{"name":"CONTAINER_CPU_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.cpu","divisor":"0"}}},{"name":"CONTAINER_MEMORY_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.memory","divisor":"0"}}},{"name":"CONTAINER_MEMORY_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.memory","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.ephemeral-storage","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.ephemeral-storage","divisor":"0"}}},{"name":"DB_HOST","value":"pg-88cc-pooler-rw.implentio"},{"name":"DB_NAME","value":"app"},{"name":"DB_PORT","value":"5432"},{"name":"JAVA_OPTS","value":"-Xms4g -Xmx4g"},{"name":"SNOWFLAKE_ACCOUNT","value":"izb31483.prod3.us-west-2.aws"},{"name":"SNOWFLAKE_DATABASE","value":"PROD"},{"name":"SNOWFLAKE_SCHEMA","value":"public"},{"name":"SNOWFLAKE_WAREHOUSE","value":"WH_APPLICATION_XSM"},{"name":"SPRING_BATCH_JOB_NAME","value":"cdmMappingJob"},{"name":"SPRING_DATASOURCE_POSTGRES_PASSWORD","value":"${DB_PASSWORD}"},{"name":"SPRING_DATASOURCE_POSTGRES_URL","value":"jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=cdm"},{"name":"SPRING_DATASOURCE_POSTGRES_USERNAME","value":"${DB_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_ACCOUNT","value":"${SNOWFLAKE_ACCOUNT}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_DATABASE","value":"${SNOWFLAKE_DATABASE}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_PASSWORD","value":"${SNOWFLAKE_PASSWORD}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_SCHEMA","value":"${SNOWFLAKE_SCHEMA}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_USERNAME","value":"${SNOWFLAKE_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_WAREHOUSE","value":"${SNOWFLAKE_WAREHOUSE}"},{"name":"brandId","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464"},{"name":"ingestionId","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1"},{"name":"invoiceId"},{"name":"source","value":"NETSUITE_SALESORDER_VW"},{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_PASSWORD","optional":false}}},{"name":"DB_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_USER","optional":false}}},{"name":"SNOWFLAKE_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_PASSWORD","optional":false}}},{"name":"SNOWFLAKE_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_USER","optional":false}}}],"resources":{"limits":{"memory":"5000Mi"},"requests":{"cpu":"1","memory":"5000Mi"}},"volumeMounts":[{"name":"podinfo","mountPath":"/etc/podinfo"}],"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true,"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":false}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"s3.amazonaws.com","bucket":"argo-5f172fe57e0cb410","region":"us-west-2","key":"reconciliation-cdmmapping-gpwsz/reconciliation-cdmmapping-gpwsz-entry-3274882298"}},"tolerations":[{"key":"spot","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"burstable","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"arm64","operator":"Equal","value":"true","effect":"NoSchedule"}],"schedulerName":"panfactum","serviceAccountName":"reconciliation-cdmmapping-23a87cb67861bab5"}
      ARGO_NODE_ID:                       reconciliation-cdmmapping-gpwsz-3274882298
      ARGO_INCLUDE_SCRIPT_OUTPUT:         false
      ARGO_DEADLINE:                      2024-10-16T17:08:45Z
      ARGO_PROGRESS_FILE:                 /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:  1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:   3s
      AWS_STS_REGIONAL_ENDPOINTS:         regional
      AWS_DEFAULT_REGION:                 us-west-2
      AWS_REGION:                         us-west-2
      AWS_ROLE_ARN:                       arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
      AWS_WEB_IDENTITY_TOKEN_FILE:        /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /mainctrfs/etc/podinfo from podinfo (rw)
      /tmp from tmp-dir-argo (rw,path="0")
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzxb2 (ro)
  main:
    Container ID:  containerd://f461377eb4839419cd969a72b052054779c619f55d2603d8ffe4154042c407b2
    Image:         730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf
    Image ID:      730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine@sha256:29a7459bd470b0deffecf87497a6ec3cd60be62200dfa24ab56d863867988202
    Port:          <none>
    Host Port:     <none>
    Command:
      /var/run/argo/argoexec
      emissary
      --loglevel
      info
      --log-format
      json
      --
      /bin/sh
      start_your_engine.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 15 Oct 2024 10:08:52 -0700
      Finished:     Tue, 15 Oct 2024 10:10:50 -0700
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  5000Mi
    Requests:
      cpu:     1
      memory:  5000Mi
    Environment:
      POD_IP:                                  (v1:status.podIP)
      POD_NAME:                               reconciliation-cdmmapping-gpwsz-entry-3274882298 (v1:metadata.name)
      POD_NAMESPACE:                          implentio (v1:metadata.namespace)
      NAMESPACE:                              implentio (v1:metadata.namespace)
      POD_SERVICE_ACCOUNT:                     (v1:spec.serviceAccountName)
      NODE_NAME:                               (v1:spec.nodeName)
      NODE_IP:                                 (v1:status.hostIP)
      CONTAINER_CPU_REQUEST:                  1 (requests.cpu)
      CONTAINER_MEMORY_REQUEST:               5242880000 (requests.memory)
      CONTAINER_MEMORY_LIMIT:                 5242880000 (limits.memory)
      CONTAINER_EPHEMERAL_STORAGE_REQUEST:    0 (requests.ephemeral-storage)
      CONTAINER_EPHEMERAL_STORAGE_LIMIT:      0 (limits.ephemeral-storage)
      DB_HOST:                                pg-88cc-pooler-rw.implentio
      DB_NAME:                                app
      DB_PORT:                                5432
      JAVA_OPTS:                              -Xms4g -Xmx4g
      SNOWFLAKE_ACCOUNT:                      izb31483.prod3.us-west-2.aws
      SNOWFLAKE_DATABASE:                     PROD
      SNOWFLAKE_SCHEMA:                       public
      SNOWFLAKE_WAREHOUSE:                    WH_APPLICATION_XSM
      SPRING_BATCH_JOB_NAME:                  cdmMappingJob
      SPRING_DATASOURCE_POSTGRES_PASSWORD:    ${DB_PASSWORD}
      SPRING_DATASOURCE_POSTGRES_URL:         jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=cdm
      SPRING_DATASOURCE_POSTGRES_USERNAME:    ${DB_USER}
      SPRING_DATASOURCE_SNOWFLAKE_ACCOUNT:    ${SNOWFLAKE_ACCOUNT}
      SPRING_DATASOURCE_SNOWFLAKE_DATABASE:   ${SNOWFLAKE_DATABASE}
      SPRING_DATASOURCE_SNOWFLAKE_PASSWORD:   ${SNOWFLAKE_PASSWORD}
      SPRING_DATASOURCE_SNOWFLAKE_SCHEMA:     ${SNOWFLAKE_SCHEMA}
      SPRING_DATASOURCE_SNOWFLAKE_USERNAME:   ${SNOWFLAKE_USER}
      SPRING_DATASOURCE_SNOWFLAKE_WAREHOUSE:  ${SNOWFLAKE_WAREHOUSE}
      brandId:                                f6dd1d37-d0c6-4295-b0e6-6467437ef464
      ingestionId:                            49bd4c74-bd66-48ba-81e3-ec55ce86fba1
      invoiceId:                              
      source:                                 NETSUITE_SALESORDER_VW
      DB_PASSWORD:                            <set to the key 'DB_PASSWORD' in secret 'reconciliation-cdmmapping-23a87cb67861bab5'>         Optional: false
      DB_USER:                                <set to the key 'DB_USER' in secret 'reconciliation-cdmmapping-23a87cb67861bab5'>             Optional: false
      SNOWFLAKE_PASSWORD:                     <set to the key 'SNOWFLAKE_PASSWORD' in secret 'reconciliation-cdmmapping-23a87cb67861bab5'>  Optional: false
      SNOWFLAKE_USER:                         <set to the key 'SNOWFLAKE_USER' in secret 'reconciliation-cdmmapping-23a87cb67861bab5'>      Optional: false
      ARGO_CONTAINER_NAME:                    main
      ARGO_TEMPLATE:                          {"name":"entry","inputs":{"parameters":[{"name":"image_version","default":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","value":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","description":""},{"name":"jobName","default":"cdmMappingJob","value":"cdmMappingJob","enum":["healthCheckJob","SyntheticDataCreationJob","cdmMappingJob","ServiceChargeReconciliationJob"],"description":""},{"name":"brandId","default":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","description":""},{"name":"invoiceId","default":"","value":"","description":""},{"name":"ingestionId","default":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","description":""},{"name":"source","default":"NETSUITE_SALESORDER_VW","value":"NETSUITE_SALESORDER_VW","description":""}]},"outputs":{},"affinity":{"nodeAffinity":{}},"metadata":{},"container":{"name":"","image":"730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","command":["/bin/sh","start_your_engine.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_SERVICE_ACCOUNT","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.serviceAccountName"}}},{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"NODE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}},{"name":"CONTAINER_CPU_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.cpu","divisor":"0"}}},{"name":"CONTAINER_MEMORY_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.memory","divisor":"0"}}},{"name":"CONTAINER_MEMORY_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.memory","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.ephemeral-storage","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.ephemeral-storage","divisor":"0"}}},{"name":"DB_HOST","value":"pg-88cc-pooler-rw.implentio"},{"name":"DB_NAME","value":"app"},{"name":"DB_PORT","value":"5432"},{"name":"JAVA_OPTS","value":"-Xms4g -Xmx4g"},{"name":"SNOWFLAKE_ACCOUNT","value":"izb31483.prod3.us-west-2.aws"},{"name":"SNOWFLAKE_DATABASE","value":"PROD"},{"name":"SNOWFLAKE_SCHEMA","value":"public"},{"name":"SNOWFLAKE_WAREHOUSE","value":"WH_APPLICATION_XSM"},{"name":"SPRING_BATCH_JOB_NAME","value":"cdmMappingJob"},{"name":"SPRING_DATASOURCE_POSTGRES_PASSWORD","value":"${DB_PASSWORD}"},{"name":"SPRING_DATASOURCE_POSTGRES_URL","value":"jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=cdm"},{"name":"SPRING_DATASOURCE_POSTGRES_USERNAME","value":"${DB_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_ACCOUNT","value":"${SNOWFLAKE_ACCOUNT}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_DATABASE","value":"${SNOWFLAKE_DATABASE}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_PASSWORD","value":"${SNOWFLAKE_PASSWORD}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_SCHEMA","value":"${SNOWFLAKE_SCHEMA}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_USERNAME","value":"${SNOWFLAKE_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_WAREHOUSE","value":"${SNOWFLAKE_WAREHOUSE}"},{"name":"brandId","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464"},{"name":"ingestionId","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1"},{"name":"invoiceId"},{"name":"source","value":"NETSUITE_SALESORDER_VW"},{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_PASSWORD","optional":false}}},{"name":"DB_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_USER","optional":false}}},{"name":"SNOWFLAKE_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_PASSWORD","optional":false}}},{"name":"SNOWFLAKE_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_USER","optional":false}}}],"resources":{"limits":{"memory":"5000Mi"},"requests":{"cpu":"1","memory":"5000Mi"}},"volumeMounts":[{"name":"podinfo","mountPath":"/etc/podinfo"}],"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true,"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":false}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"s3.amazonaws.com","bucket":"argo-5f172fe57e0cb410","region":"us-west-2","key":"reconciliation-cdmmapping-gpwsz/reconciliation-cdmmapping-gpwsz-entry-3274882298"}},"tolerations":[{"key":"spot","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"burstable","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"arm64","operator":"Equal","value":"true","effect":"NoSchedule"}],"schedulerName":"panfactum","serviceAccountName":"reconciliation-cdmmapping-23a87cb67861bab5"}
      ARGO_NODE_ID:                           reconciliation-cdmmapping-gpwsz-3274882298
      ARGO_INCLUDE_SCRIPT_OUTPUT:             false
      ARGO_DEADLINE:                          2024-10-16T17:08:45Z
      ARGO_PROGRESS_FILE:                     /var/run/argo/progress
      ARGO_PROGRESS_PATCH_TICK_DURATION:      1m0s
      ARGO_PROGRESS_FILE_TICK_DURATION:       3s
      AWS_STS_REGIONAL_ENDPOINTS:             regional
      AWS_DEFAULT_REGION:                     us-west-2
      AWS_REGION:                             us-west-2
      AWS_ROLE_ARN:                           arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
      AWS_WEB_IDENTITY_TOKEN_FILE:            /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/podinfo from podinfo (rw)
      /var/run/argo from var-run-argo (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dzxb2 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  var-run-argo:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tmp-dir-argo:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  kube-api-access-dzxb2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  linkerd-proxy-init-xtables-lock:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  linkerd-identity-end-entity:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  linkerd-identity-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 arm64=true:NoSchedule
                             burstable=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
                             spot=true:NoSchedule
Events:
  Type     Reason     Age                From       Message
  ----     ------     ----               ----       -------
  Normal   Scheduled  3m4s               panfactum  Successfully assigned implentio/reconciliation-cdmmapping-gpwsz-entry-3274882298 to ip-10-0-202-45.us-west-2.compute.internal
  Normal   Pulled     3m4s               kubelet    Container image "590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy-init:v2.4.0" already present on machine
  Normal   Created    3m4s               kubelet    Created container linkerd-init
  Normal   Started    3m4s               kubelet    Started container linkerd-init
  Normal   Pulled     3m3s               kubelet    Container image "590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy:edge-24.5.1" already present on machine
  Normal   Created    3m3s               kubelet    Created container linkerd-proxy
  Normal   Started    3m3s               kubelet    Started container linkerd-proxy
  Normal   Pulling    3m1s               kubelet    Pulling image "590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5"
  Normal   Pulled     3m                 kubelet    Successfully pulled image "590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5" in 577ms (577ms including waiting)
  Normal   Created    3m                 kubelet    Created container init
  Normal   Started    3m                 kubelet    Started container init
  Normal   Pulling    2m59s              kubelet    Pulling image "590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5"
  Normal   Pulled     2m58s              kubelet    Successfully pulled image "590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5" in 576ms (576ms including waiting)
  Normal   Created    2m58s              kubelet    Created container wait
  Normal   Started    2m58s              kubelet    Started container wait
  Normal   Pulled     2m58s              kubelet    Container image "730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf" already present on machine
  Normal   Created    2m58s              kubelet    Created container main
  Normal   Started    2m58s              kubelet    Started container main
  Normal   Killing    58s                kubelet    Stopping container linkerd-proxy
  Warning  Unhealthy  34s (x3 over 54s)  kubelet    Readiness probe failed: Get "http://10.0.246.100:4191/ready": dial tcp 10.0.246.100:4191: connect: connection refused
  Warning  Unhealthy  34s (x3 over 54s)  kubelet    Liveness probe failed: Get "http://10.0.246.100:4191/live": dial tcp 10.0.246.100:4191: connect: connection refused
fullykubed commented 1 month ago

@wesbragagt That isn't a manifest. That is the output of describe from k9s which does not show all the fields (such as terminationGracePeriodSeconds).

wesbragagt commented 1 month ago

terminationGracePeriodSeconds

Gotcha, I'm looking for one I can extract perhaps from Argo.

wesbragagt commented 1 month ago

@fullykubed apologies on my confusion, here is the proper pod manifest for a problematic case:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    karpenter.sh/do-not-disrupt: "true"
    kubectl.kubernetes.io/default-container: main
    linkerd.io/created-by: linkerd/proxy-injector edge-24.5.1
    linkerd.io/inject: enabled
    linkerd.io/proxy-version: edge-24.5.1
    linkerd.io/trust-root-sha256: 7ffb0b03cd0909363d64ff5fd8f5b19f37994ffdfad7e6552b73a113bf97711b
    workflows.argoproj.io/node-id: reconciliation-cdmmapping-2qgnw-1533535950
    workflows.argoproj.io/node-name: reconciliation-cdmmapping-2qgnw(0)
  creationTimestamp: "2024-10-15T19:02:01Z"
  labels:
    id: reconciliation-cdmmapping-4313c52255bf7ae9
    linkerd.io/control-plane-ns: linkerd
    linkerd.io/workload-ns: implentio
    panfactum.com/environment: production
    panfactum.com/local: "false"
    panfactum.com/module: wf_spec
    panfactum.com/prevent-lifetime-eviction: "true"
    panfactum.com/region: us-west-2
    panfactum.com/root-module: implentio_workflows
    panfactum.com/scheduler: "true"
    panfactum.com/stack-commit: c61f7564067d148447fb8cfb1c8d8e2b5a91de4d
    panfactum.com/stack-version: edge.24-09-04
    panfactum.com/workload: reconciliation-cdmmapping
    workflows.argoproj.io/completed: "false"
    workflows.argoproj.io/workflow: reconciliation-cdmmapping-2qgnw
  name: reconciliation-cdmmapping-2qgnw-entry-1533535950
  namespace: implentio
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Workflow
    name: reconciliation-cdmmapping-2qgnw
    uid: 222eba95-4848-470f-88a0-df1aba746737
  resourceVersion: "136006893"
  uid: 5f2c0a81-0915-457a-beae-f346633a6d83
spec:
  activeDeadlineSeconds: 86399
  affinity:
    nodeAffinity: {}
  containers:
  - command:
    - argoexec
    - wait
    - --loglevel
    - info
    - --log-format
    - json
    env:
    - name: ARGO_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ARGO_POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: GODEBUG
      value: x509ignoreCN=0
    - name: ARGO_WORKFLOW_NAME
      value: reconciliation-cdmmapping-2qgnw
    - name: ARGO_WORKFLOW_UID
      value: 222eba95-4848-470f-88a0-df1aba746737
    - name: ARGO_CONTAINER_NAME
      value: wait
    - name: ARGO_TEMPLATE
      value: '{"name":"entry","inputs":{"parameters":[{"name":"image_version","default":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","value":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","description":""},{"name":"jobName","default":"cdmMappingJob","value":"cdmMappingJob","enum":["healthCheckJob","SyntheticDataCreationJob","cdmMappingJob","ServiceChargeReconciliationJob"],"description":""},{"name":"brandId","default":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","description":""},{"name":"invoiceId","default":"","value":"","description":""},{"name":"ingestionId","default":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","description":""},{"name":"source","default":"NETSUITE_SALESORDER_VW","value":"NETSUITE_SALESORDER_VW","description":""}]},"outputs":{},"affinity":{"nodeAffinity":{}},"metadata":{},"container":{"name":"","image":"730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","command":["/bin/sh","start_your_engine.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_SERVICE_ACCOUNT","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.serviceAccountName"}}},{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"NODE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}},{"name":"CONTAINER_CPU_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.cpu","divisor":"0"}}},{"name":"CONTAINER_MEMORY_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.memory","divisor":"0"}}},{"name":"CONTAINER_MEMORY_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.memory","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.ephemeral-storage","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.ephemeral-storage","divisor":"0"}}},{"name":"DB_HOST","value":"pg-88cc-pooler-rw.implentio"},{"name":"DB_NAME","value":"app"},{"name":"DB_PORT","value":"5432"},{"name":"JAVA_OPTS","value":"-Xms4g
        -Xmx4g"},{"name":"SNOWFLAKE_ACCOUNT","value":"izb31483.prod3.us-west-2.aws"},{"name":"SNOWFLAKE_DATABASE","value":"PROD"},{"name":"SNOWFLAKE_SCHEMA","value":"public"},{"name":"SNOWFLAKE_WAREHOUSE","value":"WH_APPLICATION_XSM"},{"name":"SPRING_BATCH_JOB_NAME","value":"cdmMappingJob"},{"name":"SPRING_DATASOURCE_POSTGRES_PASSWORD","value":"${DB_PASSWORD}"},{"name":"SPRING_DATASOURCE_POSTGRES_URL","value":"jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=cdm"},{"name":"SPRING_DATASOURCE_POSTGRES_USERNAME","value":"${DB_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_ACCOUNT","value":"${SNOWFLAKE_ACCOUNT}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_DATABASE","value":"${SNOWFLAKE_DATABASE}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_PASSWORD","value":"${SNOWFLAKE_PASSWORD}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_SCHEMA","value":"${SNOWFLAKE_SCHEMA}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_USERNAME","value":"${SNOWFLAKE_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_WAREHOUSE","value":"${SNOWFLAKE_WAREHOUSE}"},{"name":"brandId","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464"},{"name":"ingestionId","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1"},{"name":"invoiceId"},{"name":"source","value":"NETSUITE_SALESORDER_VW"},{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_PASSWORD","optional":false}}},{"name":"DB_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_USER","optional":false}}},{"name":"SNOWFLAKE_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_PASSWORD","optional":false}}},{"name":"SNOWFLAKE_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_USER","optional":false}}}],"resources":{"limits":{"memory":"5000Mi"},"requests":{"cpu":"1","memory":"5000Mi"}},"volumeMounts":[{"name":"podinfo","mountPath":"/etc/podinfo"}],"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true,"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":false}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"s3.amazonaws.com","bucket":"argo-5f172fe57e0cb410","region":"us-west-2","key":"reconciliation-cdmmapping-2qgnw/reconciliation-cdmmapping-2qgnw-entry-1533535950"}},"tolerations":[{"key":"spot","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"burstable","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"arm64","operator":"Equal","value":"true","effect":"NoSchedule"}],"schedulerName":"panfactum","serviceAccountName":"reconciliation-cdmmapping-23a87cb67861bab5"}'
    - name: ARGO_NODE_ID
      value: reconciliation-cdmmapping-2qgnw-1533535950
    - name: ARGO_INCLUDE_SCRIPT_OUTPUT
      value: "false"
    - name: ARGO_DEADLINE
      value: "2024-10-16T19:02:01Z"
    - name: ARGO_PROGRESS_FILE
      value: /var/run/argo/progress
    - name: ARGO_PROGRESS_PATCH_TICK_DURATION
      value: 1m0s
    - name: ARGO_PROGRESS_FILE_TICK_DURATION
      value: 3s
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: us-west-2
    - name: AWS_REGION
      value: us-west-2
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: 590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5
    imagePullPolicy: Always
    name: wait
    resources:
      limits:
        memory: 70Mi
      requests:
        cpu: 10m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /mainctrfs/etc/podinfo
      name: podinfo
    - mountPath: /tmp
      name: tmp-dir-argo
      subPath: "0"
    - mountPath: /var/run/argo
      name: var-run-argo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qd94k
      readOnly: true
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  - command:
    - /var/run/argo/argoexec
    - emissary
    - --loglevel
    - info
    - --log-format
    - json
    - --
    - /bin/sh
    - start_your_engine.sh
    env:
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: POD_SERVICE_ACCOUNT
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: NODE_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    - name: CONTAINER_CPU_REQUEST
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: requests.cpu
    - name: CONTAINER_MEMORY_REQUEST
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: requests.memory
    - name: CONTAINER_MEMORY_LIMIT
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: limits.memory
    - name: CONTAINER_EPHEMERAL_STORAGE_REQUEST
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: requests.ephemeral-storage
    - name: CONTAINER_EPHEMERAL_STORAGE_LIMIT
      valueFrom:
        resourceFieldRef:
          divisor: "0"
          resource: limits.ephemeral-storage
    - name: DB_HOST
      value: pg-88cc-pooler-rw.implentio
    - name: DB_NAME
      value: app
    - name: DB_PORT
      value: "5432"
    - name: JAVA_OPTS
      value: -Xms4g -Xmx4g
    - name: SNOWFLAKE_ACCOUNT
      value: izb31483.prod3.us-west-2.aws
    - name: SNOWFLAKE_DATABASE
      value: PROD
    - name: SNOWFLAKE_SCHEMA
      value: public
    - name: SNOWFLAKE_WAREHOUSE
      value: WH_APPLICATION_XSM
    - name: SPRING_BATCH_JOB_NAME
      value: cdmMappingJob
    - name: SPRING_DATASOURCE_POSTGRES_PASSWORD
      value: ${DB_PASSWORD}
    - name: SPRING_DATASOURCE_POSTGRES_URL
      value: jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=cdm
    - name: SPRING_DATASOURCE_POSTGRES_USERNAME
      value: ${DB_USER}
    - name: SPRING_DATASOURCE_SNOWFLAKE_ACCOUNT
      value: ${SNOWFLAKE_ACCOUNT}
    - name: SPRING_DATASOURCE_SNOWFLAKE_DATABASE
      value: ${SNOWFLAKE_DATABASE}
    - name: SPRING_DATASOURCE_SNOWFLAKE_PASSWORD
      value: ${SNOWFLAKE_PASSWORD}
    - name: SPRING_DATASOURCE_SNOWFLAKE_SCHEMA
      value: ${SNOWFLAKE_SCHEMA}
    - name: SPRING_DATASOURCE_SNOWFLAKE_USERNAME
      value: ${SNOWFLAKE_USER}
    - name: SPRING_DATASOURCE_SNOWFLAKE_WAREHOUSE
      value: ${SNOWFLAKE_WAREHOUSE}
    - name: brandId
      value: f6dd1d37-d0c6-4295-b0e6-6467437ef464
    - name: ingestionId
      value: 49bd4c74-bd66-48ba-81e3-ec55ce86fba1
    - name: invoiceId
    - name: source
      value: NETSUITE_SALESORDER_VW
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          key: DB_PASSWORD
          name: reconciliation-cdmmapping-23a87cb67861bab5
          optional: false
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          key: DB_USER
          name: reconciliation-cdmmapping-23a87cb67861bab5
          optional: false
    - name: SNOWFLAKE_PASSWORD
      valueFrom:
        secretKeyRef:
          key: SNOWFLAKE_PASSWORD
          name: reconciliation-cdmmapping-23a87cb67861bab5
          optional: false
    - name: SNOWFLAKE_USER
      valueFrom:
        secretKeyRef:
          key: SNOWFLAKE_USER
          name: reconciliation-cdmmapping-23a87cb67861bab5
          optional: false
    - name: ARGO_CONTAINER_NAME
      value: main
    - name: ARGO_TEMPLATE
      value: '{"name":"entry","inputs":{"parameters":[{"name":"image_version","default":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","value":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","description":""},{"name":"jobName","default":"cdmMappingJob","value":"cdmMappingJob","enum":["healthCheckJob","SyntheticDataCreationJob","cdmMappingJob","ServiceChargeReconciliationJob"],"description":""},{"name":"brandId","default":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","description":""},{"name":"invoiceId","default":"","value":"","description":""},{"name":"ingestionId","default":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","description":""},{"name":"source","default":"NETSUITE_SALESORDER_VW","value":"NETSUITE_SALESORDER_VW","description":""}]},"outputs":{},"affinity":{"nodeAffinity":{}},"metadata":{},"container":{"name":"","image":"730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","command":["/bin/sh","start_your_engine.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_SERVICE_ACCOUNT","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.serviceAccountName"}}},{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"NODE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}},{"name":"CONTAINER_CPU_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.cpu","divisor":"0"}}},{"name":"CONTAINER_MEMORY_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.memory","divisor":"0"}}},{"name":"CONTAINER_MEMORY_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.memory","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.ephemeral-storage","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.ephemeral-storage","divisor":"0"}}},{"name":"DB_HOST","value":"pg-88cc-pooler-rw.implentio"},{"name":"DB_NAME","value":"app"},{"name":"DB_PORT","value":"5432"},{"name":"JAVA_OPTS","value":"-Xms4g
        -Xmx4g"},{"name":"SNOWFLAKE_ACCOUNT","value":"izb31483.prod3.us-west-2.aws"},{"name":"SNOWFLAKE_DATABASE","value":"PROD"},{"name":"SNOWFLAKE_SCHEMA","value":"public"},{"name":"SNOWFLAKE_WAREHOUSE","value":"WH_APPLICATION_XSM"},{"name":"SPRING_BATCH_JOB_NAME","value":"cdmMappingJob"},{"name":"SPRING_DATASOURCE_POSTGRES_PASSWORD","value":"${DB_PASSWORD}"},{"name":"SPRING_DATASOURCE_POSTGRES_URL","value":"jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=cdm"},{"name":"SPRING_DATASOURCE_POSTGRES_USERNAME","value":"${DB_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_ACCOUNT","value":"${SNOWFLAKE_ACCOUNT}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_DATABASE","value":"${SNOWFLAKE_DATABASE}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_PASSWORD","value":"${SNOWFLAKE_PASSWORD}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_SCHEMA","value":"${SNOWFLAKE_SCHEMA}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_USERNAME","value":"${SNOWFLAKE_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_WAREHOUSE","value":"${SNOWFLAKE_WAREHOUSE}"},{"name":"brandId","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464"},{"name":"ingestionId","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1"},{"name":"invoiceId"},{"name":"source","value":"NETSUITE_SALESORDER_VW"},{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_PASSWORD","optional":false}}},{"name":"DB_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_USER","optional":false}}},{"name":"SNOWFLAKE_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_PASSWORD","optional":false}}},{"name":"SNOWFLAKE_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_USER","optional":false}}}],"resources":{"limits":{"memory":"5000Mi"},"requests":{"cpu":"1","memory":"5000Mi"}},"volumeMounts":[{"name":"podinfo","mountPath":"/etc/podinfo"}],"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true,"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":false}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"s3.amazonaws.com","bucket":"argo-5f172fe57e0cb410","region":"us-west-2","key":"reconciliation-cdmmapping-2qgnw/reconciliation-cdmmapping-2qgnw-entry-1533535950"}},"tolerations":[{"key":"spot","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"burstable","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"arm64","operator":"Equal","value":"true","effect":"NoSchedule"}],"schedulerName":"panfactum","serviceAccountName":"reconciliation-cdmmapping-23a87cb67861bab5"}'
    - name: ARGO_NODE_ID
      value: reconciliation-cdmmapping-2qgnw-1533535950
    - name: ARGO_INCLUDE_SCRIPT_OUTPUT
      value: "false"
    - name: ARGO_DEADLINE
      value: "2024-10-16T19:02:01Z"
    - name: ARGO_PROGRESS_FILE
      value: /var/run/argo/progress
    - name: ARGO_PROGRESS_PATCH_TICK_DURATION
      value: 1m0s
    - name: ARGO_PROGRESS_FILE_TICK_DURATION
      value: 3s
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: us-west-2
    - name: AWS_REGION
      value: us-west-2
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: 730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf
    imagePullPolicy: IfNotPresent
    name: main
    resources:
      limits:
        memory: 5000Mi
      requests:
        cpu: "1"
        memory: 5000Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 1000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/podinfo
      name: podinfo
    - mountPath: /var/run/argo
      name: var-run-argo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qd94k
      readOnly: true
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - args:
    - --incoming-proxy-port
    - "4143"
    - --outgoing-proxy-port
    - "4140"
    - --proxy-uid
    - "2102"
    - --inbound-ports-to-ignore
    - 4190,4191,4567,4568
    - --outbound-ports-to-ignore
    - 4567,4568
    - --log-format
    - json
    - --log-level
    - warn
    env:
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: us-west-2
    - name: AWS_REGION
      value: us-west-2
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: 590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy-init:v2.4.0
    imagePullPolicy: IfNotPresent
    name: linkerd-init
    resources:
      limits:
        cpu: 100m
        memory: 10Mi
      requests:
        cpu: 10m
        memory: 10Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
      privileged: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 65534
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /run
      name: linkerd-proxy-init-xtables-lock
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qd94k
      readOnly: true
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  - env:
    - name: _pod_name
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: _pod_ns
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: _pod_nodeName
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: LINKERD2_PROXY_LOG
      value: warn,linkerd=warn,linkerd2_proxy=warn
    - name: LINKERD2_PROXY_LOG_FORMAT
      value: json
    - name: LINKERD2_PROXY_DESTINATION_SVC_ADDR
      value: linkerd-dst-headless.linkerd.svc.cluster.local.:8086
    - name: LINKERD2_PROXY_DESTINATION_PROFILE_NETWORKS
      value: 10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16,fd00::/8
    - name: LINKERD2_PROXY_POLICY_SVC_ADDR
      value: linkerd-policy.linkerd.svc.cluster.local.:8090
    - name: LINKERD2_PROXY_POLICY_WORKLOAD
      value: |
        {"ns":"$(_pod_ns)", "pod":"$(_pod_name)"}
    - name: LINKERD2_PROXY_INBOUND_DEFAULT_POLICY
      value: all-unauthenticated
    - name: LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS
      value: 10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16,fd00::/8
    - name: LINKERD2_PROXY_CONTROL_STREAM_INITIAL_TIMEOUT
      value: 3s
    - name: LINKERD2_PROXY_CONTROL_STREAM_IDLE_TIMEOUT
      value: 5m
    - name: LINKERD2_PROXY_CONTROL_STREAM_LIFETIME
      value: 1h
    - name: LINKERD2_PROXY_INBOUND_CONNECT_TIMEOUT
      value: 100ms
    - name: LINKERD2_PROXY_OUTBOUND_CONNECT_TIMEOUT
      value: 1000ms
    - name: LINKERD2_PROXY_OUTBOUND_DISCOVERY_IDLE_TIMEOUT
      value: 5s
    - name: LINKERD2_PROXY_INBOUND_DISCOVERY_IDLE_TIMEOUT
      value: 90s
    - name: LINKERD2_PROXY_CONTROL_LISTEN_ADDR
      value: '[::]:4190'
    - name: LINKERD2_PROXY_ADMIN_LISTEN_ADDR
      value: '[::]:4191'
    - name: LINKERD2_PROXY_OUTBOUND_LISTEN_ADDR
      value: 127.0.0.1:4140
    - name: LINKERD2_PROXY_OUTBOUND_LISTEN_ADDRS
      value: 127.0.0.1:4140,[::1]:4140
    - name: LINKERD2_PROXY_INBOUND_LISTEN_ADDR
      value: '[::]:4143'
    - name: LINKERD2_PROXY_INBOUND_IPS
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIPs
    - name: LINKERD2_PROXY_INBOUND_PORTS
    - name: LINKERD2_PROXY_DESTINATION_PROFILE_SUFFIXES
      value: svc.cluster.local.
    - name: LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE
      value: 10000ms
    - name: LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE
      value: 10000ms
    - name: LINKERD2_PROXY_INBOUND_SERVER_HTTP2_KEEP_ALIVE_INTERVAL
      value: 10s
    - name: LINKERD2_PROXY_INBOUND_SERVER_HTTP2_KEEP_ALIVE_TIMEOUT
      value: 3s
    - name: LINKERD2_PROXY_OUTBOUND_SERVER_HTTP2_KEEP_ALIVE_INTERVAL
      value: 10s
    - name: LINKERD2_PROXY_OUTBOUND_SERVER_HTTP2_KEEP_ALIVE_TIMEOUT
      value: 3s
    - name: LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION
      value: 25,587,3306,4444,5432,6379,9300,11211
    - name: LINKERD2_PROXY_DESTINATION_CONTEXT
      value: |
        {"ns":"$(_pod_ns)", "nodeName":"$(_pod_nodeName)", "pod":"$(_pod_name)"}
    - name: _pod_sa
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.serviceAccountName
    - name: _l5d_ns
      value: linkerd
    - name: _l5d_trustdomain
      value: cluster.local
    - name: LINKERD2_PROXY_IDENTITY_DIR
      value: /var/run/linkerd/identity/end-entity
    - name: LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS
      value: |
        -----BEGIN CERTIFICATE-----
        MIIC2TCCAn+gAwIBAgIUJ/mP5KMLYTB3q4fS7ZODFDhtC/4wCgYIKoZIzj0EAwIw
        ZTESMBAGA1UEChMJcGFuZmFjdHVtMRQwEgYDVQQLEwtlbmdpbmVlcmluZzE5MDcG
        A1UEAxMwaHR0cDovL3ZhdWx0LWFjdGl2ZS52YXVsdC5zdmMuY2x1c3Rlci5sb2Nh
        bDo4MjAwMB4XDTI0MDYyMDIwMzYwMFoXDTM0MDYxODIwMzYzMFowZTESMBAGA1UE
        ChMJcGFuZmFjdHVtMRQwEgYDVQQLEwtlbmdpbmVlcmluZzE5MDcGA1UEAxMwaHR0
        cDovL3ZhdWx0LWFjdGl2ZS52YXVsdC5zdmMuY2x1c3Rlci5sb2NhbDo4MjAwMFkw
        EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE/aK5bxPHsw1RpK2PYtNSXp2E4eSIetEE
        Ad4hgXiGeWnwW2UF1FsHA3mbO1n8oCXh5JeA28BrB0XqYHCepwA+cqOCAQswggEH
        MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR2dGag
        Sinb//5+toDC27lqur4C2jAfBgNVHSMEGDAWgBR2dGagSinb//5+toDC27lqur4C
        2jBWBggrBgEFBQcBAQRKMEgwRgYIKwYBBQUHMAKGOmh0dHA6Ly92YXVsdC1hY3Rp
        dmUudmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw6ODIwMC92MS9wa2kvY2EwTAYDVR0f
        BEUwQzBBoD+gPYY7aHR0cDovL3ZhdWx0LWFjdGl2ZS52YXVsdC5zdmMuY2x1c3Rl
        ci5sb2NhbDo4MjAwL3YxL3BraS9jcmwwCgYIKoZIzj0EAwIDSAAwRQIgapJxAslc
        qgsVWb4k1dDzDzaiT5XPTAOqO9iFawc1kCcCIQDETlt99S2A/UaP6H7SSBulPMRX
        UEXOrR9AWkahiJeUQw==
        -----END CERTIFICATE-----
    - name: LINKERD2_PROXY_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/tokens/linkerd-identity-token
    - name: LINKERD2_PROXY_IDENTITY_SVC_ADDR
      value: linkerd-identity-headless.linkerd.svc.cluster.local.:8080
    - name: LINKERD2_PROXY_IDENTITY_LOCAL_NAME
      value: $(_pod_sa).$(_pod_ns).serviceaccount.identity.linkerd.cluster.local
    - name: LINKERD2_PROXY_IDENTITY_SVC_NAME
      value: linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
    - name: LINKERD2_PROXY_DESTINATION_SVC_NAME
      value: linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
    - name: LINKERD2_PROXY_POLICY_SVC_NAME
      value: linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
    - name: LINKERD2_PROXY_SHUTDOWN_GRACE_PERIOD
      value: 30000ms
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: us-west-2
    - name: AWS_REGION
      value: us-west-2
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: 590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy:edge-24.5.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /live
        port: 4191
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: linkerd-proxy
    ports:
    - containerPort: 4143
      name: linkerd-proxy
      protocol: TCP
    - containerPort: 4191
      name: linkerd-admin
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /ready
        port: 4191
        scheme: HTTP
      initialDelaySeconds: 2
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        memory: 200Mi
      requests:
        memory: 10Mi
    restartPolicy: Always
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 2102
      seccompProfile:
        type: RuntimeDefault
    startupProbe:
      failureThreshold: 120
      httpGet:
        path: /ready
        port: 4191
        scheme: HTTP
      periodSeconds: 1
      successThreshold: 1
      timeoutSeconds: 1
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /var/run/linkerd/identity/end-entity
      name: linkerd-identity-end-entity
    - mountPath: /var/run/secrets/tokens
      name: linkerd-identity-token
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qd94k
      readOnly: true
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  - command:
    - argoexec
    - init
    - --loglevel
    - info
    - --log-format
    - json
    env:
    - name: ARGO_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ARGO_POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: GODEBUG
      value: x509ignoreCN=0
    - name: ARGO_WORKFLOW_NAME
      value: reconciliation-cdmmapping-2qgnw
    - name: ARGO_WORKFLOW_UID
      value: 222eba95-4848-470f-88a0-df1aba746737
    - name: ARGO_CONTAINER_NAME
      value: init
    - name: ARGO_TEMPLATE
      value: '{"name":"entry","inputs":{"parameters":[{"name":"image_version","default":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","value":"57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","description":""},{"name":"jobName","default":"cdmMappingJob","value":"cdmMappingJob","enum":["healthCheckJob","SyntheticDataCreationJob","cdmMappingJob","ServiceChargeReconciliationJob"],"description":""},{"name":"brandId","default":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464","description":""},{"name":"invoiceId","default":"","value":"","description":""},{"name":"ingestionId","default":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1","description":""},{"name":"source","default":"NETSUITE_SALESORDER_VW","value":"NETSUITE_SALESORDER_VW","description":""}]},"outputs":{},"affinity":{"nodeAffinity":{}},"metadata":{},"container":{"name":"","image":"730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf","command":["/bin/sh","start_your_engine.sh"],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_SERVICE_ACCOUNT","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.serviceAccountName"}}},{"name":"NODE_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"spec.nodeName"}}},{"name":"NODE_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.hostIP"}}},{"name":"CONTAINER_CPU_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.cpu","divisor":"0"}}},{"name":"CONTAINER_MEMORY_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.memory","divisor":"0"}}},{"name":"CONTAINER_MEMORY_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.memory","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_REQUEST","valueFrom":{"resourceFieldRef":{"resource":"requests.ephemeral-storage","divisor":"0"}}},{"name":"CONTAINER_EPHEMERAL_STORAGE_LIMIT","valueFrom":{"resourceFieldRef":{"resource":"limits.ephemeral-storage","divisor":"0"}}},{"name":"DB_HOST","value":"pg-88cc-pooler-rw.implentio"},{"name":"DB_NAME","value":"app"},{"name":"DB_PORT","value":"5432"},{"name":"JAVA_OPTS","value":"-Xms4g
        -Xmx4g"},{"name":"SNOWFLAKE_ACCOUNT","value":"izb31483.prod3.us-west-2.aws"},{"name":"SNOWFLAKE_DATABASE","value":"PROD"},{"name":"SNOWFLAKE_SCHEMA","value":"public"},{"name":"SNOWFLAKE_WAREHOUSE","value":"WH_APPLICATION_XSM"},{"name":"SPRING_BATCH_JOB_NAME","value":"cdmMappingJob"},{"name":"SPRING_DATASOURCE_POSTGRES_PASSWORD","value":"${DB_PASSWORD}"},{"name":"SPRING_DATASOURCE_POSTGRES_URL","value":"jdbc:postgresql://${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=cdm"},{"name":"SPRING_DATASOURCE_POSTGRES_USERNAME","value":"${DB_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_ACCOUNT","value":"${SNOWFLAKE_ACCOUNT}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_DATABASE","value":"${SNOWFLAKE_DATABASE}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_PASSWORD","value":"${SNOWFLAKE_PASSWORD}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_SCHEMA","value":"${SNOWFLAKE_SCHEMA}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_USERNAME","value":"${SNOWFLAKE_USER}"},{"name":"SPRING_DATASOURCE_SNOWFLAKE_WAREHOUSE","value":"${SNOWFLAKE_WAREHOUSE}"},{"name":"brandId","value":"f6dd1d37-d0c6-4295-b0e6-6467437ef464"},{"name":"ingestionId","value":"49bd4c74-bd66-48ba-81e3-ec55ce86fba1"},{"name":"invoiceId"},{"name":"source","value":"NETSUITE_SALESORDER_VW"},{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_PASSWORD","optional":false}}},{"name":"DB_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"DB_USER","optional":false}}},{"name":"SNOWFLAKE_PASSWORD","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_PASSWORD","optional":false}}},{"name":"SNOWFLAKE_USER","valueFrom":{"secretKeyRef":{"name":"reconciliation-cdmmapping-23a87cb67861bab5","key":"SNOWFLAKE_USER","optional":false}}}],"resources":{"limits":{"memory":"5000Mi"},"requests":{"cpu":"1","memory":"5000Mi"}},"volumeMounts":[{"name":"podinfo","mountPath":"/etc/podinfo"}],"securityContext":{"capabilities":{"drop":["ALL"]},"privileged":false,"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true,"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":false}},"archiveLocation":{"archiveLogs":true,"s3":{"endpoint":"s3.amazonaws.com","bucket":"argo-5f172fe57e0cb410","region":"us-west-2","key":"reconciliation-cdmmapping-2qgnw/reconciliation-cdmmapping-2qgnw-entry-1533535950"}},"tolerations":[{"key":"spot","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"burstable","operator":"Equal","value":"true","effect":"NoSchedule"},{"key":"arm64","operator":"Equal","value":"true","effect":"NoSchedule"}],"schedulerName":"panfactum","serviceAccountName":"reconciliation-cdmmapping-23a87cb67861bab5"}'
    - name: ARGO_NODE_ID
      value: reconciliation-cdmmapping-2qgnw-1533535950
    - name: ARGO_INCLUDE_SCRIPT_OUTPUT
      value: "false"
    - name: ARGO_DEADLINE
      value: "2024-10-16T19:02:01Z"
    - name: ARGO_PROGRESS_FILE
      value: /var/run/argo/progress
    - name: ARGO_PROGRESS_PATCH_TICK_DURATION
      value: 1m0s
    - name: ARGO_PROGRESS_FILE_TICK_DURATION
      value: 3s
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: us-west-2
    - name: AWS_REGION
      value: us-west-2
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::590183845935:role/reconciliation-cdmmapping-23a87cb6786-20240919161241914900000005
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    image: 590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5
    imagePullPolicy: Always
    name: init
    resources:
      limits:
        memory: 70Mi
      requests:
        cpu: 10m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/argo
      name: var-run-argo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-qd94k
      readOnly: true
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true
  nodeName: ip-10-0-202-45.us-west-2.compute.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  priorityClassName: default
  restartPolicy: Never
  schedulerName: panfactum
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: OnRootMismatch
    runAsGroup: 1000
    runAsNonRoot: false
    runAsUser: 1000
  serviceAccount: reconciliation-cdmmapping-23a87cb67861bab5
  serviceAccountName: reconciliation-cdmmapping-23a87cb67861bab5
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: spot
    operator: Equal
    value: "true"
  - effect: NoSchedule
    key: burstable
    operator: Equal
    value: "true"
  - effect: NoSchedule
    key: arm64
    operator: Equal
    value: "true"
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: aws-iam-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: sts.amazonaws.com
          expirationSeconds: 86400
          path: token
  - emptyDir: {}
    name: var-run-argo
  - emptyDir: {}
    name: tmp-dir-argo
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.labels
        path: labels
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podinfo
  - name: kube-api-access-qd94k
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
  - emptyDir: {}
    name: linkerd-proxy-init-xtables-lock
  - emptyDir:
      medium: Memory
    name: linkerd-identity-end-entity
  - name: linkerd-identity-token
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          audience: identity.l5d.io
          expirationSeconds: 86400
          path: linkerd-identity-token
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-15T19:02:02Z"
    status: "True"
    type: PodReadyToStartContainers
  - lastProbeTime: null
    lastTransitionTime: "2024-10-15T19:02:07Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2024-10-15T19:02:08Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2024-10-15T19:02:08Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2024-10-15T19:02:01Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://b58b0f2dd1d1b5ba56499a81e49473302148dadc367d618eb216ab15312bb315
    image: 730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine:57424dc3ec0c2ff904d5916c9faad9be0ff08fcf
    imageID: 730335560480.dkr.ecr.us-west-2.amazonaws.com/reconciliation-engine@sha256:29a7459bd470b0deffecf87497a6ec3cd60be62200dfa24ab56d863867988202
    lastState: {}
    name: main
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-10-15T19:02:08Z"
  - containerID: containerd://4cacd6f97cea90d870e17322a57141e78a51f97074b056dc073c8575bd0be095
    image: 590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5
    imageID: 590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec@sha256:32a568bd1ecb2691a61aa4a646d90b08fe5c4606a2d5cbf264565b1ced98f12b
    lastState: {}
    name: wait
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-10-15T19:02:08Z"
  hostIP: 10.0.202.45
  hostIPs:
  - ip: 10.0.202.45
  initContainerStatuses:
  - containerID: containerd://ef55445a4e7175af70e0ed5d8fbfeb954b5000e50366ef9bd421ad1b905fe3df
    image: 590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy-init:v2.4.0
    imageID: 590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy-init@sha256:5bd804267a4e0b585c5e6e1e1cbf5d91887ed73be84e35fe784df2331b6e9c61
    lastState: {}
    name: linkerd-init
    ready: true
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://ef55445a4e7175af70e0ed5d8fbfeb954b5000e50366ef9bd421ad1b905fe3df
        exitCode: 0
        finishedAt: "2024-10-15T19:02:03Z"
        reason: Completed
        startedAt: "2024-10-15T19:02:02Z"
  - containerID: containerd://732e5c3a673fb0de3c569e78f690014d0ed0a4c0d22894a21bb19959f540d676
    image: 590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy:edge-24.5.1
    imageID: 590183845935.dkr.ecr.us-west-2.amazonaws.com/github/linkerd/proxy@sha256:6ecc3ede913be8014a3f93c34bf6a2e6fbd1f4009f3d39d134b925d609529402
    lastState: {}
    name: linkerd-proxy
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2024-10-15T19:02:04Z"
  - containerID: containerd://a57b1203f3ec32c35a6cc4b1939a8e36303f5143b626a2326526156d4da37196
    image: 590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec:v3.5.5
    imageID: 590183845935.dkr.ecr.us-west-2.amazonaws.com/quay/argoproj/argoexec@sha256:32a568bd1ecb2691a61aa4a646d90b08fe5c4606a2d5cbf264565b1ced98f12b
    lastState: {}
    name: init
    ready: true
    restartCount: 0
    started: false
    state:
      terminated:
        containerID: containerd://a57b1203f3ec32c35a6cc4b1939a8e36303f5143b626a2326526156d4da37196
        exitCode: 0
        finishedAt: "2024-10-15T19:02:07Z"
        reason: Completed
        startedAt: "2024-10-15T19:02:06Z"
  phase: Running
  podIP: 10.0.246.102
  podIPs:
  - ip: 10.0.246.102
  qosClass: Burstable
  startTime: "2024-10-15T19:02:01Z"
fullykubed commented 1 month ago

@wesbragagt So you can see terminationGracePeriodSeconds is 30. Usually, linux will keep idle, unclosed TCP connections around for 60 seconds (the socket's TIME_WAIT), so I imagine that what I described above is what is happening.

wesbragagt commented 1 month ago

@fullykubed what is confusing to me is that the main container does clean up connections and exits 0 according to logs at least:

│ main 2024-10-15T19:04:07.409Z  INFO 25 --- [reconciliation-engine] [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=cdmMappingJob]] completed with the following parameters: [{'ingestionId':'{value=49 │
│ main 2024-10-15T19:04:07.413Z  INFO 25 --- [reconciliation-engine] [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-2 - Shutdown initiated...                                                                     │
│ main 2024-10-15T19:04:07.957Z  INFO 25 --- [reconciliation-engine] [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-2 - Shutdown completed.                                                                       │
│ main 2024-10-15T19:04:07.958Z  INFO 25 --- [reconciliation-engine] [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...                                                                     │
│ main 2024-10-15T19:04:07.962Z  INFO 25 --- [reconciliation-engine] [ionShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.                                                                       │
│ main Job result exit code: 0                                                                                                                                                                                                             │
│ main {"argo":true,"error":null,"level":"info","msg":"sub-process exited","time":"2024-10-15T19:04:09.001Z"}
wesbragagt commented 1 month ago

@fullykubed Your comment makes sense now to me. Moving forward would a sensible 60s termination grace period do a better job at making sure these are properly terminated?

I was looking through the wf_spec module https://github.com/Panfactum/stack/blob/edge.24-09-04/packages/infrastructure/wf_spec/main.tf#L31

fullykubed commented 1 month ago

@wesbragagt Here is what I know for a fact: if all the TCP connections are closed, the linkerd-proxy sidecar will close immediately after the other containers exit. This has been extensively tested on the version of linkerd we use.

While I cannot speak to what your application does, I am nearly certain it is not closing its connections prior to exiting. Can you show me the code snippet where it does this? By the environment variables, it looks like it is managing multiple database connections. An exit code of 0 doesn't mean it has closed any connections.

The correct thing to do would be to close the connections prior to exit. Barring that, you should ensure that config.linkerd.io/shutdown-grace-period (default 120) < terminationGracePeriodSeconds (default 30) to avoid having a SIGKILL sent to the proxy and thus generating the exit code 137 (128 + 9: SIGKILL is signal 9).

wesbragagt commented 1 month ago

@fullykubed this process is using the framework SpringBatch with HikariCP which handles connection closing automatically. We have confirmed that those connections are properly closed. At that point I'd assume it would be up to linkerd to finish closing anything else that is open. The fact that there is a known issue between linkerd and Argo workflows makes me suspect it's a linkerd problem https://github.com/linkerd/linkerd2/issues/8033.

Do you have any recommendations as far as querying for linkerd's connection status?

As far as code snippet I can provide you the main command script that is run:

#!/bin/sh

set -e -o pipefail

echo "Brand ID: $brandId"
echo "Ingestion ID: $ingestionId"
echo "Invoice ID: $invoiceId"
echo "Source: $source"

# get the 'brandId', 'ingestionId', and 'invoiceId' from environment variables
brandId=$brandId
ingestionId=$ingestionId
invoiceId=$invoiceId
source=$source
jobName=$SPRING_BATCH_JOB_NAME

#starter properties starts with brandId
starter_properties="--spring.batch.job.name=$jobName brandId=$brandId,java.util.UUID"
#if ingestionId is set, add that
if [ -n "$ingestionId" ]; then
  starter_properties="$starter_properties ingestionId=$ingestionId,java.util.UUID"
fi
#if invoiceId is set, add that
if [ -n "$invoiceId" ]; then
  starter_properties="$starter_properties invoiceId=$invoiceId,java.util.UUID"
fi
#if source is set, add that
if [ -n "$source" ]; then
  starter_properties="$starter_properties source=$source"
fi

#set default memory size
if [ -z "$JAVA_OPTS" ]; then
  JAVA_OPTS="-Xms1g -Xmx4g"
fi

echo "Job parameters: $starter_properties"

java $JAVA_OPTS --add-opens=java.base/java.nio=ALL-UNNAMED -jar /app.jar $starter_properties

jobResult=$?

echo "Job result exit code: $jobResult"

exit $jobResult
wesbragagt commented 1 month ago

@fullykubed perhaps a way to configure terminationGracePeriodSeconds at the workflow level for every pod in the workflow is the way to go here given it needs to be > linkerd-graceful-shutdown.

fullykubed commented 1 month ago

@wesbragagt I think it is worth me reiterating this one more time: If all the TCP connections are closed, the linkerd-proxy sidecar will close immediately after the other containers exit. This has been extensively tested on the version of linkerd we use. In other words, AFAIK there is no possibility that linkerd-proxy will remain active if the TCP connections were closed and all other containers have exited.

Do you have any recommendations as far as querying for linkerd's connection status?

On my end, I strongly believe that the fact that linkerd-proxy is not immediately exiting is the indication that the connections are not closed. To get any more information you'd need to get PCAPs which is not a trivial process.

However, you are saying the following:

We have confirmed that those connections are properly closed.

How? The above script does not indicate anything. As far as I can see, the libraries you are using don't have any guarantees around closing connections prior to process exit. If you are relying on the library's guarantees rather than doing it yourselves, can you please link the library's documentation that indicates this is automatically handled? 90%+ of libraries that I have seen do not do this automatically.

Also I want to point out that there are multiple open issues on their GitHub regarding connection leaks: example 1, example 2. As a result, even if there was a guarantee from the library's documentation, I wouldn't exclude the possibility that the library version you are using is bugged.

fullykubed commented 1 month ago

perhaps a way to configure terminationGracePeriodSeconds at the workflow level for every pod in the workflow is the way to go here given it needs to be > linkerd-graceful-shutdown.

We can look into it but this isn't a trivial customization to make. As far as I am aware, Argo doesn't have a field that exposes this natively (hence why it isn't an input to our modules). It is definitely possible, but it probably won't be on the immediate next 30 day roadmap.

For now, if you cannot solve the connection leak, I think the solution of setting config.linkerd.io/shutdown-grace-period to some very low value makes total sense and should be very low risk.

wesbragagt commented 1 month ago

@fullykubed Thank you, you've provided a level of detail that will help our team test and assess how we should be configuring HikariCP to handle connection closing. Likely from the issues reported it is not truly an easy out-of-the-box feature for long running transactions and we should have some manual fallback to ensure it is truly closed. Taking some ideas from this issue as well. https://github.com/brettwooldridge/HikariCP/issues/210

fullykubed commented 1 month ago

I am going to keep this open as it would make sense on our end to eventually add some assurances that config.linkerd.io/shutdown-grace-period is equal to terminationGracePeriodSeconds - 5 (or something similar). Not sure why this isn't the default in Linkerd TBH b/c I cannot see a reason for it to have a different default.

fullykubed commented 1 month ago

Closing as this will be fixed in the next release.