search

unionai-oss / deploy-flyte

A set of IaC artifacts to automatically configure the infrastructure resources needed by a Flyte deployment
Apache License 2.0
15 stars 15 forks source link

Cloudwatch not getting logs. #5

Open uriafranko opened 9 months ago

uriafranko commented 9 months ago

Hey again :)

The logs are not getting into the CloudWatch for some reason.

Logs section in the helm file

logging:
    level: 4 (tried also 1)
    plugins:
      cloudwatch:
        enabled: true
        templateUri: |-
          https://console.aws.amazon.com/cloudwatch/home?region=eu-central-1#logEventViewer:group=/aws/eks/flyte-staging/cluster;stream=var.log.containers.{{ .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log

CloudWatch can't find the log stream:

CleanShot 2023-09-21 at 17 28 04

Looking at the other log streams it seems that most of the pods get log streams but none of the workflow/tasks get them.

Any idea what is not configured right? (tried adding CloudWatch full access policy to the role as well)

qchenevier commented 6 months ago

@uriafranko did you get any success to export the logs to cloudwatch ? I'm also trying to do this.

kumare3 commented 6 months ago

cc @davidmirror-ops ?

davidmirror-ops commented 6 months ago

@uriafranko @qchenevier

FWIW I started experimenting with this today and, besides setting up an agent (fluentbit, etc) I'm still exploring what else is missing for this to work.

If any of you find something before me, please share it here.

Otherwise, I'll keep working on this and will let you know.