The loki setup our infra team has is a bit different than usual. My logs reach loki correctly and are available for me to query in grafana correctly.
However in airflow UI when I go to a specific task instance and navigate to the logs screen then the request to loki to read times out. Turns out the infra team is routing read and writes to loki as two different services (for scalability, I think) the writes go to https://loki-distributor.abccompany.net
however the reads must come from
https://_loki-query-frontend_.abccompany.net/loki/api/v1/query_range
I am not really sure how to go about solving it, but to me it seems like a second connection/ hook to be able to make a query for the logs to show up back in airflow.
The loki setup our infra team has is a bit different than usual. My logs reach loki correctly and are available for me to query in grafana correctly. However in airflow UI when I go to a specific task instance and navigate to the logs screen then the request to loki to read times out. Turns out the infra team is routing read and writes to loki as two different services (for scalability, I think) the writes go to
https://loki-distributor.abccompany.net however the reads must come from https://_loki-query-frontend_.abccompany.net/loki/api/v1/query_range
I am not really sure how to go about solving it, but to me it seems like a second connection/ hook to be able to make a query for the logs to show up back in airflow.
Any suggestions.