Add custom labels to pushed logs

[lozbrown]

Hi

Would it be possible to add custom labels to pushed logs, something configured in config?

our loki has a lot of stuff in it and would be difficult to find logs in.

What labels are added to pushed logs as is?

[lozbrown]

As examples

I need to tag logs with X-Scope-OrgID, there are also several others that would be useful statically to identify specific instances etc

Furthermore it would be great if each task log was labelled with dag id task id run id logical_date

[snjypl]

you can do it by creating a custom logger class extending the LokiTaskHandler and overriding the get_labels method. here is an example:

~/airflow/config/lokilogger.py

from grafana_loki_provider.log.loki_task_handler import LokiTaskHandler
from typing import Dict, TYPE_CHECKING

if TYPE_CHECKING:
    from airflow.models.taskinstance import TaskInstance

class CustomLokiHandler(LokiTaskHandler):

    def get_labels(self, ti: "TaskInstance")-> Dict[str, str]:
        labels = super().get_labels(ti)
        labels['run_id'] =str(ti.run_id)
        labels['my_custom_label']='xyz'
        return labels

update the airflow log_config to use the custom class

       elif REMOTE_BASE_LOG_FOLDER.startswith('loki'):
        LOKI_HANDLER: Dict[str, Dict[str, Union[str, bool]]] = {
            'task': {
                'class': 'lokilogger.CustomLokiHandler',
                'formatter': 'airflow',
                'name':"airflow_task",
                'base_log_folder': str(os.path.expanduser(BASE_LOG_FOLDER)),
                'filename_template': FILENAME_TEMPLATE
            },
        }

        DEFAULT_LOGGING_CONFIG['handlers'].update(LOKI_HANDLER)