OpenTelemetry implementation is validated and configured in veda-backend

[smohiudd]

What

OpenTelementry was added to the STAC, Raster and Ingest APIs however required more testing and validation. This includes confirming what is currently observed and emitted and what is needed for tracking usage across the catalog.

PI Objective

Objective 5: Tracking Backend Usage With Analytics

Acceptance Criteria

• [x] Document what traces, metric and logs are needed for fine grained observability of the STAC catalog.
• [x] Changes made to veda-backend in dev

[smohiudd]

The current observability in dev is not capturing the STAC endpoints so this needs to be added...working on a PR to including this.

[smohiudd]

Here is a PR that has the monitoring changes: https://github.com/NASA-IMPACT/veda-backend/pull/371

I've included the following metadata in the logs which should give us more flexibility in Grafana by allowing us to see both path parameters and post body data:

            ctx = {
                "path": request.url.path,
                "path_params": request.path_params,
                "body": body_json,
                "route": self.path,
                "method": request.method,
            }

A GET request output log would look something like this:

{
    "level": "INFO",
    "location": "route_handler:44",
    "message": "Received request",
    "timestamp": "2024-05-08 00:04:00,947+0000",
    "service": "stac-api",
    "namespace": "veda-backend-monitor",
    "cold_start": false,
    "function_name": "veda-backend-uah-monitor-stacapilambdaCB4D003E-hsdBDxBmEFhJ",
    "function_memory_size": "8000",
    "function_arn": "arn:aws:lambda:us-west-2:853558080719:function:veda-backend-uah-monitor-stacapilambdaCB4D003E-hsdBDxBmEFhJ",
    "function_request_id": "8742cbf4-e11a-4ee4-b2f0-e94789768c9c",
    "correlation_id": "8742cbf4-e11a-4ee4-b2f0-e94789768c9c",
    "fastapi": {
        "path": "/api/stac/collections/oco2-geos-l3-daily",
        "path_params": {
            "collection_id": "oco2-geos-l3-daily"
        },
        "body": null,
        "route": "/collections/{collection_id}",
        "method": "GET"
    },
    "xray_trace_id": "1-663ac170-06a671fd67cec42b6f50a26e"
}

A POST output log would look something like this:

{
    "level": "INFO",
    "location": "route_handler:43",
    "message": "Received request",
    "timestamp": "2024-05-07 19:22:50,984+0000",
    "service": "stac-api",
    "namespace": "veda-backend-monitor",
    "cold_start": false,
    "function_name": "veda-backend-uah-monitor-stacapilambdaCB4D003E-hsdBDxBmEFhJ",
    "function_memory_size": "8000",
    "function_arn": "arn:aws:lambda:us-west-2:853558080719:function:veda-backend-uah-monitor-stacapilambdaCB4D003E-hsdBDxBmEFhJ",
    "function_request_id": "e67a25a6-1da6-40c8-acd7-a7b1f491b74b",
    "correlation_id": "e67a25a6-1da6-40c8-acd7-a7b1f491b74b",
    "fastapi": {
        "path": "/api/stac/search",
        "path_params": {},
        "body": {
            "filter-lang": "cql2-json",
            "filter": {
                "op": "and",
                "args": [
                    {
                        "op": ">=",
                        "args": [
                            {
                                "property": "datetime"
                            },
                            "2015-01-01T00:00:00.000Z"
                        ]
                    },
                    {
                        "op": "<=",
                        "args": [
                            {
                                "property": "datetime"
                            },
                            "2015-01-01T23:59:59.999Z"
                        ]
                    },
                    {
                        "op": "eq",
                        "args": [
                            {
                                "property": "collection"
                            },
                            "campfire-ndvi-diff"
                        ]
                    }
                ]
            },
            "limit": 500,
            "fields": {
                "include": [
                    "bbox"
                ],
                "exclude": [
                    "collection",
                    "links"
                ]
            }
        },
        "route": "/search",
        "method": "POST"
    },
    "xray_trace_id": "1-663a7f8a-005bdb055a55a7e52c37152f"
}

As an example I created a table in Grafana what shows the top collections that were accessed using the collections/{collection_id} endpoint. These counts are pulled from the logs and aggregated.