Google Cloud Logging with FastAPI and custom log formatters

[ntoshev]

Hi!

I'm running a Docker container with FastAPI in Google Cloud Run and would like to get full structured logging and nested logs with correlation_ids. The way to do it is described in https://cloud.google.com/run/docs/logging#writing_structured_logs

I get the correlation id from the request header and enable a custom formatter to make Cloud logging nest the logs. I couldn't get the custom formatter from the documentation to work (https://github.com/bobbui/json-logging-python/tree/dc80372b4981ddb035768a861f596a28f7fe2133#26-custom-log-formatter, I get request_response_data is not a property of record). My code looks like this:

PROJECT = '....'
class CloudRunJSONLog(json_logging.JSONLogWebFormatter):
    'Logger for Google Cloud Run correlating with the requests'
    def _format_log_object(self, record, request_util):
        json_log_object = super(CloudRunJSONLog, self)._format_log_object(record, request_util)

        if 'correlation_id' in json_log_object and len(json_log_object['correlation_id'])>1: # Missing correlation id actually contains '-'
            trace = json_log_object['correlation_id'].split('/')
            json_log_object["logging.googleapis.com/trace"]=f"projects/{PROJECT}/traces/{trace[0]}"

        return json_log_object
# ...
json_logging.CORRELATION_ID_HEADERS = ['X-Cloud-Trace-Context']
json_logging.init_fastapi(custom_formatter=CloudRunJSONLog, enable_json=True)

This works with my own logs, however at least some logs made by FastAPI don't use the custom formatter. For example, if the data I pass to the handler fail validation, the resulting log is not handled via the custom log handler and does not appear nested in the logs explorer.

Is there a proper way to do this?

[bobbui]

could u pls provide fully working python file

[ntoshev]

Sure

import uvicorn, json_logging, logging, os, sys
from fastapi import FastAPI

PROJECT = 'myproject'
class CloudRunJSONLog(json_logging.JSONLogWebFormatter):
    'Logger for Google Cloud Run correlating with the requests'
    def _format_log_object(self, record, request_util):
        json_log_object = super(CloudRunJSONLog, self)._format_log_object(record, request_util)

        if 'correlation_id' in json_log_object and len(json_log_object['correlation_id'])>1:
            trace = json_log_object['correlation_id'].split('/')
            json_log_object["logging.googleapis.com/trace"]=f"projects/{PROJECT}/traces/{trace[0]}"

        return json_log_object

app = FastAPI()
logging.basicConfig(level=logging.INFO)
json_logging.CORRELATION_ID_HEADERS = ['X-Cloud-Trace-Context']
json_logging.init_fastapi(custom_formatter=CloudRunJSONLog, enable_json=True)
json_logging.init_request_instrument(app)
json_logging.config_root_logger()
logging.info('Started in environment', extra=dict(props=dict(env=dict(**os.environ))) )

@app.get("/")
async def root(url: str):
    logging.info('Request!!!', extra=dict(props={"url":url}))
    return True

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

[bobbui]

@ntoshev sorry i has been busy taking care of some personal stuff. Are u still having some problem?

[bobbui]

closed as no response from author