Initiate logging using dictConfig

[ss153g]

Question

logger = logging.getLogger(name="google.cloud.sql.connector")

Is there an example of how to get the Cloud SQL connector using dictConfig so that it integrates well with Cloud Logging? I'm using this configuration here: https://cloud.google.com/python/docs/reference/logging/latest/std-lib-integration#manual-handler-configuration

I'd like to know how to integrate SQL Connector into it.

Code

client = google.cloud.logging.Client()
LOGGING = {
    "version": 1,
    "formatters": {
        "simple": {
            "format": "[%(asctime)s] %(levelname)s %(module)s:%(funcName)s:%(lineno)d: %(name)s: %(message)s\n",
            "datefmt": "%Y-%m-%d %H:%M:%S"
        },
        "json": {
            "()": "config.JSONFormatter",
            # "fmt_keys": {
            #     "filename": "filename"
            # },
        },
    },
    "handlers": {
        "cloud_logging_handler": {
            "class": "google.cloud.logging.handlers.CloudLoggingHandler",
            "formatter": "json",
            "client": client,
        },
        "structured_log_handler": {
            "class": "google.cloud.logging.handlers.StructuredLogHandler",
        },
        # "sql_connector_handler": {
        #     "class": "google.cloud.sql.connector",
        #     "formatter": "json",
        # }
    },
    # "root": {"handlers": [], "level": "WARNING"},
    "root": {
        "handlers": [],
        "level": "DEBUG",
    },
    "loggers": {
        "cloud_logger": {
            "handlers": ["cloud_logging_handler"],
            "level": "DEBUG",
        },
        "structured_logger": {
            "handlers": ["structured_log_handler"],
            "level": "DEBUG",
        },
        # "sql_connector": {
        #     "handlers": ["sql_connector_handler"],
        #     "level": "DEBUG"
        # },
    },
}

logging.config.dictConfig(LOGGING)

Additional Details

No response

[jackwotherspoon]

Hi @ss153g thanks for the question!

Let me configure a working setup using dictConfig and get back to you on this with a template to use.

Thanks!

[jackwotherspoon]

Quick follow-up question for you @ss153g 😄

What kind of environment are you attempting to deploy your application to? Local dev? Cloud Run? Let me know as I want to make sure I test the same env as you.

Thanks!

[ss153g]

Hi @jackwotherspoon , I'm deploying the app in App Engine Standard environment. If you can test Cloud Run as well that be great. I eventually want to migrate to that.

[jackwotherspoon]

@ss153g Providing an update here.

I think the root of your issue is that you are trying to create a handler for the Cloud SQL Connector, which is unnecessary. It does not need it's own handler, can just use the structured logger or cloud logger or both.

client = google.cloud.logging.Client()

LOGGING = {
    "version": 1,
    "formatters": {
        "simple": {
            "format": "[%(asctime)s] %(levelname)s %(module)s:%(funcName)s:%(lineno)d: %(name)s: %(message)s\n",
            "datefmt": "%Y-%m-%d %H:%M:%S",
        },
    },
    "handlers": {
        "cloud_logging_handler": {
            "class": "google.cloud.logging.handlers.CloudLoggingHandler",
            "client": client,
        },
        "structured_log_handler": {
            "class": "google.cloud.logging.handlers.StructuredLogHandler"
        },
    },
    "root": {"handlers": [], "level": "WARNING"},
    "loggers": {
        "cloud_logger": {"handlers": ["cloud_logging_handler"], "level": "INFO"},
        "structured_logger": {
            "handlers": ["structured_log_handler"],
            "level": "INFO",
        },
        "google.cloud.sql.connector": {
            "handlers": ["cloud_logging_handler", "structured_log_handler"],
            "level": "DEBUG",
        },
    },
}
logging.config.dictConfig(LOGGING)

Give something like this a try and let me know if it works in your environment 😄

[ss153g]

Thanks for sending this over.

I updated my Python version to 3.13 and since then I cannot seem to run any logging at all. I'm getting this error message: TypeError: Couldn't build proto file into descriptor pool: duplicate symbol 'google.logging.v2.firstlineno'

Once I'm done figuring out the google.cloud.logging error, I will check it out and get back to you. Thanks again for your help! Really appreciate it.

On Fri, Oct 18, 2024 at 10:31 AM Jack Wotherspoon @.***> wrote:

@ss153g https://github.com/ss153g Providing an update here.

I think the root of your issue is that you are trying to create a handler for the Cloud SQL Connector, which is unnecessary. It does not need it's own handler, can just use the structured logger or cloud logger or both.

client = google.cloud.logging.Client() LOGGING = { "version": 1, "formatters": { "simple": { "format": "[%(asctime)s] %(levelname)s %(module)s:%(funcName)s:%(lineno)d: %(name)s: %(message)s\n", "datefmt": "%Y-%m-%d %H:%M:%S", }, }, "handlers": { "cloud_logging_handler": { "class": "google.cloud.logging.handlers.CloudLoggingHandler", "client": client, }, "structured_log_handler": { "class": "google.cloud.logging.handlers.StructuredLogHandler" }, }, "root": {"handlers": [], "level": "WARNING"}, "loggers": { "cloud_logger": {"handlers": ["cloud_logging_handler"], "level": "INFO"}, "structured_logger": { "handlers": ["structured_log_handler"], "level": "INFO", }, "google.cloud.sql.connector": { "handlers": ["cloud_logging_handler"], "level": "DEBUG", }, }, }logging.config.dictConfig(LOGGING)

Give something like this a try and let me know if it works in your environment 😄

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/cloud-sql-python-connector/issues/1174#issuecomment-2422738375, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOURLG6ZJW3NB2AFGTRDX3Z4ESVXAVCNFSM6AAAAABPWVSZ5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRSG4ZTQMZXGU . You are receiving this because you were mentioned.Message ID: <GoogleCloudPlatform/cloud-sql-python-connector/issues/1174/2422738375@ github.com>

[ss153g]

So I uploaded the code to App Engine with 3.12 and sure enough I can see the logs from SQL Connector now. :) :) :) Thanks again for your help!

On Fri, Oct 18, 2024 at 5:39 PM Shahrukh Syed @.***> wrote:

Thanks for sending this over.

I updated my Python version to 3.13 and since then I cannot seem to run any logging at all. I'm getting this error message: TypeError: Couldn't build proto file into descriptor pool: duplicate symbol 'google.logging.v2.firstlineno'

Once I'm done figuring out the google.cloud.logging error, I will check it out and get back to you. Thanks again for your help! Really appreciate it.

On Fri, Oct 18, 2024 at 10:31 AM Jack Wotherspoon < @.***> wrote:

@ss153g https://github.com/ss153g Providing an update here.

I think the root of your issue is that you are trying to create a handler for the Cloud SQL Connector, which is unnecessary. It does not need it's own handler, can just use the structured logger or cloud logger or both.

client = google.cloud.logging.Client() LOGGING = { "version": 1, "formatters": { "simple": { "format": "[%(asctime)s] %(levelname)s %(module)s:%(funcName)s:%(lineno)d: %(name)s: %(message)s\n", "datefmt": "%Y-%m-%d %H:%M:%S", }, }, "handlers": { "cloud_logging_handler": { "class": "google.cloud.logging.handlers.CloudLoggingHandler", "client": client, }, "structured_log_handler": { "class": "google.cloud.logging.handlers.StructuredLogHandler" }, }, "root": {"handlers": [], "level": "WARNING"}, "loggers": { "cloud_logger": {"handlers": ["cloud_logging_handler"], "level": "INFO"}, "structured_logger": { "handlers": ["structured_log_handler"], "level": "INFO", }, "google.cloud.sql.connector": { "handlers": ["cloud_logging_handler"], "level": "DEBUG", }, }, }logging.config.dictConfig(LOGGING)

Give something like this a try and let me know if it works in your environment 😄

— Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/cloud-sql-python-connector/issues/1174#issuecomment-2422738375, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOURLG6ZJW3NB2AFGTRDX3Z4ESVXAVCNFSM6AAAAABPWVSZ5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRSG4ZTQMZXGU . You are receiving this because you were mentioned.Message ID: <GoogleCloudPlatform/cloud-sql-python-connector/issues/1174/2422738375@ github.com>