stac-utils / pgstac

Schema, functions and a python library for storing and accessing STAC collections and items in PostgreSQL
MIT License
153 stars 39 forks source link

Excessive growth of database logs #242

Closed drnextgis closed 7 months ago

drnextgis commented 9 months ago

We encountered an issue where the database logs experienced rapid growth, particularly under heavy load. After making adjustments of some PostgreSQL settings, we discovered that the growth could only be halted by setting log_min_messages: fatal. However, this approach, as outlined in the PostgreSQL documentation, disables the logging of warnings. As a database administrator, I prioritize messages that serve as alerts regarding potential issues requiring attention (warnings), rather than regular logs.

I've observed that while NOTICE level is utilized in some areas within pgstac, LOG level is applied elsewhere. I'm curious about the difference between the two and whether it would be sensible to uniformly use NOTICE instead of LOG. Currently, within seconds, our log file has grown from 0 to approximately 10GB, inundated with records such as:

2024-02-22 19:48:21 UTC:10.100.84.65(59944):backend_admin@postgis:[15032]:STATEMENT:  
                        SELECT * FROM search($1::text::jsonb);

2024-02-22 19:48:21 UTC:10.100.84.65(59944):backend_admin@postgis:[15032]:LOG:  QUERY: 

        SELECT * FROM items

        WHERE

        datetime >= '2024-03-01 00:00:00+00' AND datetime < '2024-04-01 00:00:00+00'

        AND (id = ANY ('{20220916_111003}')  AND collection = ANY ('{quickview}')  AND (st_disjoint(geometry, '0103000020E610000001000000A90000008886494EA0204140A42620F546383F402DD6392C991A41408ED492089A513F4055DB82E7A21A4140407139C04C543F403655B49E963A41403AA581...
bitner commented 7 months ago

Looking into this, I didn't realize that log_min_messages treats "LOG" as higher than "NOTICE" where in client_min_messages "LOG" is at a lower level than "NOTICE". I'll adjust all of these calls to be at "INFO" or "DEBUG1".

I added the use of "LOG" in addition to "NOTICE" in order to have multiple levels of verbosity.