OOM when updating Europe

[CoWayger]

Hi, I have running instance with Europe imported. It works fine except it is running out of memory on updates. This is my specs: 4 thread, 64 GB RAM, Europe extract + shapefile + update minutely.

Graphana stats:

Docker logs: INFO: Total execution time: 45384 milliseconds. [2023-08-29 06:47:48] 269 filtering diff [2023-08-29 06:48:01] 538 pid 269 still running [2023-08-29 06:49:01] 550 pid 269 still running /home/renderer/src/regional/trim_osc.py:211: FutureWarning: GzipFile was opened for writing, but this will change in future Python releases. Specify the mode argument for opening it for writing. of = gzip.GzipFile(fileobj=of) [2023-08-29 06:49:48] 269 importing diff tail: /var/log/tiles/osm2pgsql.log: file truncated 2023-08-29 06:49:48 osm2pgsql version 1.6.0 2023-08-29 06:49:48 Database version: 14.5 (Ubuntu 14.5-0ubuntu0.22.04.1) 2023-08-29 06:49:48 PostGIS version: 3.2 2023-08-29 06:49:49 Setting up table 'planet_osm_point' 2023-08-29 06:49:49 Setting up table 'planet_osm_line' 2023-08-29 06:49:49 Setting up table 'planet_osm_polygon' 2023-08-29 06:49:49 Setting up table 'planet_osm_roads' [2023-08-29 06:50:01] 583 pid 269 still running Processing: Node(50k 0.8k/s) Way(0k 0.00k/s) Relation(0 0.0/s)[2023-08-29 06:51:01] 595 pid 269 still running Processing: Node(90k 0.8k/s) Way(0k 0.00k/s) Relation(0 0.0/s)[2023-08-29 06:52:01] 607 pid 269 still running Processing: Node(95k 0.7k/s) Way(16k 0.27k/s) Relation(0 0.0/s)[2023-08-29 06:53:02] 619 pid 269 still running Processing: Node(95k 0.7k/s) Way(43k 0.36k/s) Relation(0 0.0/s)[2023-08-29 06:54:01] 631 pid 269 still running Processing: Node(95k 0.7k/s) Way(63k 0.36k/s) Relation(0 0.0/s)[2023-08-29 06:55:01] 643 pid 269 still running Processing: Node(95k 0.7k/s) Way(87k 0.36k/s) Relation(0 0.0/s)[2023-08-29 06:56:01] 663 pid 269 still running Processing: Node(95k 0.7k/s) Way(89k 0.37k/s) Relation(0 0.0/s)[2023-08-29 06:57:01] 695 pid 269 still running Processing: Node(95k 0.7k/s) Way(89k 0.27k/s) Relation(100 3.0/s)[2023-08-29 06:58:01] 754 pid 269 still running Processing: Node(95k 0.7k/s) Way(89k 0.27k/s) Relation(1090 11.7/s)[2023-08-29 06:59:01] 801 pid 269 still running Processing: Node(95k 0.7k/s) Way(89k 0.27k/s) Relation(5800 37.9/s)[2023-08-29 07:00:01] 813 pid 269 still running Processing: Node(95k 0.7k/s) Way(89k 0.27k/s) Relation(11430 53.9/s)[2023-08-29 07:01:01] 825 pid 269 still running Processing: Node(95k 0.7k/s) Way(89k 0.27k/s) Relation(15550 56.5/s)[2023-08-29 07:02:01] 837 pid 269 still running Processing: Node(95k 0.7k/s) Way(89k 0.27k/s) Relation(20950 62.5/s)[2023-08-29 07:03:01] 849 pid 269 still running 2023-08-29 07:03:31 Reading input files done in 822s (13m 42s). 2023-08-29 07:03:31 Processed 95131 nodes in 132s (2m 12s) - 721/s 2023-08-29 07:03:31 Processed 89510 ways in 325s (5m 25s) - 275/s 2023-08-29 07:03:31 Processed 24498 relations in 365s (6m 5s) - 67/s 2023-08-29 07:03:38 Going over 10678 pending ways (using 4 threads) Left to process: 0....... 2023-08-29 07:03:53 Processing 10678 pending ways took 15s at a rate of 711.87/s 2023-08-29 07:03:53 Going over 46373 pending relations (using 4 threads) Left to process: 45736...[2023-08-29 07:04:01] 899 pid 269 still running Left to process: 40979...[2023-08-29 07:05:01] 911 pid 269 still running Left to process: 36660...[2023-08-29 07:06:01] 923 pid 269 still running Left to process: 33460...[2023-08-29 07:07:01] 935 pid 269 still running Left to process: 30147...[2023-08-29 07:08:01] 947 pid 269 still running Left to process: 23331...[2023-08-29 07:09:01] 959 pid 269 still running Left to process: 19670...[2023-08-29 07:10:01] 971 pid 269 still running Left to process: 14745...[2023-08-29 07:11:01] 983 pid 269 still running

Env: "Env": [ "EXPIRY_DELETEFROM=19", "OSM2PGSQL_EXTRA_ARGS=-C 1024", "THREADS=4", "MAX_INTERVAL_SECONDS=60", "UPDATES=enabled", "EXPIRY_TOUCHFROM=13", "REPLICATION_URL=https://planet.openstreetmap.org/replication/minute/", "EXPIRY_MINZOOM=13", "EXPIRY_MAXZOOM=20", "ALLOW_CORS=enabled", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "DEBIAN_FRONTEND=noninteractive", "AUTOVACUUM=on" ]

My ideas: Maybe server can't keep up and invalidated tiles are somehow batched in memory till its exhausted? Is it good idea to put minutely updates on europe extract?