Change max_memory=8196 to max_memory=4096. First there was the bug (8196 -> 8192), which caused this step to run with 2048, 4096 and 8192 MB of RAM. The latter value was too large for normal datasets. In the case of datasets with a lot of noise, this volume would still not be enough. But this amount of memory cost us a lot. Second, I limited the maximum value to 4096 MB, as was the original version by Lachlan.
I looked at the statistics in the perf_profile_entry table for datasets that used 4096 MB of RAM or more. In most cases, the actual RAM consumption was up to 1.5 GB. This is the most expensive step for us, so changing the base value of runtime_memory like this will save 30-40% of money for heavy datasets. You can view the raw data with the following request:
SELECT
id,
profile_id,
start,
finish,
extra_data ->> 'cost' AS costs,
extra_data ->> 'runtime_memory' AS runtime_memory,
extra_data ->> 'max_memory' AS max_memory,
extra_data ->> 'max_time' AS max_time,
extra_data
FROM perf_profile_entry
WHERE
start >= '2024-11-01' AND
finish <= '2024-11-30' AND
name='process_centr_segment' AND
(extra_data ->> 'runtime_memory')::numeric >= 4096
ORDER BY id DESC;
max_memory=8196
tomax_memory=4096
. First there was the bug (8196 -> 8192), which caused this step to run with 2048, 4096 and 8192 MB of RAM. The latter value was too large for normal datasets. In the case of datasets with a lot of noise, this volume would still not be enough. But this amount of memory cost us a lot. Second, I limited the maximum value to 4096 MB, as was the original version by Lachlan.perf_profile_entry
table for datasets that used 4096 MB of RAM or more. In most cases, the actual RAM consumption was up to 1.5 GB. This is the most expensive step for us, so changing the base value ofruntime_memory
like this will save 30-40% of money for heavy datasets. You can view the raw data with the following request: