Recorder purging records continuously

The problem

I noticed that about the time the recorder purge runs 4am, my swap space went up to near 100%. In the course of investigating that today, I noted that HA was deleting that oldest record in my Postgresql database in realtime, rather that what had been the behavior last time I reviewed which was to do a single mass delete at the scheduled time.

I stoped Home Assistant and restarted and the last record in the database is now constant, and no more continuous deleting.

No errors indicated in HA log.

What is version of Home Assistant Core has the issue?

core-2021.5.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Recorder

Link to integration documentation on our website

https://www.home-assistant.io/integrations/recorder/

Example YAML snippet

# database
recorder:
    purge_keep_days: 65
    db_url: !secret postgres_url
    auto_purge: true

postgres_url: postgresql://xxx:yyy@192.168.2.242/homeassistant



### Anything in the logs that might be useful for us?

_No response_

### Additional information

I will report after the auto purge tomorrow morning to see if the behavior returns.

I checked the recorder history again this morning and the issue is occuring again, it appears to again have started at the recorder purge time of 4:13 am local. Prior to this time, after restarting Home Assistant and Postgresql containers, the oldest record in history file for states was constant. Now, this morning, the oldest records in both events and states tables are changing. As I stated above it appears the recorder is again constantly deleting the oldest record in both tables.

I am not a db expert, so I really do not know how to see what activities are occurring in database on a real time basis. That said, the first picture below shows a query on the events table that appears to be re-running over and over with a new timestamp each time.

Second picture below shows swap space changes again going to 100% starting at 4:13 am local.

Queries I am using to see the oldest record changing in both events and states tables shown below.

SELECT
    state_id,
    entity_id,
    last_updated
FROM
    states
WHERE
    last_updated = (
        SELECT
            MIN(last_updated)
        FROM
            states);

SELECT
*
FROM
    events
WHERE
    time_fired = (
        SELECT
            MIN(time_fired)
        FROM
            events);

Here is recorder log at debug level, seems to show the purge function constantly running. These messages continued until I shutdown and restarted HA:

2021-06-10 13:33:08 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 110 event ids to remove
2021-06-10 13:33:08 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 110 state ids to remove
2021-06-10 13:33:08 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 110 states to remove old_state_id
2021-06-10 13:33:08 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 110 states
2021-06-10 13:33:08 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 110 events
2021-06-10 13:33:08 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet
2021-06-10 13:33:08 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-04-06 20:33:08+00:00
2021-06-10 13:33:17 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 104 event ids to remove
2021-06-10 13:33:17 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 104 state ids to remove
2021-06-10 13:33:17 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 104 states to remove old_state_id
2021-06-10 13:33:17 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 104 states
2021-06-10 13:33:17 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 104 events
2021-06-10 13:33:17 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet
2021-06-10 13:33:17 DEBUG (Recorder) [homeassistant.components.recorder] Sending keepalive
2021-06-10 13:33:17 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-04-06 20:33:17+00:00
2021-06-10 13:33:26 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 80 event ids to remove
2021-06-10 13:33:26 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 80 state ids to remove
2021-06-10 13:33:26 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 80 states to remove old_state_id
2021-06-10 13:33:26 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 80 states
2021-06-10 13:33:26 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 80 events
2021-06-10 13:33:26 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet
2021-06-10 13:33:26 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-04-06 20:33:26+00:00

I have upgraded Home Assistant to 2021.6.6 and the problem continues to occur.

It is disappointing to not see any response to this bug report by any of the developers!

Below is config, log and sql info that I am still seeing.

It appears that HA is stuck in some loop trying to remove records.

# configuration.yaml
# database
recorder:
    purge_keep_days: 65
    db_url: !secret postgres_url
    auto_purge: true
    # auto_purge: false

this query appears to be constantly repeating

SELECT events.event_id AS events_event_id 
FROM events 
WHERE events.time_fired < '2021-04-19T15:51:20.303441+00:00'::timestamptz 
 LIMIT 1000

SELECT events.event_id AS events_event_id 
FROM events 
WHERE events.time_fired < '2021-04-19T15:52:53.819384+00:00'::timestamptz 
 LIMIT 1000

this is the home-assistant.log with debug turned on for recorder purge, these messages are constantly repeating since 4:12 am

2021-06-23 08:55:56 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet
2021-06-23 08:55:56 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-04-19 15:55:56+00:00
2021-06-23 08:56:05 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 84 event ids to remove
2021-06-23 08:56:05 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 84 state ids to remove
2021-06-23 08:56:05 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 84 states to remove old_state_id
2021-06-23 08:56:05 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 84 states
2021-06-23 08:56:05 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 84 events
2021-06-23 08:56:05 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet
2021-06-23 08:56:05 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-04-19 15:56:05+00:00
2021-06-23 08:56:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 101 event ids to remove
2021-06-23 08:56:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 101 state ids to remove
2021-06-23 08:56:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 101 states to remove old_state_id
2021-06-23 08:56:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 101 states
2021-06-23 08:56:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 101 events
2021-06-23 08:56:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet
2021-06-23 08:56:13 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-04-19 15:56:13+00:00
2021-06-23 08:56:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 78 event ids to remove
2021-06-23 08:56:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 78 state ids to remove
2021-06-23 08:56:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 78 states to remove old_state_id
2021-06-23 08:56:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 78 states
2021-06-23 08:56:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 78 events
2021-06-23 08:56:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet
2021-06-23 08:56:22 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-04-19 15:56:22+00:00
2021-06-23 08:56:30 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 93 event ids to remove
2021-06-23 08:56:30 DEBUG (Recorder) [homeassistant.components.recorder.purge] Selected 93 state ids to remove
2021-06-23 08:56:30 DEBUG (Recorder) [homeassistant.components.recorder.purge] Updated 93 states to remove old_state_id
2021-06-23 08:56:30 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 93 states
2021-06-23 08:56:30 DEBUG (Recorder) [homeassistant.components.recorder.purge] Deleted 93 events
2021-06-23 08:56:30 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging hasn't fully completed yet
2021-06-23 08:56:30 DEBUG (Recorder) [homeassistant.components.recorder.purge] Purging states and events before target 2021-04-19 15:56:30+00:00

It is disappointing to not see any response to this bug report by any of the developers!

There are 1.2K issues open. Some patience is appreciated from you as well.

Thank you for the reply @frenck . I understand the volume of bugs, and appreciate the developers work. I posted the bug 14 days ago, would be good to understand what an acceptable time frame is.

Similar issues seems to have occured in past as is shown with similar error messages in this post showing them with version 2021.2.0:

https://community.home-assistant.io/t/tha-old-issue-with-purge-postgres-db/277092/24

According to the closed bug report shown below, this is expected behavior.

However, purge behavior seems to have two distinct paths. At point prior to my noting the constant database operation, I believe I could see the purge function finishing and the oldest record was 65 days plus about 5 minutes after the 4:12 am start of the nightly purge.

Now, the purge function seems to fail to finish and then resets the 'oldest' time to do the purge. And as a result seems to get stuck in this constant delete loops. This constant purging seems to cause a memory leak and as a result the system swap space seems to grow, if left in this behavior, the system will show significant degradation at some point. Swap picture again shown below, growth appears to start when purge starts.

I do not see that I have changed anything in my configuration to increase the number of records in the history, such to cause a larger purge requirement.

Service recorder.purge restarts until all rows are purged #39777
https://github.com/home-assistant/core/issues/39777

I posted the bug 14 days ago, would be good to understand what an acceptable time frame is.

Any. This is an open-source project and relies on contributions from others (mainly volunteers). Feel free to jump in and help out.

As I said, I do appreciate the wonderful work being done by the contributors to this project.

'Any.' As you state, there is quite a backlog of bug reports. The point I will raise, is that the leaders of this project have set a defined timeline for new releases. Perhaps it would be a good idea to dedicate one or more of these releases to doing nothing but reducing this backlog.

is that the leaders of this project have set a defined timeline for new releases

I cut a release once a month, it is not related to this or any other issue. That is a fixed schedule. There is no backlog or planning. This is not a commercial company, it is an open-source project.

As said, this project relies on contributions, from volunteers.

Well, again, your work is very much appreciated. Hope you take much satisfaction in seeing the number of households that are running HA on the analytics page.

@deepcoder , if you want to test the PR then the easiest way is to use the custom_components functionality. Basically:

1) Download the PR code via the green button here. 2) Copy "homeassistant/components/recorder" from the .zip file to "custom_components/recorder" in your HA home directory. 3) Add the line "version": "1.2.3", to "custom_components/recorder/manifest.json" above the "domain" property. 4) Turn on logging by adding custom_components.recorder.purge: debug to the recorder section in "configuration.yaml". 5) Reboot HA and you should be using the modified purge. You can test by calling the service from the Developer Tools panel.

To revert to stock HA then you just remove the "custom_components/recorder" directory. Most importantly, before doing any of the above, make a copy of your HA system including recorder database files and test using the copy!

Thank you @PeteBa for the helpful steps, I will give it a try during the day .

Hello @PeteBa , followed your instructions and I believe I was successful in a first test of your modifications to the recorder purge function. I will let it run automatic mode for the next couple nights and report. See summary below.

Question: I am still getting the 'Purging hasn't fully completed yet' message, should this be treated as normal, or is this still a problem?

Thank you for your work!

# oldest record in states table prior to purge

189396663,sensor.medium_bedroom_atc_mi_temperature,2021-04-20 13:34:54.086568-07

# started 2021-04-21 20:42

service: logger.set_level
data: {"custom_components.recorder.purge":"debug"}

service: recorder.purge
data: {"keep_days": 64}

# log seems to show purge completes for requested period and then stops

# first entry
2021-06-24 13:42:20 DEBUG (Recorder) [custom_components.recorder.purge] Purging states and events before target 2021-04-21 20:42:20+00:00
2021-06-24 13:42:20 DEBUG (Recorder) [custom_components.recorder.purge] Selected 1000 event ids to remove
2021-06-24 13:42:20 DEBUG (Recorder) [custom_components.recorder.purge] Selected 997 state ids to remove
2021-06-24 13:42:20 DEBUG (Recorder) [custom_components.recorder.purge] Updated 997 states to remove old_state_id
2021-06-24 13:42:20 DEBUG (Recorder) [custom_components.recorder.purge] Deleted 997 states
2021-06-24 13:42:20 DEBUG (Recorder) [custom_components.recorder.purge] Deleted 1000 events
2021-06-24 13:42:20 DEBUG (Recorder) [custom_components.recorder.purge] Purging hasn't fully completed yet

# last entry
2021-06-24 13:44:19 DEBUG (Recorder) [custom_components.recorder.purge] Purging states and events before target 2021-04-21 20:42:20+00:00
2021-06-24 13:44:19 DEBUG (Recorder) [custom_components.recorder.purge] Selected 0 event ids to remove
2021-06-24 13:44:19 DEBUG (Recorder) [custom_components.recorder.purge] Deleted 0 recorder_runs

# oldest record in states table is remaining stable at:
190254939,sensor.purple_aqi_sp03,2021-04-21 13:42:20.804547-07

# NO further log entries showing for recorder purge, no other errors indicated. Swap space seems stable.

Great stuff. There is no "purge complete" message. Basically, the batches continue until there are no more records to delete. So you can tell that it finished when it reports "0 records to remove".

@PeteBa very good work homie, thanks for contriburing in this issue. I think we are on the right way, because purge has stopped working well for a very long time, and the problem is clearly present.

After observing 3 cycles of the recorder purge operation, it looks like your fix has solved the continuous record deletion problem. Yeah! Thanks!

I am still seeing what appears to be a memory leak as a result of the purge function running. Below are graphs of the last two runs of the purge service. There appears to be a very distinct memory allocation that does not get released at the end of the purge operations.

I am pretty far above my paygrade here in python space, however are there any memory monitoring operations that I might be able to insert in the test code to see if I can observe where the memory is being allocated?

The vast majority of the code in purge section seems to be direct calls into sqlalchemy code. A quick search for 'sqlalchemy memory leak' does return a considerable discussion.

The operation on my postgresql recorder install takes about 2 minutes to complete on about 2 million total records between states, events and recorder_runs.

Is it possible to dynamically change the MAX_ROWS_TO_PURGE constant? Interested to understand why 1000 records was picked? What would be effect of changing this constant to a larger value?

Again thank you for your work @PeteBa !

My two-bits worth ... swap and memory usage are not fully symmetrical. i.e. allocating memory may cause the system to swap lower priority memory to disk but freeing memory may not result in immediate re-loading of swap. It may only be re-loaded when needed.

With the patch from the PR, are you seeing both physical memory and swap max'ing out and is this causing performance issues from thrashing ? From the graph above it doesn't look like it has reached that point yet.

Thanks for info. The swap increase as a result of purge predates the problem of the continuous purge. The continuous purge just accelerated the swap space going to 100%. Yes, I have yet to fully understand why I see swap increase and not real memory. That said, you are correct the swap usage is not to a critical point yet. But, based on my experience and what you can see by the distinct increase in swap with each execution of the purge service, swap will hit 100%. The system will operate there for some number of additional cycles, however at a point both HA and Linux in general will start to exhibit some odd behaviors. If you try and 'flush' the swap memory (swapoff swapon), that seems to just force the orphan memory into real memory, however still orphan.

I think you are running dockers for both HA and postgres, right ? Just out of interest, can you identify which process is using the swap ? The command below should list by processes. You may also want to try on the postgres docker and the host machine.

echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r

Yes correct, running HA 2021.6.6 and PostgreSQL 12.3 in docker containers on Ubuntu 20.04 LTS host. Here is info you are looking for I hope. Also, at bottom is output from 'smem' command, I am interesting is learning about the 'systemd-coredump postgres:' lines, I am not sure what they indicated. I am going to shutdown and restart both HA and postgresql containers prior to the purge run upcoming.

echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r

# inside postgresql container
root@macmini2012:/# echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r
NAME PID SWAP
postgres 41 1876 kB
postgres 32 1520 kB
postgres 30 1508 kB
postgres 29 1380 kB
postgres 31 1316 kB
postgres 1 1312 kB
postgres 28 1308 kB
postgres 27 1228 kB
postgres 13757 1092 kB
postgres 22483 940 kB
postgres 22207 924 kB
postgres 21892 924 kB
postgres 21772 924 kB
postgres 21737 924 kB

# inside home assistant container
bash-5.0# echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r
NAME PID SWAP
python3 207 30620 kB
s6-svscan 1 24 kB
s6-supervise 204 16 kB
s6-supervise 39 12 kB
bash-5.0# 

# in host
user@macmini2012:~$ echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r
NAME PID SWAP
awk: cannot open /proc/1595547/status (No such file or directory)
awk: cannot open /proc/1595548/status (No such file or directory)
gnome-shell 1498 144380 kB
python3 9642 30620 kB
node 7485 22612 kB
node 7371 21224 kB
npm 8004 17700 kB
jupyter-lab 9273 15712 kB
ibus-extension- 1547 13472 kB
dockerd 1687 13168 kB
evolution-alarm 1835 11664 kB
containerd 931 11136 kB
vino-server 1952 10508 kB
gsd-xsettings 1841 9416 kB
ibus-x11 1549 9380 kB
gsd-media-keys 1824 9108 kB
gsd-power 1826 9052 kB
gsd-wacom 1838 8816 kB
gsd-keyboard 1822 7376 kB
gsd-color 1816 7192 kB
dropbox 1970 6092 kB
tracker-miner-f 1193 5960 kB
node-red 8095 5376 kB
goa-daemon 1588 5264 kB
python 7741 5008 kB
Xorg 1205 4944 kB
python 7829 4608 kB
pulseaudio 1191 4340 kB
python 6536 4096 kB
python 7661 3940 kB
python 7461 3884 kB
snapd 854 3596 kB
evolution-calen 1592 3340 kB
gnome-shell-cal 1571 3192 kB
php-cgi 3457 3104 kB
php-cgi 3464 3076 kB
php-cgi 3467 3072 kB
php-cgi 3466 3068 kB
php-cgi 3465 3068 kB
evolution-sourc 1580 3020 kB
evolution-addre 1627 2980 kB
networkd-dispat 834 2808 kB
unattended-upgr 1176 2692 kB
grafana-server 6842 2440 kB
python3 6733 2436 kB
gnome-session-b 1484 2108 kB
appdaemon 7891 2056 kB
postgres 9897 1876 kB
gsd-printer 1946 1856 kB
gjs 1789 1824 kB
gsd-datetime 1817 1764 kB
fwupd 331052 1548 kB
postgres 6468 1520 kB
postgres 6466 1508 kB
bash 11930 1476 kB
ibus-daemon 1542 1460 kB
postgres 6465 1380 kB
smbd-notifyd 1747 1336 kB
cleanupd 1748 1332 kB
postgres 6467 1316 kB
postgres 6401 1312 kB
postgres 6464 1308 kB
lpqd 1751 1280 kB
postgres 6463 1228 kB
gsd-sound 1834 1200 kB
(sd-pam) 1184 1140 kB
smbd 1741 1120 kB
postgres 1179574 1092 kB
goa-identity-se 1602 1092 kB
smbd 1272127 1080 kB
colord 1935 1052 kB
systemd-resolve 774 1036 kB
geoclue 1611 980 kB
postgres 1595157 940 kB
postgres 1594024 924 kB
postgres 1585384 924 kB
postgres 1582399 924 kB
postgres 1582148 924 kB
python 7557 880 kB
systemd 1183 872 kB
gsd-smartcard 1832 868 kB
gsd-wwan 1839 828 kB
ibus-dconf 1546 820 kB
at-spi-bus-laun 1466 804 kB
gvfsd-fuse 1231 796 kB
at-spi2-registr 1565 780 kB
ibus-engine-sim 1954 776 kB
systemd-udevd 350 772 kB
gsd-a11y-settin 1815 764 kB
gsd-print-notif 1827 756 kB
gvfs-udisks2-vo 1238 752 kB
gvfsd-trash 1801 752 kB
gsd-usb-protect 1837 732 kB
gvfs-gphoto2-vo 1762 724 kB
gvfs-afc-volume 1766 716 kB
gvfsd 1215 628 kB
lighttpd 3443 624 kB
xdg-permission- 1569 620 kB
gsd-screensaver 1830 576 kB
gvfs-mtp-volume 1754 556 kB
gdm-x-session 1203 524 kB
gvfs-goa-volume 1758 488 kB
portainer 2523 480 kB
gsd-sharing 1831 460 kB
gnome-session-c 1477 456 kB
ibus-portal 1551 404 kB
gsd-housekeepin 1820 396 kB
containerd-shim 2471 392 kB
systemd 1 388 kB
gnome-keyring-d 1196 376 kB
NetworkManager 821 364 kB
gsd-disk-utilit 1893 356 kB
gdm3 1003 344 kB
gsd-rfkill 1828 340 kB
polkitd 841 320 kB
containerd-shim 6265 316 kB
dbus-daemon 1201 308 kB
containerd-shim 6500 288 kB
containerd-shim 6379 280 kB
dbus-daemon 1471 268 kB
bash 3433 268 kB
containerd-shim 6677 260 kB
rsyslogd 846 244 kB
bash 3429 244 kB
dconf-service 1617 240 kB
udisksd 860 236 kB
containerd-shim 6820 224 kB
bash 3427 224 kB
hack_hdhomerun. 7427 212 kB
usbhid-ups 6776 184 kB
mosquitto 6287 176 kB
cron 7341 168 kB
cron 815 160 kB
ModemManager 938 156 kB
dbus-daemon 819 152 kB
cron 3446 152 kB
upowerd 1589 124 kB
canonical-livep 916 120 kB
smartd 849 116 kB
nmbd 1695 104 kB
cupsd 929447 100 kB
sh 7340 96 kB
accounts-daemon 808 92 kB
sh 7350 88 kB
gdm-session-wor 1177 84 kB
tmux: server11929 80 kB
cups-browsed 929449 76 kB
avahi-daemon 863 76 kB
pihole-FTL 3442 64 kB
tini 9235 60 kB
containerd-shim 2496 60 kB
tail 7338 44 kB
systemd-logind 858 44 kB
upsd 6778 40 kB
cron 1595379 40 kB
cron 1595377 40 kB
cron 1595376 40 kB
cron 1595375 40 kB
cron 1595374 40 kB
cron 1595373 40 kB
cron 1595372 40 kB
cron 1595371 40 kB
cron 1595370 40 kB
cron 1595369 40 kB
cron 1595368 40 kB
cron 1595366 40 kB
cron 1595365 40 kB
docker-init 7412 36 kB
docker-init 6961 36 kB
sshd 960 32 kB
docker-init 7814 32 kB
docker-init 7725 32 kB
docker-init 6697 32 kB
avahi-daemon 812 32 kB
kerneloops 1704 28 kB
systemd-journal 324 24 kB
s6-svscan 9399 24 kB
docker-init 7645 24 kB
docker-init 6521 24 kB
thermald 859 20 kB
s6-svscan 6975 20 kB
s6-supervise 7001 20 kB
s6-supervise 9639 16 kB
s6-supervise 7000 16 kB
boltd 940 16 kB
systemd-timesyn 776 12 kB
s6-svscan 2522 12 kB
s6-supervise 9444 12 kB
s6-supervise 7336 12 kB
s6-supervise 7334 12 kB
s6-supervise 7333 12 kB
s6-supervise 3425 12 kB
s6-supervise 2645 12 kB
s6-supervise 3424 8 kB
s6-supervise 3422 8 kB
docker-init 7543 8 kB
wpa_supplicant 861 4 kB
switcheroo-cont 857 4 kB

user@macmini2012:~$ sudo smem -s swap -r
  PID User     Command                         Swap      USS      PSS      RSS
 1498 user     /usr/bin/gnome-shell          139188   105176   110817   134380
 9642 root     python3 -m homeassistant --    30572   365068   365073   365084
 7485 user     homebridge                     22604    83748    89538    96404
 7371 root     hb-service                     21212    55012    60793    67640
 8004 user     npm                            17696     2836    13478    24124
 9273 user     /opt/conda/bin/python3.8 /o    15708    31788    31898    32012
 1687 root     /usr/bin/dockerd -H fd:// -    13160    56240    56259    59016
  931 root     /usr/bin/containerd            11076    21304    21313    22680
 1547 user     /usr/libexec/ibus-extension    10424     3312     3723    15520
 1835 user     /usr/libexec/evolution-data     9464     8176     9263    25760
 1952 user     /usr/lib/vino/vino-server       7644     4628     6154    23012
 1824 user     /usr/libexec/gsd-media-keys     6456     4544     5254    18128
 1826 user     /usr/libexec/gsd-power          6276     4088     4583    17152
 1816 user     /usr/libexec/gsd-color          6172     4940     5548    18288
 1970 user     /home/user/.dropbox-dist/dr     6044   341768   342352   355660
 1193 user     /usr/libexec/tracker-miner-     5916     4048     4565    13312
 1549 user     /usr/libexec/ibus-x11 --kil     5584     4336     4760    16448
 1822 user     /usr/libexec/gsd-keyboard       5552     5092     5568    17796
 8095 user     node-red                        5376    63268    73910    84556
 1588 user     /usr/libexec/goa-daemon         5268     2760     3687    16440
 1838 user     /usr/libexec/gsd-wacom          5012     5160     5572    17576
 7741 user     python /home/user/raven_sce     5004     9168     9984    14364
 1205 root     /usr/lib/xorg/Xorg vt2 -dis     4912    21936    22717    28804
 9897 systemd-coredump postgres: user homeassistan     4880      456    20192    96584
 7829 user     python /home/user/purple/pu     4592    11768    12645    17232
 6468 systemd-coredump postgres: logical replicati     4516      152      566     5864
 6466 systemd-coredump postgres: autovacuum launch     4508      304     9228    49644
 6465 systemd-coredump postgres: walwriter             4368       72     1166     8680
 1191 user     /usr/bin/pulseaudio --daemo     4332     1176     1663     9504
 6401 systemd-coredump postgres                        4312     2296     4104    16464
 6464 systemd-coredump postgres: background writer     4296      260    33995   142440
 6463 systemd-coredump postgres: checkpointer          4228      784    34813   144152
 1841 user     /usr/libexec/gsd-xsettings      4168     7432     7973    20024
1179574 systemd-coredump postgres: user homeassistan     4096     1120    32863   140380
 6536 user     python /home/user/pulsepoin     4096    14048    15281    20576
1607577 systemd-coredump postgres: user homeassistan     4092     1312     3967    25568
1607591 systemd-coredump postgres: user homeassistan     3944     1964     5899    33292
1607582 systemd-coredump postgres: user homeassistan     3944     1864     5698    33048
1607571 systemd-coredump postgres: user homeassistan     3944     1956     6057    34468
1607562 systemd-coredump postgres: user homeassistan     3944     2000     6151    34720
 7661 user     python /home/user/winix/win     3924    10272    11455    16700
 7461 user     python /home/user/hdhomerun     3884     7908     8730    13180
  854 root     /usr/lib/snapd/snapd            3596    16568    16575    17784
 1592 user     /usr/libexec/evolution-cale     3232     1368     1890    14228
 1571 user     /usr/libexec/gnome-shell-ca     3192        4      293     8108
 3457 www-data /usr/bin/php-cgi                3104    10188    11596    15624
 3464 www-data /usr/bin/php-cgi                3076       44      490     2732
 3467 www-data /usr/bin/php-cgi                3072       48      494     2736
 3466 www-data /usr/bin/php-cgi                3068       52      498     2740
 3465 www-data /usr/bin/php-cgi                3068       52      498     2740
 1580 user     /usr/libexec/evolution-sour     2964     1408     1896    13848
 1627 user     /usr/libexec/evolution-addr     2932     1112     1922    13856
  834 root     /usr/bin/python3 /usr/bin/n     2804     5512     6833    15536
 1176 root     /usr/bin/python3 /usr/share     2692     7240     8620    18752
 6842 user     grafana-server --homepath=/     2436    27256    27256    27260
 6733 root     python3 /home/user/nut_ups/     2436     9176     9314     9644
 7891 root     /usr/local/bin/python /usr/     2044    32836    32837    32844
 1484 user     /usr/libexec/gnome-session-     1936      840     1086    10424
 1946 user     /usr/libexec/gsd-printer        1852      148      393    10148
 1789 user     /usr/bin/gjs /usr/share/gno     1820     3824     6461    17464
 1817 user     /usr/libexec/gsd-datetime       1764      312      603    10896
331052 root     /usr/libexec/fwupd/fwupd        1496    41132    42419    55024
 1542 user     ibus-daemon --panel disable     1456       48      104     5032
11930 user     -bash                           1444      380      642     3052
 1747 root     /usr/sbin/smbd --foreground     1348      184      677     5272
 1748 root     /usr/sbin/smbd --foreground     1340      188      651     4980
 6467 systemd-coredump postgres: stats collector       1312      220      474     4160
 1751 root     /usr/sbin/smbd --foreground     1300      384     1291     7484
 1834 user     /usr/libexec/gsd-sound          1196       32      182     6420
1272127 root     /usr/sbin/smbd --foreground     1152    12844    15316    24992
 1741 root     /usr/sbin/smbd --foreground     1140      584     3741    16468
 1184 user     (sd-pam)                        1136     1492     2150     4016
 1602 user     /usr/libexec/goa-identity-s     1088      352      484     5856
 1935 colord   /usr/libexec/colord             1040     4628     4995    11452
  774 systemd-resolve /lib/systemd/systemd-resolv     1036     3560     3953    10020
 1611 geoclue  /usr/libexec/geoclue             972     5192     5922    16296
 1183 user     /lib/systemd/systemd --user      872     1724     2362     7204
 1832 user     /usr/libexec/gsd-smartcard       864     1280     1436     6752
 1839 user     /usr/libexec/gsd-wwan            824      252      353     5400
 1546 user     /usr/libexec/ibus-dconf          820        4       55     4772
 7557 user     python /home/user/sensor_mo      812    18440    19697    25020
 1466 user     /usr/libexec/at-spi-bus-lau      804        4       65     4888
 1231 user     /usr/libexec/gvfsd-fuse /ru      796      120      171     4916
 1954 user     /usr/libexec/ibus-engine-si      772       28       78     4748
  350 root     /lib/systemd/systemd-udevd       772     2828     2858     5184
 1815 user     /usr/libexec/gsd-a11y-setti      760        8       51     4488
 1827 user     /usr/libexec/gsd-print-noti      756      872     1100     8428
 1801 user     /usr/libexec/gvfsd-trash --      740      804      888     6368
 1837 user     /usr/libexec/gsd-usb-protec      732      168      218     5164
 1238 user     /usr/libexec/gvfs-udisks2-v      644     1760     1914     7632
 1565 user     /usr/libexec/at-spi2-regist      632      228      300     5716
 3443 www-data lighttpd -D -f /etc/lighttp      624      228     1227     3356
 1762 user     /usr/libexec/gvfs-gphoto2-v      624      448      495     5160
 1766 user     /usr/libexec/gvfs-afc-volum      616      536      754     7008
 1569 user     /usr/libexec/xdg-permission      616       16       54     4212
 1830 user     /usr/libexec/gsd-screensave      576      104      140     4364
 1203 user     /usr/lib/gdm3/gdm-x-session      524      160      197     4640
 1215 user     /usr/libexec/gvfsd               484      552      608     5384
 2523 root     /portainer                       468    24556    24556    24560
 1831 user     /usr/libexec/gsd-sharing         456     1236     1455     7780
 1477 user     /usr/libexec/gnome-session-      452        4       33     3548
 1551 user     /usr/libexec/ibus-portal         396      420      470     5240
 1820 user     /usr/libexec/gsd-housekeepi      392      876      965     6536
 2471 root     /usr/bin/containerd-shim-ru      388     1204     1334     3452
 1754 user     /usr/libexec/gvfs-mtp-volum      380      468      511     4956
 1196 user     /usr/bin/gnome-keyring-daem      376     1456     1688     5860
    1 root     /sbin/init                       376     2380     3609     9180
  821 root     /usr/sbin/NetworkManager --      364     5480     5866    14788
 1893 user     /usr/libexec/gsd-disk-utili      352     1168     1294     6048
 1828 user     /usr/libexec/gsd-rfkill          340      588      628     4872
 1758 user     /usr/libexec/gvfs-goa-volum      332      480      521     4684
 6265 root     /usr/bin/containerd-shim-ru      316     1360     1491     3616
 1201 user     /usr/bin/dbus-daemon --sess      308     1188     1307     4432
  841 root     /usr/lib/policykit-1/polkit      296     3156     3254     8644
 1003 root     /usr/sbin/gdm3                   284     1716     1854     7132
 3433 root     bash ./run                       268       12      218     1424
 6500 root     /usr/bin/containerd-shim-ru      260     1432     1573     3752
 6677 root     /usr/bin/containerd-shim-ru      256     1416     1547     3672
 3429 root     bash ./run                       244       32      255     1528
 1617 user     /usr/libexec/dconf-service       236      704      738     4728
  860 root     /usr/lib/udisks2/udisksd         236     3804     4097    10424
 3427 root     bash ./run                       224       56      293     1584
 1471 user     /usr/bin/dbus-daemon --conf      224      304      417     3592
  846 syslog   /usr/sbin/rsyslogd -n -iNON      224     2300     2324     4864
 6820 root     /usr/bin/containerd-shim-ru      220     1484     1615     3740
 7427 user     /bin/bash /home/user/hdhome      208     1184     1445     2952
 6776 user     /usr/share/nut/usbhid-ups -      180      240      335      620
 6287 user     /usr/sbin/mosquitto -c /mos      176     3240     3241     3248
 7341 root     /usr/sbin/cron -f -l 5           168      268      642     2072
  815 root     /usr/sbin/cron -f                160      152      172     2424
 3446 root     /usr/sbin/cron -f                152      124      356     1704
  819 messagebus /usr/bin/dbus-daemon --syst      152     2112     2259     5592
  938 root     /usr/sbin/ModemManager --fi      136     2756     2865     7832
  916 root     /snap/canonical-livepatch/9      120     9264     9264     9268
  849 root     /usr/sbin/smartd -n              116     1740     1827     5592
 6379 root     /usr/bin/containerd-shim-ru      108     2696     3490     6912
 1695 root     /usr/sbin/nmbd --foreground      104     2316     3708    12384
929447 root     /usr/sbin/cupsd -l               100     3056     3409     9848
 1589 root     /usr/lib/upower/upowerd          100     1848     2103     8712
 7340 root     sh /usr/bin/with-localenv s       92        4      296     1448
 7350 root     sh ./run                          84        4      329     1584
11929 user     tmux                              80     3124     3159     5088
  863 avahi    avahi-daemon: chroot helper       72      100      309     1572
929449 root     /usr/sbin/cups-browsed            68     4684     5400    15252
 3442 root     pihole-FTL no-daemon              64     7812     8120     9560
  808 root     /usr/lib/accountsservice/ac       64     1652     1708     6636
 2496 root     /usr/bin/containerd-shim-ru       60     3032     3650     7092
 9235 user     tini -g -- start.sh jupyter       56       44      154      268
 7338 user     tail -f --follow=name /home       44       76      318     1412
  858 root     /lib/systemd/systemd-logind       40     1320     1599     5992
 6778 user     /usr/sbin/upsd                    36       96      191      476
 7412 user     /sbin/docker-init -- /home/       32       16       65      416
 6961 root     /sbin/docker-init -- /init        32       16       65      416
 1177 root     gdm-session-worker [pam/gdm       32     2028     2334     8036
  960 root     sshd: /usr/sbin/sshd -D [li       32      888     1207     5500
  812 avahi    avahi-daemon: running [macm       32     1084     1311     4192
 7814 user     /sbin/docker-init -- python       28       16       65      416
 7725 user     /sbin/docker-init -- python       28       16       65      416
 6697 root     /sbin/docker-init -- docker       28       16       65      416
 1704 kernoops /usr/sbin/kerneloops --test       28      444      483     2400
 6521 user     /sbin/docker-init -- python       24       16       65      416
  324 root     /lib/systemd/systemd-journa       24    16528    40572    68544
 9399 root     s6-svscan -t0 /var/run/s6/s       20        4        4        8
 7645 user     /sbin/docker-init -- python       20       24       73      424
 7001 root     s6-supervise docker-cmd           16        4        4        8
 6975 root     s6-svscan -t0 /var/run/s6/s       16        4        4        8
  859 root     /usr/sbin/thermald --system       16     1216     1568     8244
 9639 root     s6-supervise home-assistant       12        4        4        8
 7000 root     s6-supervise s6-fdholderd         12        4        4        8
 3425 root     s6-supervise cron                 12        4        4        8
 2645 root     s6-supervise s6-fdholderd         12        4        4        8
 2522 root     s6-svscan -t0 /var/run/s6/s       12        4        4        8
  940 root     /usr/lib/bolt/boltd               12     1152     1212     6320
  776 systemd-timesync /lib/systemd/systemd-timesy       12      896     1092     5244
 9444 root     s6-supervise s6-fdholderd          8        4        4        8
 7543 user     /sbin/docker-init -- python        8       32       81      432
 7336 root     s6-supervise cron                  8        4        4        8
 7334 root     s6-supervise hb-service-log        8        4        4        8
 7333 root     s6-supervise homebridge-con        8        4        4        8
 3424 root     s6-supervise pihole-FTL            8        8        8       12
 3422 root     s6-supervise lighttpd              8        8        8       12
 2018 root     bpfilter_umh                       4      800      800      804
1607596 root     /usr/bin/python3 /usr/bin/s        0    12320    13331    18924
1607594 root     sudo smem -s swap -r               0     1364     1530     4672
1607390 user     -bash                              0     1756     2294     5224
1607388 user     sshd: user@pts/2                   0      564     1473     5968
1607329 root     sshd: user [priv]                  0      572     1992     9128
1605208 user     less                               0      472      608     2656
1605196 user     man smem                           0     2108     2227     4512
1594537 user     -bash                              0     1748     2277     5156
1594536 user     sshd: user@pts/0                   0      568     1488     6124
1594431 root     sshd: user [priv]                  0      564     1990     9120
1179565 user     python3 -u ./get_mikrotik_c        0    48264    50363    61900
930260 lp       /usr/lib/cups/notifier/dbus        0      820     1069     6708
930259 lp       /usr/lib/cups/notifier/dbus        0      996     1240     6908
 9378 root     /usr/bin/containerd-shim-ru        0     2120     2749     5768
 9213 root     /usr/bin/containerd-shim-ru        0     1660     1791     3916
 9197 root     /usr/bin/docker-proxy -prot        0      712      779     2404
 7982 root     /usr/bin/containerd-shim-ru        0     3084     3710     7168
 7966 root     /usr/bin/docker-proxy -prot        0      644      711     2336
 7870 root     /usr/bin/containerd-shim-ru        0     1648     1779     3904
 7793 root     /usr/bin/containerd-shim-ru        0     1632     1763     3888
 7703 root     /usr/bin/containerd-shim-ru        0     1656     1787     3912
 7623 root     /usr/bin/containerd-shim-ru        0     1684     1815     3940
 7520 root     /usr/bin/containerd-shim-ru        0     1776     2058     4640
 7391 root     /usr/bin/containerd-shim-ru        0     2860     3470     6912
 6939 root     /usr/bin/containerd-shim-ru        0     1700     1831     3956
 6804 root     /usr/bin/docker-proxy -prot        0      648      714     2336
 6660 root     /usr/bin/docker-proxy -prot        0     2516     2585     4480
 6248 root     /usr/bin/docker-proxy -prot        0      724      791     2496
 6235 root     /usr/bin/docker-proxy -prot        0      640      702     2400
 3608 user     update-notifier                    0     9756    10326    22772
 3605 user     /usr/libexec/gvfsd-metadata        0      716      762     5180
 3520 uuidd    /usr/sbin/uuidd --socket-ac        0      244      264     2292
 2453 root     /usr/bin/docker-proxy -prot        0      820      882     2588
 2440 root     /usr/bin/docker-proxy -prot        0      648      708     2324
 2427 root     /usr/bin/docker-proxy -prot        0      640      702     2360
 2415 root     /usr/bin/docker-proxy -prot        0      612      673     2276
 2402 root     /usr/bin/docker-proxy -prot        0      712      773     2456
 1706 kernoops /usr/sbin/kerneloops               0      480      521     2416
 1449 user     /usr/bin/ssh-agent /usr/bin        0      472      478     1328
 1382 user     /usr/libexec/gnome-session-        0     1672     1889    10848
 1208 rtkit    /usr/libexec/rtkit-daemon          0      348      376     2916
  861 root     /sbin/wpa_supplicant -u -s         0     1628     1725     4564
  857 root     /usr/libexec/switcheroo-con        0      672      723     5212
  842 nvidia-persistenced /usr/bin/nvidia-persistence        0      540      552     2172
  831 root     /usr/sbin/mbpfan -f                0      156      169     1812
  827 root     /usr/sbin/irqbalance --fore        0      504      528     3568
  813 root     /usr/lib/bluetooth/bluetoot        0     1264     1292     4320
  809 root     /usr/sbin/acpid                    0      228      243     1992

I have restarted ubuntu 20.04 host, and just started a minimal number of containers, output of command you gave me and from smem, two snapshots, one right after boot, second after starting minimal container. Both seem to show no swap usage at this time.

I then manually executed the purge service

unfortunatly I did not see hardly any swap space increase it was very small usage. There do seem to be a number of 'systemd-coredump postgres:' entries, however none of them are using swap space.

I am now wondering if this test might not have been useful as there are very few sensors adding records to the recorder database while the purge was executing this time. I think should have a full load running during the purge.....

#right after reboot of physical machine, HA and postgresql container NOT yet running

user@macmini2012:~$ echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r
NAME PID SWAP
user@macmini2012:~$

user@macmini2012:~$ sudo smem -s swap -r
[sudo] password for user:
  PID User     Command                         Swap      USS      PSS      RSS
 3560 root     /usr/bin/python3 /usr/bin/s        0    12064    13302    18576
 3559 root     sudo smem -s swap -r               0     1468     1602     4808
 3529 user     -bash                              0     2800     2883     5192
 3528 user     sshd: user@pts/0                   0      600     1753     6040
 3433 root     sshd: user [priv]                  0      724     2340     9012
 3379 lp       /usr/lib/cups/notifier/dbus        0      820     1047     6932
 3378 lp       /usr/lib/cups/notifier/dbus        0      828     1048     6908
 3337 uuidd    /usr/sbin/uuidd --socket-ac        0      244      276     2964
 3305 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3304 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3303 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3302 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3301 www-data /usr/bin/php-cgi                   0    17852    20393    27620
 3289 root     pihole-FTL no-daemon               0     6224     6542     8100
 3286 root     /usr/sbin/cron -f                  0      476      752     2268
 3285 www-data lighttpd -D -f /etc/lighttp        0     1460     2934     5652
 3273 root     bash ./run                         0      284      825     2792
 3272 root     bash ./run                         0      280      824     2800
 3271 root     bash ./run                         0      280      832     2852
 3269 root     s6-supervise cron                  0       12       23       60
 3267 root     s6-supervise pihole-FTL            0       16       27       64
 3266 root     s6-supervise lighttpd              0       16       27       64
 2497 root     s6-supervise s6-fdholderd          0       16       27       64
 2371 root     /portainer                         0    37688    37688    37692
 2370 root     s6-svscan -t0 /var/run/s6/s        0       56       60       68
 2327 root     /usr/bin/containerd-shim-ru        0     1928     5828     9732
 2326 root     /usr/bin/containerd-shim-ru        0     2000     5900     9804
 2300 root     /usr/bin/docker-proxy -prot        0      824     1201     3724
 2287 root     /usr/bin/docker-proxy -prot        0      648     1026     3636
 2273 root     /usr/bin/docker-proxy -prot        0      676     1053     3564
 2260 root     /usr/bin/docker-proxy -prot        0      644     1022     3628
 2246 root     /usr/bin/docker-proxy -prot        0      736     1112     3624
 1870 root     bpfilter_umh                       0      804      804      808
 1838 user     /usr/sbin/smbd --foreground        0     2568     5444    17492
 1816 user     /home/user/.dropbox-dist/dr        0   359100   360271   379376
 1795 colord   /usr/libexec/colord                0     5976     6536    14912
 1794 user     /usr/lib/vino/vino-server          0    12768    17353    43152
 1792 user     /usr/libexec/gsd-printer           0     2020     2389    15164
 1732 user     /usr/libexec/gsd-disk-utili        0     1524     1709     7336
 1730 user     /usr/libexec/ibus-engine-si        0      816      947     7220
 1677 user     /usr/libexec/evolution-data        0    17032    26342    61428
 1668 user     /usr/libexec/gsd-xsettings         0    10768    12059    29932
 1665 user     /usr/libexec/gsd-wwan              0     1132     1393     8644
 1664 user     /usr/libexec/gsd-wacom             0    10124    10956    28232
 1662 user     /usr/libexec/gsd-usb-protec        0      932     1017     7432
 1661 user     /usr/libexec/gsd-sound             0     1240     1496     9692
 1660 user     /usr/libexec/gsd-smartcard         0     3068     3285    10356
 1659 user     /usr/libexec/gsd-sharing           0     1744     2430    10932
 1658 user     /usr/libexec/gsd-screensave        0      680      737     6104
 1657 user     /usr/libexec/gsd-rfkill            0      916      985     6732
 1656 user     /usr/libexec/gsd-print-noti        0     1580     1863    11332
 1655 user     /usr/libexec/gsd-power             0    10368    11255    29232
 1652 user     /usr/libexec/gsd-media-keys        0    10876    12273    31260
 1650 user     /usr/libexec/gsd-keyboard          0    10632    11506    29360
 1648 user     /usr/libexec/gsd-housekeepi        0     1080     1213     8376
 1646 user     /usr/libexec/gsd-datetime          0     2152     2643    16376
 1645 user     /usr/libexec/gsd-color             0    10944    12159    30408
 1644 user     /usr/libexec/gsd-a11y-setti        0      792      863     6800
 1624 user     /usr/libexec/gvfsd-trash --        0     1464     1683     8896
 1612 user     /usr/bin/gjs /usr/share/gno        0     5612    10965    27040
 1597 root     /usr/lib/packagekit/package        0     3160     4385    15896
 1589 user     /usr/libexec/gvfs-afc-volum        0     1252     1490     8876
 1585 user     /usr/libexec/gvfs-gphoto2-v        0     1272     1405     7432
 1581 user     /usr/libexec/gvfs-goa-volum        0      900     1015     6584
 1575 user     /usr/libexec/gvfs-mtp-volum        0      940     1003     6664
 1571 root     /usr/sbin/smbd --foreground        0      340     1761    10964
 1569 root     /usr/sbin/smbd --foreground        0      156      997     7548
 1568 root     /usr/sbin/smbd --foreground        0      156     1082     8220
 1562 root     /usr/sbin/smbd --foreground        0     2164     6098    22336
 1526 kernoops /usr/sbin/kerneloops               0      476      535     2472
 1524 kernoops /usr/sbin/kerneloops --test        0      476      539     2556
 1516 root     /usr/sbin/nmbd --foreground        0     2456     4152    14052
 1508 root     /usr/bin/dockerd -H fd:// -        0   102824   102881   106252
 1447 user     /usr/libexec/evolution-addr        0     5412     9180    29940
 1437 user     /usr/libexec/dconf-service         0      964     1017     5964
 1427 user     /usr/libexec/goa-identity-s        0     1868     2049     9492
 1424 geoclue  /usr/libexec/geoclue               0     5836     6890    19388
 1413 user     /usr/libexec/evolution-cale        0     5728     9110    30916
 1410 root     /usr/lib/upower/upowerd            0     2008     2305    10064
 1409 user     /usr/libexec/goa-daemon            0     7276    14607    38488
 1401 user     /usr/libexec/evolution-sour        0     4340     5962    25356
 1392 user     /usr/libexec/gnome-shell-ca        0     3372     5335    20784
 1390 user     /usr/libexec/xdg-permission        0      744      810     6288
 1386 user     /usr/libexec/at-spi2-regist        0      944     1053     7852
 1374 user     /usr/libexec/ibus-portal           0      820      956     7356
 1371 user     /usr/libexec/ibus-x11 --kil        0     9952    10840    28092
 1369 user     /usr/libexec/ibus-extension        0    13840    14814    32724
 1368 user     /usr/libexec/ibus-dconf            0      904     1046     7308
 1364 user     ibus-daemon --panel disable        0     1624     1767     8244
 1320 user     /usr/bin/gnome-shell               0   250584   266813   311652
 1306 user     /usr/libexec/gnome-session-        0     2748     3294    15792
 1299 user     /usr/libexec/gnome-session-        0      472      514     5176
 1273 user     /usr/bin/dbus-daemon --conf        0      512      663     4348
 1267 user     /usr/libexec/at-spi-bus-lau        0      820      917     7096
 1235 user     /usr/bin/ssh-agent /usr/bin        0      696      704     1524
 1149 user     /usr/libexec/gnome-session-        0     1680     2152    13868
 1093 user     /usr/libexec/gvfs-udisks2-v        0     2420     2731    10832
 1087 user     /usr/libexec/gvfsd-fuse /ru        0     1140     1202     6836
 1072 user     /usr/libexec/gvfsd                 0     1020     1205     7796
 1065 root     /usr/lib/xorg/Xorg vt2 -dis        0    36248    39459    49208
 1064 rtkit    /usr/libexec/rtkit-daemon          0      348      381     3256
 1060 user     /usr/lib/gdm3/gdm-x-session        0      772      821     6484
 1054 user     /usr/bin/dbus-daemon --sess        0     2564     2718     6548
 1053 user     /usr/bin/gnome-keyring-daem        0     2296     2705     7960
 1048 user     /usr/libexec/tracker-miner-        0    11500    12797    25980
 1046 user     /usr/bin/pulseaudio --daemo        0     7848     8776    19956
 1038 user     (sd-pam)                           0     2020     3066     5764
 1037 user     /lib/systemd/systemd --user        0     2676     3806    10836
  912 root     gdm-session-worker [pam/gdm        0     2092     2535     9904
  911 root     /usr/bin/python3 /usr/share        0     8632    11014    23172
  870 root     /usr/sbin/gdm3                     0     2060     2256     9296
  859 root     sshd: /usr/sbin/sshd -D [li        0      956     1741     7292
  837 root     /usr/lib/bolt/boltd                0     1156     1237     7652
  835 root     /usr/sbin/ModemManager --fi        0     4528     4726    11020
  831 root     /usr/bin/containerd                0    48976    48985    50352
  816 root     /snap/canonical-livepatch/9        0     9608     9608     9612
  809 root     /usr/sbin/cups-browsed             0     4128     4710    14876
  769 avahi    avahi-daemon: chroot helper        0      144      371     1724
  767 root     /sbin/wpa_supplicant -u -s         0     2788     2874     6440
  766 root     /usr/lib/udisks2/udisksd           0     4524     4939    13488
  765 root     /usr/sbin/thermald --system        0     1648     2082    10316
  764 root     /lib/systemd/systemd-logind        0     1484     1988     8304
  763 root     /usr/libexec/switcheroo-con        0      648      732     6488
  758 root     /usr/lib/snapd/snapd               0    32188    32195    33324
  755 root     /usr/sbin/smartd -n                0     2252     2349     6404
  753 syslog   /usr/sbin/rsyslogd -n -iNON        0     3036     3082     6036
  750 nvidia-persistenced /usr/bin/nvidia-persistence        0      532      545     2160
  747 root     /usr/lib/policykit-1/polkit        0     3444     3550    10128
  740 root     /usr/bin/python3 /usr/bin/n        0     8444    10183    20384
  738 root     /usr/sbin/mbpfan -f                0      156      171     1868
  732 root     /usr/sbin/irqbalance --fore        0      512      540     3792
  726 root     /usr/sbin/NetworkManager --        0     7536     8176    19968
  724 messagebus /usr/bin/dbus-daemon --syst        0     2140     2295     6032
  722 root     /usr/sbin/cupsd -l                 0     2540     2821     9536
  720 root     /usr/sbin/cron -f                  0      316      347     3028
  719 root     /usr/lib/bluetooth/bluetoot        0     1380     1419     4952
  718 avahi    avahi-daemon: running [macm        0      596      856     4228
  715 root     /usr/sbin/acpid                    0      228      244     1928
  714 root     /usr/lib/accountsservice/ac        0     1688     1763     8080
  681 systemd-timesync /lib/systemd/systemd-timesy        0      892     1320     7436
  679 systemd-resolve /lib/systemd/systemd-resolv        0     4736     5372    13200
  354 root     /lib/systemd/systemd-udevd         0     4280     4408     7544
  327 root     /lib/systemd/systemd-journa        0    39132    44155    54620
    1 root     /sbin/init                         0     1800     3845    11888

# minimal number of containers running on host
just :
pihole
home assistant
postgresq
portainer
mosquitto

user@macmini2012:~$ echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r
NAME PID SWAP

user@macmini2012:~$ sudo smem -s swap -r
  PID User     Command                         Swap      USS      PSS      RSS
 6080 root     /usr/bin/python3 /usr/bin/s        0    12424    13647    18968
 6079 root     sudo smem -s swap -r               0     1316     1427     4652
 6037 user     sleep 20                           0      124      142     1832
 5886 user     /bin/bash /usr/local/sbin/s        0      232      750     3268
 5869 user     /bin/sh -c /usr/local/sbin/        0      220      231     1616
 5856 root     /usr/sbin/CRON -f                  0      624      818     3672
 5706 systemd-coredump postgres: user homeassistan        0     7272    68840   155520
 5618 systemd-coredump postgres: user homeassistan        0     1216     3827    14836
 5181 systemd-coredump postgres: user homeassistan        0     5356    66451   152156
 5143 root     python3 -m homeassistant --        0   213412   279788   346168
 5140 root     s6-supervise home-assistant        0       16       34       60
 4953 root     s6-supervise s6-fdholderd          0       16       34       60
 4898 root     s6-svscan -t0 /var/run/s6/s        0       52       56       64
 4877 root     /usr/bin/containerd-shim-ru        0     1828     3422     9772
 4440 systemd-coredump postgres: logical replicati        0      452     1245     7660
 4439 systemd-coredump postgres: stats collector          0      236      742     5612
 4438 systemd-coredump postgres: autovacuum launch        0      580     1534     8580
 4437 systemd-coredump postgres: walwriter                0      212     1929    10264
 4436 systemd-coredump postgres: background writer        0      344    15288    49536
 4435 systemd-coredump postgres: checkpointer             0      172      729     5940
 4368 systemd-coredump postgres                           0     8480    12146    26112
 4346 root     /usr/bin/containerd-shim-ru        0     1960     3554     9904
 4254 user     /usr/sbin/mosquitto -c /mos        0     3968     3970     3976
 4233 root     /usr/bin/containerd-shim-ru        0     1956     3550     9900
 4216 root     /usr/bin/docker-proxy -prot        0      748     1018     3744
 4202 root     /usr/bin/docker-proxy -prot        0      656      926     3644
 3566 user     update-notifier                    0     9504    10538    29188
 3563 user     /usr/libexec/gvfsd-metadata        0      800      857     6336
 3529 user     -bash                              0     1856     2402     5192
 3528 user     sshd: user@pts/0                   0      600     1741     6040
 3433 root     sshd: user [priv]                  0      724     2311     9012
 3379 lp       /usr/lib/cups/notifier/dbus        0      820     1046     6932
 3378 lp       /usr/lib/cups/notifier/dbus        0      828     1047     6908
 3337 uuidd    /usr/sbin/uuidd --socket-ac        0      244      275     2964
 3305 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3304 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3303 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3302 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3301 www-data /usr/bin/php-cgi                   0    17852    20393    27620
 3289 root     pihole-FTL no-daemon               0     6396     6714     8272
 3286 root     /usr/sbin/cron -f                  0      476      752     2268
 3285 www-data lighttpd -D -f /etc/lighttp        0     1460     2934     5652
 3273 root     bash ./run                         0      284      825     2792
 3272 root     bash ./run                         0      280      824     2800
 3271 root     bash ./run                         0      280      832     2852
 3269 root     s6-supervise cron                  0       12       23       60
 3267 root     s6-supervise pihole-FTL            0       16       27       64
 3266 root     s6-supervise lighttpd              0       16       27       64
 2497 root     s6-supervise s6-fdholderd          0       16       27       64
 2371 root     /portainer                         0    38204    38204    38208
 2370 root     s6-svscan -t0 /var/run/s6/s        0       56       60       68
 2327 root     /usr/bin/containerd-shim-ru        0     2064     3640     9940
 2326 root     /usr/bin/containerd-shim-ru        0     2224     3795    10080
 2300 root     /usr/bin/docker-proxy -prot        0      824     1093     3724
 2287 root     /usr/bin/docker-proxy -prot        0      648      918     3636
 2273 root     /usr/bin/docker-proxy -prot        0      676      945     3564
 2260 root     /usr/bin/docker-proxy -prot        0      644      914     3628
 2246 root     /usr/bin/docker-proxy -prot        0      736     1005     3624
 1870 root     bpfilter_umh                       0      804      804      808
 1838 root     /usr/sbin/smbd --foreground        0     2632     5508    17556
 1816 user     /home/user/.dropbox-dist/dr        0   361672   362899   382248
 1795 colord   /usr/libexec/colord                0     5976     6587    14912
 1794 user     /usr/lib/vino/vino-server          0    12768    17284    43152
 1792 user     /usr/libexec/gsd-printer           0     2020     2377    15164
 1732 user     /usr/libexec/gsd-disk-utili        0     1524     1706     7336
 1730 user     /usr/libexec/ibus-engine-si        0      816      943     7220
 1677 user     /usr/libexec/evolution-data        0    17032    26296    61428
 1668 user     /usr/libexec/gsd-xsettings         0    10768    11999    29932
 1665 user     /usr/libexec/gsd-wwan              0     1132     1388     8644
 1664 user     /usr/libexec/gsd-wacom             0    10124    10897    28232
 1662 user     /usr/libexec/gsd-usb-protec        0      932     1014     7432
 1661 user     /usr/libexec/gsd-sound             0     1240     1491     9692
 1660 user     /usr/libexec/gsd-smartcard         0     3068     3301    10356
 1659 user     /usr/libexec/gsd-sharing           0     1744     2430    10932
 1658 user     /usr/libexec/gsd-screensave        0      680      735     6104
 1657 user     /usr/libexec/gsd-rfkill            0      920      984     6736
 1656 user     /usr/libexec/gsd-print-noti        0     1580     1860    11332
 1655 user     /usr/libexec/gsd-power             0    10368    11198    29232
 1652 user     /usr/libexec/gsd-media-keys        0    10884    12219    31268
 1650 user     /usr/libexec/gsd-keyboard          0    10632    11448    29360
 1648 user     /usr/libexec/gsd-housekeepi        0     1168     1298     8464
 1646 user     /usr/libexec/gsd-datetime          0     2152     2636    16376
 1645 user     /usr/libexec/gsd-color             0    10944    12101    30408
 1644 user     /usr/libexec/gsd-a11y-setti        0      792      860     6800
 1624 user     /usr/libexec/gvfsd-trash --        0     1484     1701     8916
 1612 user     /usr/bin/gjs /usr/share/gno        0     5612    10965    27040
 1589 user     /usr/libexec/gvfs-afc-volum        0     1252     1489     8876
 1585 user     /usr/libexec/gvfs-gphoto2-v        0     1272     1402     7432
 1581 user     /usr/libexec/gvfs-goa-volum        0      900     1012     6584
 1575 user     /usr/libexec/gvfs-mtp-volum        0      940      999     6664
 1571 root     /usr/sbin/smbd --foreground        0      340     1760    10964
 1569 root     /usr/sbin/smbd --foreground        0      156      995     7548
 1568 root     /usr/sbin/smbd --foreground        0      156     1082     8220
 1562 root     /usr/sbin/smbd --foreground        0     2164     6094    22336
 1526 kernoops /usr/sbin/kerneloops               0      476      533     2472
 1524 kernoops /usr/sbin/kerneloops --test        0      476      536     2556
 1516 root     /usr/sbin/nmbd --foreground        0     2480     4175    14076
 1508 root     /usr/bin/dockerd -H fd:// -        0   103760   103815   107188
 1447 user     /usr/libexec/evolution-addr        0     5416     9251    29944
 1437 user     /usr/libexec/dconf-service         0      964     1014     5964
 1427 user     /usr/libexec/goa-identity-s        0     1876     2055     9500
 1424 geoclue  /usr/libexec/geoclue               0     5840     6892    19392
 1413 user     /usr/libexec/evolution-cale        0     5728     9108    30916
 1410 root     /usr/lib/upower/upowerd            0     2008     2304    10064
 1409 user     /usr/libexec/goa-daemon            0     7288    14630    38504
 1401 user     /usr/libexec/evolution-sour        0     4340     5957    25356
 1392 user     /usr/libexec/gnome-shell-ca        0     3372     5340    20784
 1390 user     /usr/libexec/xdg-permission        0      744      807     6288
 1386 user     /usr/libexec/at-spi2-regist        0      944     1050     7852
 1374 user     /usr/libexec/ibus-portal           0      820      956     7356
 1371 user     /usr/libexec/ibus-x11 --kil        0     9952    10780    28092
 1369 user     /usr/libexec/ibus-extension        0    13840    14745    32724
 1368 user     /usr/libexec/ibus-dconf            0      904     1046     7308
 1364 user     ibus-daemon --panel disable        0     1624     1765     8244
 1320 user     /usr/bin/gnome-shell               0   248772   264931   309804
 1306 user     /usr/libexec/gnome-session-        0     2744     3275    15792
 1299 user     /usr/libexec/gnome-session-        0      472      514     5176
 1273 user     /usr/bin/dbus-daemon --conf        0      524      670     4360
 1267 user     /usr/libexec/at-spi-bus-lau        0      820      914     7096
 1235 user     /usr/bin/ssh-agent /usr/bin        0      696      704     1524
 1149 user     /usr/libexec/gnome-session-        0     1680     2134    13868
 1093 user     /usr/libexec/gvfs-udisks2-v        0     2472     2778    10884
 1087 user     /usr/libexec/gvfsd-fuse /ru        0     1140     1202     6836
 1072 user     /usr/libexec/gvfsd                 0     1024     1207     7800
 1065 root     /usr/lib/xorg/Xorg vt2 -dis        0    36280    39494    49240
 1064 rtkit    /usr/libexec/rtkit-daemon          0      348      379     3256
 1060 user     /usr/lib/gdm3/gdm-x-session        0      772      820     6484
 1054 user     /usr/bin/dbus-daemon --sess        0     2564     2714     6548
 1053 user     /usr/bin/gnome-keyring-daem        0     2296     2703     7960
 1048 user     /usr/libexec/tracker-miner-        0    11660    12941    25980
 1046 user     /usr/bin/pulseaudio --daemo        0     7848     8777    19956
 1038 user     (sd-pam)                           0     2028     3068     5764
 1037 user     /lib/systemd/systemd --user        0     2676     3806    10836
  912 root     gdm-session-worker [pam/gdm        0     2092     2507     9904
  911 root     /usr/bin/python3 /usr/share        0     9772    11598    23172
  870 root     /usr/sbin/gdm3                     0     2060     2250     9296
  859 root     sshd: /usr/sbin/sshd -D [li        0      956     1740     7292
  837 root     /usr/lib/bolt/boltd                0     1156     1236     7652
  835 root     /usr/sbin/ModemManager --fi        0     4528     4724    11020
  831 root     /usr/bin/containerd                0    50688    50697    52064
  816 root     /snap/canonical-livepatch/9        0    10076    10076    10080
  809 root     /usr/sbin/cups-browsed             0     4176     4738    14876
  769 avahi    avahi-daemon: chroot helper        0      144      369     1724
  767 root     /sbin/wpa_supplicant -u -s         0     2788     2872     6440
  766 root     /usr/lib/udisks2/udisksd           0     4524     4936    13488
  765 root     /usr/sbin/thermald --system        0     1648     2091    10316
  764 root     /lib/systemd/systemd-logind        0     1484     1989     8304
  763 root     /usr/libexec/switcheroo-con        0      648      724     6488
  758 root     /usr/lib/snapd/snapd               0    32200    32207    33336
  755 root     /usr/sbin/smartd -n                0     2252     2354     6404
  753 syslog   /usr/sbin/rsyslogd -n -iNON        0     3040     3085     6040
  750 nvidia-persistenced /usr/bin/nvidia-persistence        0      540      553     2228
  747 root     /usr/lib/policykit-1/polkit        0     3444     3547    10128
  740 root     /usr/bin/python3 /usr/bin/n        0     8444    10180    20384
  738 root     /usr/sbin/mbpfan -f                0      156      170     1868
  732 root     /usr/sbin/irqbalance --fore        0      512      539     3792
  726 root     /usr/sbin/NetworkManager --        0     7544     8184    19976
  724 messagebus /usr/bin/dbus-daemon --syst        0     2136     2286     6028
  722 root     /usr/sbin/cupsd -l                 0     2540     2820     9536
  720 root     /usr/sbin/cron -f                  0      172      274     3028
  719 root     /usr/lib/bluetooth/bluetoot        0     1384     1420     4956
  718 avahi    avahi-daemon: running [macm        0      768     1024     4400
  715 root     /usr/sbin/acpid                    0      228      243     1928
  714 root     /usr/lib/accountsservice/ac        0     1688     1762     8080
  681 systemd-timesync /lib/systemd/systemd-timesy        0      900     1328     7444
  679 systemd-resolve /lib/systemd/systemd-resolv        0     4744     5380    13208
  354 root     /lib/systemd/systemd-udevd         0     4280     4408     7544
  327 root     /lib/systemd/systemd-journa        0    39364    44447    54972
    1 root     /sbin/init                         0     1808     3848    11888

# after run of purge with 'keep 64 days' this should have doubled the purge I normally do

user@macmini2012:~$ echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r
NAME PID SWAP
(sd-pam) 1038 628 kB
systemd 1037 472 kB
dockerd 1508 444 kB
python3 5143 312 kB
containerd 831 288 kB
snapd 758 180 kB
unattended-upgr 911 80 kB
networkd-dispat 740 80 kB
portainer 2371 44 kB
geoclue 1424 28 kB
colord 1795 28 kB
s6-svscan 4898 20 kB
s6-supervise 5140 16 kB
s6-supervise 4953 16 kB
polkitd 747 16 kB
cups-browsed 809 16 kB
systemd-udevd 354 12 kB
udisksd 766 8 kB
sshd 859 8 kB
sshd 3433 8 kB
rsyslogd 753 8 kB
gdm3 870 8 kB
cupsd 722 8 kB
canonical-livep 816 8 kB
systemd-resolve 679 4 kB
sshd 3528 4 kB
smartd 755 4 kB
nmbd 1516 4 kB
mosquitto 4254 4 kB
ModemManager 835 4 kB
dbus-daemon 724 4 kB
accounts-daemon 714 4 kB

user@macmini2012:~$ sudo smem -s swap -r
  PID User     Command                         Swap      USS      PSS      RSS
 1038 user     (sd-pam)                         624     1432     2456     5076
 1037 user     /lib/systemd/systemd --user      468     2208     3339    10368
 1508 root     /usr/bin/dockerd -H fd:// -      412   106388   106430   109548
 5143 root     python3 -m homeassistant --      300   374192   374198   374208
  831 root     /usr/bin/containerd              284    50952    50961    52328
  758 root     /usr/lib/snapd/snapd             152    28528    28535    29664
  911 root     /usr/bin/python3 /usr/share       76     9696    11524    23020
  740 root     /usr/bin/python3 /usr/bin/n       76     8368    10106    20232
 2371 root     /portainer                        40    38344    38344    38348
 1795 colord   /usr/libexec/colord               24     5952     6563    14812
 1424 geoclue  /usr/libexec/geoclue              24     5816     6820    19224
 4898 root     s6-svscan -t0 /var/run/s6/s       16       36       40       48
 5140 root     s6-supervise home-assistant       12        4       22       48
 4953 root     s6-supervise s6-fdholderd         12        4       22       48
  809 root     /usr/sbin/cups-browsed            12     4164     4726    14816
  747 root     /usr/lib/policykit-1/polkit       12     3432     3524     9836
  354 root     /lib/systemd/systemd-udevd        12     4268     4396     7488
 3433 root     sshd: user [priv]                  8      720     1949     7736
 1838 user     /usr/sbin/smbd --foreground        8     5592     8473    20276
 1571 root     /usr/sbin/smbd --foreground        8      340     1753    10864
 1569 root     /usr/sbin/smbd --foreground        8      164     1001     7548
 1568 root     /usr/sbin/smbd --foreground        8      164     1103     8348
 1562 root     /usr/sbin/smbd --foreground        8     2164     6082    22076
  859 root     sshd: /usr/sbin/sshd -D [li        8      940     1371     6248
  766 root     /usr/lib/udisks2/udisksd           8     4528     4940    13416
  753 syslog   /usr/sbin/rsyslogd -n -iNON        8     3032     3065     5776
 3528 user     sshd: user@pts/0                   4      600     1481     5040
  870 root     /usr/sbin/gdm3                     4     2056     2249     9244
  816 root     /snap/canonical-livepatch/9        4    10072    10072    10076
  679 systemd-resolve /lib/systemd/systemd-resolv        4     4744     5381    13208
 8836 root     /usr/bin/python3 /usr/bin/s        0    11680    12920    18104
 8835 root     sudo smem -s swap -r               0     1516     1670     4720
 6683 systemd-coredump postgres: user homeassistan        0     2620    23838   154712
 6682 systemd-coredump postgres: user homeassistan        0     2764    23964   154784
 6681 systemd-coredump postgres: user homeassistan        0     2880    24100   154976
 6678 systemd-coredump postgres: user homeassistan        0     2924    24146   155024
 5618 systemd-coredump postgres: user homeassistan        0     1232     2995    17484
 5181 systemd-coredump postgres: user homeassistan        0     3012    24213   155024
 4877 root     /usr/bin/containerd-shim-ru        0     2000     3594     9944
 4440 systemd-coredump postgres: logical replicati        0      452     1019     7660
 4439 systemd-coredump postgres: stats collector          0      236      620     5612
 4438 systemd-coredump postgres: autovacuum launch        0      588     1398     9836
 4437 systemd-coredump postgres: walwriter                0      212     1136    10264
 4436 systemd-coredump postgres: background writer        0      352    20054   142040
 4435 systemd-coredump postgres: checkpointer             0      896    20284   141124
 4368 systemd-coredump postgres                           0     8328    10292    26112
 4346 root     /usr/bin/containerd-shim-ru        0     2036     3630     9980
 4254 user     /usr/sbin/mosquitto -c /mos        0     3968     3970     3976
 4233 root     /usr/bin/containerd-shim-ru        0     1956     3550     9900
 4216 root     /usr/bin/docker-proxy -prot        0      748     1019     3744
 4202 root     /usr/bin/docker-proxy -prot        0      656      926     3644
 3566 user     update-notifier                    0     9504    10525    28920
 3563 user     /usr/libexec/gvfsd-metadata        0      800      857     6336
 3529 user     -bash                              0     2608     2632     4756
 3379 lp       /usr/lib/cups/notifier/dbus        0      820     1047     6884
 3378 lp       /usr/lib/cups/notifier/dbus        0      828     1047     6860
 3337 uuidd    /usr/sbin/uuidd --socket-ac        0      244      276     2964
 3305 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3304 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3303 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3302 www-data /usr/bin/php-cgi                   0       52     1126     5996
 3301 www-data /usr/bin/php-cgi                   0    17852    20393    27620
 3289 root     pihole-FTL no-daemon               0     6524     6842     8400
 3286 root     /usr/sbin/cron -f                  0      476      752     2268
 3285 www-data lighttpd -D -f /etc/lighttp        0     1460     2934     5652
 3273 root     bash ./run                         0      284      825     2792
 3272 root     bash ./run                         0      280      824     2800
 3271 root     bash ./run                         0      280      832     2852
 3269 root     s6-supervise cron                  0       12       23       60
 3267 root     s6-supervise pihole-FTL            0       16       27       64
 3266 root     s6-supervise lighttpd              0       16       27       64
 2497 root     s6-supervise s6-fdholderd          0       16       27       64
 2370 root     s6-svscan -t0 /var/run/s6/s        0       56       60       68
 2327 root     /usr/bin/containerd-shim-ru        0     2064     3640     9940
 2326 root     /usr/bin/containerd-shim-ru        0     2292     3863    10148
 2300 root     /usr/bin/docker-proxy -prot        0      824     1094     3724
 2287 root     /usr/bin/docker-proxy -prot        0      648      918     3636
 2273 root     /usr/bin/docker-proxy -prot        0      676      946     3564
 2260 root     /usr/bin/docker-proxy -prot        0      644      914     3628
 2246 root     /usr/bin/docker-proxy -prot        0      736     1005     3624
 1870 root     bpfilter_umh                       0      804      804      808
 1816 user     /home/user/.dropbox-dist/dr        0   361920   363159   382456
 1794 user     /usr/lib/vino/vino-server          0    12768    17237    43008
 1792 user     /usr/libexec/gsd-printer           0     2020     2379    15164
 1732 user     /usr/libexec/gsd-disk-utili        0     1524     1706     7340
 1730 user     /usr/libexec/ibus-engine-si        0      816      944     7220
 1677 user     /usr/libexec/evolution-data        0    17032    26283    61328
 1668 user     /usr/libexec/gsd-xsettings         0    10768    12000    29932
 1665 user     /usr/libexec/gsd-wwan              0     1132     1388     8644
 1664 user     /usr/libexec/gsd-wacom             0    10124    10874    28172
 1662 user     /usr/libexec/gsd-usb-protec        0      932     1015     7432
 1661 user     /usr/libexec/gsd-sound             0     1240     1492     9692
 1660 user     /usr/libexec/gsd-smartcard         0     3068     3301    10356
 1659 user     /usr/libexec/gsd-sharing           0     1792     2188    10400
 1658 user     /usr/libexec/gsd-screensave        0      680      736     6104
 1657 user     /usr/libexec/gsd-rfkill            0      920      986     6736
 1656 user     /usr/libexec/gsd-print-noti        0     1580     1860    11284
 1655 user     /usr/libexec/gsd-power             0    10368    11198    29156
 1652 user     /usr/libexec/gsd-media-keys        0    10884    12219    31192
 1650 user     /usr/libexec/gsd-keyboard          0    10632    11449    29312
 1648 user     /usr/libexec/gsd-housekeepi        0     1228     1358     8524
 1646 user     /usr/libexec/gsd-datetime          0     2152     2637    16376
 1645 user     /usr/libexec/gsd-color             0    10944    12102    30408
 1644 user     /usr/libexec/gsd-a11y-setti        0      792      860     6800
 1624 user     /usr/libexec/gvfsd-trash --        0     1488     1707     8920
 1612 user     /usr/bin/gjs /usr/share/gno        0     5612    10937    26928
 1589 user     /usr/libexec/gvfs-afc-volum        0     1252     1489     8876
 1585 user     /usr/libexec/gvfs-gphoto2-v        0     1272     1403     7404
 1581 user     /usr/libexec/gvfs-goa-volum        0      900     1013     6584
 1575 user     /usr/libexec/gvfs-mtp-volum        0      940     1000     6636
 1526 kernoops /usr/sbin/kerneloops               0      476      534     2472
 1524 kernoops /usr/sbin/kerneloops --test        0      476      536     2556
 1516 root     /usr/sbin/nmbd --foreground        0     2488     4187    14092
 1447 user     /usr/libexec/evolution-addr        0     5416     9252    29896
 1437 user     /usr/libexec/dconf-service         0      964     1015     5964
 1427 user     /usr/libexec/goa-identity-s        0     1876     2056     9500
 1413 user     /usr/libexec/evolution-cale        0     5728     9108    30868
 1410 root     /usr/lib/upower/upowerd            0     2008     2304    10036
 1409 user     /usr/libexec/goa-daemon            0     7288    14617    38452
 1401 user     /usr/libexec/evolution-sour        0     4340     5957    25308
 1392 user     /usr/libexec/gnome-shell-ca        0     3372     5340    20708
 1390 user     /usr/libexec/xdg-permission        0      744      808     6288
 1386 user     /usr/libexec/at-spi2-regist        0      944     1050     7824
 1374 user     /usr/libexec/ibus-portal           0      820      956     7356
 1371 user     /usr/libexec/ibus-x11 --kil        0     9952    10781    28092
 1369 user     /usr/libexec/ibus-extension        0    13840    14745    32724
 1368 user     /usr/libexec/ibus-dconf            0      904     1046     7308
 1364 user     ibus-daemon --panel disable        0     1624     1767     8196
 1320 user     /usr/bin/gnome-shell               0   266060   281891   326272
 1306 user     /usr/libexec/gnome-session-        0     2744     3276    15792
 1299 user     /usr/libexec/gnome-session-        0      472      514     5176
 1273 user     /usr/bin/dbus-daemon --conf        0      524      658     4092
 1267 user     /usr/libexec/at-spi-bus-lau        0      820      914     7068
 1235 user     /usr/bin/ssh-agent /usr/bin        0      696      704     1524
 1149 user     /usr/libexec/gnome-session-        0     1680     2134    13868
 1093 user     /usr/libexec/gvfs-udisks2-v        0     2472     2778    10836
 1087 user     /usr/libexec/gvfsd-fuse /ru        0     1140     1202     6808
 1072 user     /usr/libexec/gvfsd                 0     1024     1209     7800
 1065 root     /usr/lib/xorg/Xorg vt2 -dis        0    36280    39479    49148
 1064 rtkit    /usr/libexec/rtkit-daemon          0      348      380     3208
 1060 user     /usr/lib/gdm3/gdm-x-session        0      772      821     6484
 1054 user     /usr/bin/dbus-daemon --sess        0     2564     2702     6288
 1053 user     /usr/bin/gnome-keyring-daem        0     2296     2704     7960
 1048 user     /usr/libexec/tracker-miner-        0    11660    12941    25980
 1046 user     /usr/bin/pulseaudio --daemo        0     7848     8779    19928
  912 root     gdm-session-worker [pam/gdm        0     2092     2520     9612
  837 root     /usr/lib/bolt/boltd                0     1156     1236     7576
  835 root     /usr/sbin/ModemManager --fi        0     4528     4724    10992
  769 avahi    avahi-daemon: chroot helper        0      144      367     1660
  767 root     /sbin/wpa_supplicant -u -s         0     2788     2873     6440
  765 root     /usr/sbin/thermald --system        0     1652     2097    10292
  764 root     /lib/systemd/systemd-logind        0     1484     1989     8256
  763 root     /usr/libexec/switcheroo-con        0      648      725     6460
  755 root     /usr/sbin/smartd -n                0     2252     2354     6404
  750 nvidia-persistenced /usr/bin/nvidia-persistence        0      544      558     2220
  738 root     /usr/sbin/mbpfan -f                0      156      171     1868
  732 root     /usr/sbin/irqbalance --fore        0      512      539     3792
  726 root     /usr/sbin/NetworkManager --        0     7552     8170    19820
  724 messagebus /usr/bin/dbus-daemon --syst        0     2136     2274     5760
  722 root     /usr/sbin/cupsd -l                 0     2544     2825     9492
  720 root     /usr/sbin/cron -f                  0      316      346     2952
  719 root     /usr/lib/bluetooth/bluetoot        0     1384     1421     4928
  718 avahi    avahi-daemon: running [macm        0      776     1020     4140
  715 root     /usr/sbin/acpid                    0      228      243     1928
  714 root     /usr/lib/accountsservice/ac        0     1688     1763     8032
  681 systemd-timesync /lib/systemd/systemd-timesy        0      900     1329     7396
  327 root     /lib/systemd/systemd-journa        0    39860    45003    55588
    1 root     /sbin/init                         0     1836     3863    11888

I dont see anything odd here on the swap front. In your minimum load example, there is likely sufficient physical memory that swap is just not required. With the PR patch, the number of records being added to the recorder db at the time of the purge should not make a difference. It is more about the overall number of records already in the db that purge needs to sort through that drives the memory usage.

I agree that the postgres core dumps look odd. You could try looking at the systemd logs sudo journalctl -u postgres to see when they are happening - e.g. if concurrent with the purges - and also any surrounding context.

Thanks for thoughts on this @PeteBa !

I got another distinct bump in swap usage again at 4:12 am, see picture below. The amount of records deleted by purge was zero, as I ran the purge service manually earlier, the HA log shows it ran but with zero records deleted.

No errors generated by postgresql, I did a little research on the way smem displays postgresql, 'systemd-coredump postgres:'. I believe this not an indicator of any crash on the part of postgresql, it is just that postgresql launches all of it's processes wrapped in the systemd-coredump in order to catch dumps. This might have been configured based on the docker image I selected for my postgresql install.

Below is a dump of the output of swap space command you supplied as of this morning. Unfortunately, it appears to show that many processes have allocated swap space since the system was booted. I need to find a way to show 'deltas' in swap space growth over time and see if I can see if there is a particular process that is growing 'abby normal'. I am going to put your swap spaced analyzer in a cron job to capture the stats right before the 4:12 am purge service run and right after. Then see if I can extract some useful data over a couple evolutions.

user@macmini2012:~$  echo "NAME PID SWAP"; for file in /proc/*/status ; do awk '/^Pid|VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | grep kB | grep -wv "0 kB" | sort -k 3 -n -r
NAME PID SWAP
gnome-shell 1320 73628 kB
node 11455 26328 kB
python3 5143 24140 kB
npm 12063 16504 kB
postgres 8918 14832 kB
dockerd 1508 12372 kB
containerd 831 12024 kB
ibus-extension- 1369 9440 kB
postgres 8919 9392 kB
jupyter-lab 12281 6396 kB
tracker-miner-f 1048 6076 kB
Xorg 1065 5340 kB
goa-daemon 1409 5312 kB
pulseaudio 1046 4332 kB
evolution-calen 1413 3332 kB
evolution-addre 1447 3156 kB
snapd 758 3092 kB
gnome-shell-cal 1392 2976 kB
evolution-sourc 1401 2632 kB
networkd-dispat 740 2472 kB
unattended-upgr 911 2432 kB
gjs 1612 1924 kB
python 10546 1888 kB
gsd-datetime 1646 1844 kB
ibus-x11 1371 1840 kB
postgres 5618 1776 kB
postgres 6683 1672 kB
gnome-session-b 1306 1668 kB
python 11842 1536 kB
postgres 6678 1528 kB
postgres 4438 1428 kB
postgres 4440 1408 kB
python 11748 1368 kB
gsd-color 1645 1288 kB
postgres 4437 1252 kB
gsd-power 1655 1204 kB
postgres 4368 1196 kB
postgres 4439 1180 kB
postgres 4436 1136 kB
postgres 4435 1128 kB
gsd-keyboard 1650 1116 kB
(sd-pam) 1038 1100 kB
ibus-daemon 1364 1096 kB
goa-identity-se 1427 1084 kB
appdaemon 11917 980 kB
postgres 157032 924 kB
gvfsd-fuse 1087 908 kB
systemd 1037 884 kB
ibus-dconf 1368 812 kB
systemd-udevd 354 808 kB
grafana-server 10782 796 kB
at-spi-bus-laun 1267 792 kB
gvfsd-trash 1624 788 kB
at-spi2-registr 1386 756 kB
systemd-resolve 679 752 kB
colord 1795 700 kB
geoclue 1424 688 kB
portainer 2371 684 kB
gvfs-udisks2-vo 1093 684 kB
python 11659 656 kB
gvfs-afc-volume 1589 656 kB
xdg-permission- 1390 600 kB
python3 13361 596 kB
gsd-a11y-settin 1644 588 kB
gvfs-gphoto2-vo 1585 556 kB
gdm-x-session 1060 524 kB
gvfsd 1072 492 kB
systemd 1 472 kB
smbd-notifyd 1568 464 kB
cleanupd 1569 464 kB
gnome-session-c 1299 452 kB
smbd 1562 428 kB
lpqd 1571 428 kB
gnome-keyring-d 1053 420 kB
ibus-portal 1374 404 kB
gsd-wacom 1664 392 kB
gsd-media-keys 1652 384 kB
python 11364 376 kB
gsd-housekeepin 1648 340 kB
NetworkManager 726 320 kB
gvfs-mtp-volume 1575 320 kB
polkitd 747 316 kB
smbd 202685 308 kB
gvfs-goa-volume 1581 308 kB
gdm3 870 288 kB
bash 3273 276 kB
bash 3272 276 kB
bash 3271 276 kB
dbus-daemon 1273 260 kB
gsd-wwan 1665 256 kB
dconf-service 1437 240 kB
lighttpd 3285 220 kB
udisksd 766 216 kB
hack_hdhomerun. 11330 216 kB
dbus-daemon 1054 216 kB
gsd-print-notif 1656 204 kB
python 11537 192 kB
usbhid-ups 13377 172 kB
cron 11422 168 kB
rsyslogd 753 128 kB
dbus-daemon 724 128 kB
cron 720 116 kB
upowerd 1410 100 kB
ModemManager 835 100 kB
cron 3286 96 kB
sh 11421 92 kB
smartd 755 88 kB
sh 11431 80 kB
nmbd 1516 72 kB
mosquitto 4254 72 kB
tini 12242 60 kB
sshd 859 56 kB
cupsd 35613 52 kB
tail 11423 44 kB
canonical-livep 816 40 kB
upsd 13379 36 kB
gdm-session-wor 912 32 kB
docker-init 13324 32 kB
docker-init 11644 32 kB
docker-init 11315 32 kB
accounts-daemon 714 32 kB
docker-init 11734 28 kB
docker-init 11523 28 kB
docker-init 10927 24 kB
docker-init 10533 24 kB
s6-svscan 2370 20 kB
s6-svscan 10943 20 kB
kerneloops 1524 20 kB
systemd-timesyn 681 16 kB
s6-svscan 4898 16 kB
s6-supervise 10969 16 kB
s6-supervise 10968 16 kB
avahi-daemon 769 16 kB
thermald 765 12 kB
sshd 158070 12 kB
s6-supervise 5140 12 kB
s6-supervise 4953 12 kB
s6-supervise 3267 12 kB
s6-supervise 3266 12 kB
s6-supervise 2497 12 kB
s6-supervise 11419 12 kB
sshd 158166 8 kB
s6-supervise 3269 8 kB
s6-supervise 11417 8 kB
s6-supervise 11416 8 kB
pihole-FTL 3289 8 kB
boltd 837 8 kB
switcheroo-cont 763 4 kB

Hello @PeteBa

The code changes you did to the Home Assistant purge service continues to function as before, no longer continuously purging records from the events and states tables. I would call that problem resolved.

The older problem of the continuously growth of the swap space continues. I am collecting more data over the next few nights, however as is shown below, it does appear that the core Home Assistant program has a large increase in it's swap memory each time the purge function runs. HA does seem to grow it's use of memory over time in general, however the growth spurt that current occurs during the purge is on the order of 100 times this regular growth. There does seems to be some swap usage by the postgresql processes related to Home Assistant, but these do not seem as significant, and perhaps do go away after time. I need to collect more data.

I have added a picture of the swap space usage over time by the "python3 -m homeassistant" process and also by the prosgresql processes. The significant growth can been seen during the 4:15 am purge run time. The last column is the change in size hour to hour.

I also included a link below to a modified version of the capture script you wrote, some sample data and a small jupyter notebook I used to look at the data.

Happy to close this bug report. However, I would like to see if there is something that can be done to stop this swap memory growth by Home Assistant, it has been around since 4th quarter of 2020 on my system and does significantly reduce the length of time HA can run without restart. Prior to starting to see this issue, it was not uncommon to run HA for 3 to 4 months, now a month at most. Please let me know your thoughts on how to proceed.

https://github.com/deepcoder/data_science_public/tree/main/linux_swap_space

Unfortunately, I cant see anything definitive here. I can understand, if physical memory is tight on your server, then the HA purge service may cause the OS to swap some things out to disk. What I am unsure of is when that swap would be re-loaded into memory. I suspect that is dependent on the management strategy used by the os and might only play out over time.

To be honest, I think you may get more insight by looking at the overall memory usage pattern of HA over time. Swap is just one element and if there were a "leak" then it would be more visible looking at the overall memory usage (e.g. Swap + PSS/RSS from the smem command). It also might be interesting to run multiple manual purges to see if it grows in an unbounded and detrimental way.

Sorry I cant be more helpful - getting beyond my knowledge - but hopefully some ideas that might generate insight. If you want to close this issue and move the conversation to the forum thread referenced above then happy to keep involved.

Hello @PeteBa , Again thank you for your work on fixing the issue of this bug report, very much appreciated!

I will close the ticket now.

As for this vexing issue of the swap space growing, I will try and do some more research and see if there in any other folks with experience over on the forum. I not sure what existing discussion there is that you spoke of.

There is something rather odd going on at 4:12 am or on this machine and the purge service seems to be the only common item. The machine has 16 GB of RAM and never seems to above 8 GB used, even at 4:12 am when the swap space bump occurs, see free memory graph below. The machine has a number of processes running on it, in docker and native, none of these exhibit the same behavior that I can see.

Again thanks for all your work on Home Assistant!

Issue resolved : Recorder purging records continuously #51689

Thanks!

home-assistant / core