Script executes pg_archivecleanup without a file specified

[Organizer21]

I have an archive folder with files from the 28, 29th and 30th (today)

Running bash walarchivecleanup.sh -p /mnt/archivedir -a 2 -d gave me:

pg_archivecleanup -d /mnt/archivedir
pg_archivecleanup: must specify oldest kept WAL file
Try "pg_archivecleanup --help" for more information

Expected result was for it to remove any files from the 28 older than the current time (presume it takes the clock into account as well?) but it seems to have found none. Plus should there really be nothing to delete, should it not then exit without trying doing anything?

Running bash walarchivecleanup.sh -p /mnt/archivedir -a 1 @ 1.30pm resulted in the script being called with a file which date indicates 28th @ 1.33pm. Which at least means the script works on my system, though why 28th? Leaving anything older than 1 day would for me mean anything older than 29th @ 1.3xpm should have been included?

My Linux knowledge is on a beginner level so I'm likely missing something really obvious. The server is running Ubuntu 18:04 for reference.

[Napsty]

To verify what kind of files are actually found by the script, you can execute find /mnt/archivedir/ -maxdepth 1 -type f -not -name '*.ready' -not -name '*.backup' -mtime +2 Basically if you use -a 2 tells the script look only for files OLDER than 2 days (2x24h).

[Organizer21]

Noted, just did that now (2019-07-31 09:35 BST) with a +1 which shows me only files that older than 2019-07-29 09:35 BST (as in 48+ hours ago). As to my confusion and likely just some dumb thing I am doing, the result of +1 should clearly have been 1 day and older and not +2 as seen.

[Napsty]

Can you show the output of find /mnt/archivedir/ -maxdepth 1 -type f -not -name '*.ready' -not -name '*.backup' -mtime +1 -exec ls -la {} + ?

[Organizer21]

Hmmm, just found this which might explain the strange behaviour. https://unix.stackexchange.com/questions/92346/why-does-find-mtime-1-only-return-files-older-than-2-days

[Organizer21]

Though per previous update I think we now know the reason. Not sure converting to minutes and using "mmin" in the script or adding a that as an extra option is the better way;

Can you show the output of find /mnt/archivedir/ -maxdepth 1 -type f -not -name '*.ready' -not -name '*.backup' -mtime +1 -exec ls -la {} + ?

root@backup:/mnt# find /mnt/archivedir/ -maxdepth 1 -type f -not -name '*.ready' -not -name '*.backup' -mtime +1 -exec ls -la {} +
-rw------- 1 postgres postgres 16777216 Jul 29 08:20 /mnt/archivedir/00000001000002C200000004
-rw------- 1 postgres postgres 16777216 Jul 29 08:21 /mnt/archivedir/00000001000002C200000005
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C200000006
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C200000007
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C200000008
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C200000009
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C20000000A
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C20000000B
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C20000000C
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C20000000D
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C20000000E
-rw------- 1 postgres postgres 16777216 Jul 29 08:22 /mnt/archivedir/00000001000002C20000000F
-rw------- 1 postgres postgres 16777216 Jul 29 08:23 /mnt/archivedir/00000001000002C200000010
-rw------- 1 postgres postgres 16777216 Jul 29 08:25 /mnt/archivedir/00000001000002C200000011
-rw------- 1 postgres postgres 16777216 Jul 29 08:26 /mnt/archivedir/00000001000002C200000012
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C200000013
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C200000014
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C200000015
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C200000016
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C200000017
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C200000018
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C200000019
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C20000001A
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C20000001B
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C20000001C
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C20000001D
-rw------- 1 postgres postgres 16777216 Jul 29 08:27 /mnt/archivedir/00000001000002C20000001E
-rw------- 1 postgres postgres 16777216 Jul 29 08:28 /mnt/archivedir/00000001000002C20000001F
-rw------- 1 postgres postgres 16777216 Jul 29 08:29 /mnt/archivedir/00000001000002C200000020
-rw------- 1 postgres postgres 16777216 Jul 29 08:29 /mnt/archivedir/00000001000002C200000021
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C200000022
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C200000023
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C200000024
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C200000025
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C200000026
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C200000027
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C200000028
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C200000029
-rw------- 1 postgres postgres 16777216 Jul 29 08:31 /mnt/archivedir/00000001000002C20000002A
-rw------- 1 postgres postgres 16777216 Jul 29 08:32 /mnt/archivedir/00000001000002C20000002B
-rw------- 1 postgres postgres 16777216 Jul 29 08:32 /mnt/archivedir/00000001000002C20000002C
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C20000002D
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C20000002E
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C20000002F
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C200000030
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C200000031
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C200000032
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C200000033
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C200000034
-rw------- 1 postgres postgres 16777216 Jul 29 08:33 /mnt/archivedir/00000001000002C200000035
-rw------- 1 postgres postgres 16777216 Jul 29 08:35 /mnt/archivedir/00000001000002C200000036
-rw------- 1 postgres postgres 16777216 Jul 29 08:37 /mnt/archivedir/00000001000002C200000037
-rw------- 1 postgres postgres 16777216 Jul 29 08:38 /mnt/archivedir/00000001000002C200000038
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C200000039
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C20000003A
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C20000003B
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C20000003C
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C20000003D
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C20000003E
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C20000003F
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C200000040
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C200000041
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C200000042
-rw------- 1 postgres postgres 16777216 Jul 29 08:39 /mnt/archivedir/00000001000002C200000043
-rw------- 1 postgres postgres 16777216 Jul 29 08:40 /mnt/archivedir/00000001000002C200000044
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000045
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000046
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000047
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000048
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000049
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C20000004A
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C20000004B
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C20000004C
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C20000004D
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C20000004E
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C20000004F
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000050
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000051
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000052
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000053
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000054
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000055
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000056
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000057
-rw------- 1 postgres postgres 16777216 Jul 29 08:41 /mnt/archivedir/00000001000002C200000058
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000059
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C20000005A
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C20000005B
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C20000005C
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C20000005D
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C20000005E
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C20000005F
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000060
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000061
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000062
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000063
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000064
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000065
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000066
-rw------- 1 postgres postgres 16777216 Jul 29 08:42 /mnt/archivedir/00000001000002C200000067
-rw------- 1 postgres postgres 16777216 Jul 29 08:43 /mnt/archivedir/00000001000002C200000068
-rw------- 1 postgres postgres 16777216 Jul 29 08:43 /mnt/archivedir/00000001000002C200000069
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C20000006A
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C20000006B
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C20000006C
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C20000006D
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C20000006E
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C20000006F
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C200000070
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C200000071
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C200000072
-rw------- 1 postgres postgres 16777216 Jul 29 08:44 /mnt/archivedir/00000001000002C200000073
-rw------- 1 postgres postgres 16777216 Jul 29 08:46 /mnt/archivedir/00000001000002C200000074
-rw------- 1 postgres postgres 16777216 Jul 29 08:47 /mnt/archivedir/00000001000002C200000075
-rw------- 1 postgres postgres 16777216 Jul 29 08:47 /mnt/archivedir/00000001000002C200000076
-rw------- 1 postgres postgres 16777216 Jul 29 08:47 /mnt/archivedir/00000001000002C200000077
-rw------- 1 postgres postgres 16777216 Jul 29 08:47 /mnt/archivedir/00000001000002C200000078
-rw------- 1 postgres postgres 16777216 Jul 29 08:47 /mnt/archivedir/00000001000002C200000079
-rw------- 1 postgres postgres 16777216 Jul 29 08:47 /mnt/archivedir/00000001000002C20000007A
-rw------- 1 postgres postgres 16777216 Jul 29 08:47 /mnt/archivedir/00000001000002C20000007B
-rw------- 1 postgres postgres 16777216 Jul 29 08:47 /mnt/archivedir/00000001000002C20000007C
-rw------- 1 postgres postgres 16777216 Jul 29 08:48 /mnt/archivedir/00000001000002C20000007D
-rw------- 1 postgres postgres 16777216 Jul 29 08:48 /mnt/archivedir/00000001000002C20000007E
-rw------- 1 postgres postgres 16777216 Jul 29 08:50 /mnt/archivedir/00000001000002C20000007F
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000080
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000081
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000082
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000083
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000084
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000085
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000086
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000087
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000088
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C200000089
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C20000008A
-rw------- 1 postgres postgres 16777216 Jul 29 08:51 /mnt/archivedir/00000001000002C20000008B
-rw------- 1 postgres postgres 16777216 Jul 29 08:52 /mnt/archivedir/00000001000002C20000008C
-rw------- 1 postgres postgres 16777216 Jul 29 08:53 /mnt/archivedir/00000001000002C20000008D
-rw------- 1 postgres postgres 16777216 Jul 29 08:54 /mnt/archivedir/00000001000002C20000008E
-rw------- 1 postgres postgres 16777216 Jul 29 08:54 /mnt/archivedir/00000001000002C20000008F
-rw------- 1 postgres postgres 16777216 Jul 29 08:54 /mnt/archivedir/00000001000002C200000090
-rw------- 1 postgres postgres 16777216 Jul 29 08:54 /mnt/archivedir/00000001000002C200000091
-rw------- 1 postgres postgres 16777216 Jul 29 08:54 /mnt/archivedir/00000001000002C200000092
-rw------- 1 postgres postgres 16777216 Jul 29 08:55 /mnt/archivedir/00000001000002C200000093
-rw------- 1 postgres postgres 16777216 Jul 29 08:55 /mnt/archivedir/00000001000002C200000094
root@backup:/mnt#

[Napsty]

Thx for the link and the output. This probably explains why files betwee 24h-48h don't show up. So I guess you have files with modification date Jul 30 and (meanwhile) Jul 31, right?

[Napsty]

What about find /mnt/archivedir/ -maxdepth 1 -type f -not -name '*.ready' -not -name '*.backup' -mtime +0 -exec ls -la {} + as workaround? What does this give you?

[Organizer21]

Correct, I got about 7000 files with dates between 29 9am and 30th 9am + then another so far 5000 from 30th 9am till today and the current time.

[Organizer21]

What about find /mnt/archivedir/ -maxdepth 1 -type f -not -name '*.ready' -not -name '*.backup' -mtime +0 -exec ls -la {} + as workaround? What does this give you?

At glance it gave me all files until this one, which would be a workaround; -rw------- 1 postgres postgres 16777216 Jul 30 09:25 /mnt/archivedir/00000001000002E1000000D0

[Napsty]

So you could use -a 0 for the script as a workaround. But do you really want to remove the WAL files age>24h ? What if something goes wrong on a Friday evening and on Monday you need to re-run the WAL logs for reason x?

Concerning the -a parameter i will change the documentation a bit, so we understand this parameter as "age to keep, older delete". E.g. -a 1 means keep local WAL logs at least one day (24-48h), delete logs older than that.

[Napsty]

https://github.com/Napsty/scripts/pull/5

[baraalfa]

walarchivecleanup.sh: 34: walarchivecleanup.sh: Syntax error: word unexpected (expecting "do")

[Organizer21]

@Napsty I would happily have kept more data if I had more space (but that's not the case currently at least). That said my streaming standby or whatever they call this is technically up to speed all the time, so if the main server crashes on a Friday there won't be more WALs generated anyway, but yes if something critical happens that would need a backup restoration + some WAL files, only having 24 hours might be a bit limiting (though I hope that scenario is really rare in my case). Closing this case and thank you for all the help last year :) still using the script in other words.