update ttl enforcer class with more efficient shell command, add a test

[lkemmerer]

The comment that Lucas made about the new command not cleaning up the empty directories is a concern that the quick fix we had done in the past (including the current prod cron job) doesn't address but we probably should. There are a bunch of empty read spoke FinalizedFlightSnapshot directories hanging out because of that. I don't know what the best way to clean that up would be, but it is something we need to make sure we do.

The other thing that we need to do is to modify the logic in handleCleanup in the Spoke TTL enforcer so that it uses the else path for the read spoke. Keeping it the way that it is now will continue to ignore anything that doesn't fall into the very specific timeframes that the if path looks at, which doesn't clean up most of historical data that gets stored in the read cache.

The read path is the main thing that needs optimizing and changing. The write cleanup works pretty well, if maybe a little slowly (maybe 5 minutes each run), but the TTL enforcer for the read cache was taking 15+ minutes to run and not actually getting all of the files cleaned up. (See the Spoke TTL enforcer runtime for the past 3 months on iad.prod in Datadog for more info)

[lucasmo]

I believe find <dir> -type d -empty will find empty directories.

[lucasmo]

~$ mkdir foo && cd foo
~/foo$ find . -type d -empty -depth +0
~/foo$ mkdir bar
~/foo$ find . -type d -empty -depth +0
./bar
~/foo$ mkdir bar/baz
~/foo$ find . -type d -empty -depth +0
./bar/baz
~/foo$ touch bar/baz/boo
~/foo$ find . -type d -empty -depth +0
~/foo$

[Paul-Hess]

~$ mkdir foo && cd foo
~/foo$ find . -type d -empty -depth +0
~/foo$ mkdir bar
~/foo$ find . -type d -empty -depth +0
./bar
~/foo$ mkdir bar/baz
~/foo$ find . -type d -empty -depth +0
./bar/baz
~/foo$ touch bar/baz/boo
~/foo$ find . -type d -empty -depth +0
~/foo$

I assume you ran that on mac? I think the find on ubuntu is a little different I went with find $directory -depth -type d -empty -not -path "lost+found" -print -delete I am not sure that -depth is required there because I believe it uses that behavior by default with the -delete flag

[lucasmo]

I assume you ran that on mac? I think the find on ubuntu is a little different I went with find $directory -depth -type d -empty -not -path "lost+found" -print -delete I am not sure that -depth is required there because I believe it uses that behavior by default with the -delete flag

Yeah I ran it on mac. If you don't specify depth of >0 it will include . if the root directory is empty, which is probably undesired behavior.

[Paul-Hess]

I assume you ran that on mac? I think the find on ubuntu is a little different I went with find $directory -depth -type d -empty -not -path "lost+found" -print -delete I am not sure that -depth is required there because I believe it uses that behavior by default with the -delete flag

Yeah I ran it on mac. If you don't specify depth of >0 it will include . if the root directory is empty, which is probably undesired behavior.

ah yes, added mindepth 1 flag which is the equivalent on ubuntu