Create a better health check

[c2h5oh]

checking if go process is alive is not enough

[c2h5oh]

Sample problem - found in tmp:

-rw-------  1 root root     128M Jun 26 13:27 magick-1zlyN4ZF1LM3m
-rw-------  1 root root     1.1M Jun 17 15:08 magick-1zm6OayAws4D5
-rw-------  1 root root     1.9M Jun 26 14:03 magick-1zmyZjHd0pUFr
-rw-------  1 root root     1.9M Jun 26 13:30 magick-1znI785VXAi6T
-rw-------  1 root root     1.9M Jun 26 13:30 magick-1znVSqmSOo6GM
-rw-------  1 root root     1.9M Jun 26 14:10 magick-1znZaBlJPU1qe
-rw-------  1 root root     1.3M Jun 18 06:27 magick-1zoeJjC5I9xi6
-rw-------  1 root root     1.9M Jun 26 13:32 magick-1zoeY4dTFl9NS
-rw-------  1 root root     2.4M Jun 26 14:06 magick-1zpfitqvK30dp
-rw-------  1 root root     2.4M Jun 26 14:07 magick-1zqXQkiXtSy3A
-rw-------  1 root root     2.4M Jun 26 13:40 magick-1zqXV3nuFLt4M
-rw-------  1 root root     6.9M Jun 25 17:31 magick-1zqjv_4jo4znQ
-rw-------  1 root root     128M Jun 25 19:35 magick-1zqsfe_F6Tlsi
-rw-------  1 root root     1.1M Jun 26 00:40 magick-1zs3QSNdUXYBp
-rw-------  1 root root     2.4M Jun 26 13:59 magick-1zsRn4ZYWUqVO
-rw-------  1 root root     1.9M Jun 26 13:39 magick-1zsinfLDzcSHY
-rw-------  1 root root     1.1M Jun 17 18:14 magick-1zsz9ZYdeuqKq
-rw-------  1 root root     128M Jun 26 13:28 magick-1ztpsgqpRiWZk
-rw-------  1 root root     1.9M Jun 26 13:30 magick-1zuYbzuyu3DM7
-rw-------  1 root root     3.8M Jun 26 06:32 magick-1zuhPTMYv4clT
-rw-------  1 root root     2.4M Jun 26 14:10 magick-1zunvFg7rs41B
-rw-------  1 root root     1.9M Jun 26 13:37 magick-1zva_I5z4wyGi
-rw-------  1 root root     2.4M Jun 26 13:41 magick-1zwZAQIao9f6M
-rw-------  1 root root     1.1M Jun 25 20:55 magick-1zxM1tstR0pHm
-rw-------  1 root root     1.9M Jun 26 13:45 magick-1zxb8lQNZLFrS
-rw-------  1 root root     2.4M Jun 26 13:40 magick-1zxbHuEv4bsDD
-rw-------  1 root root     1.9M Jun 26 13:37 magick-1zxbOXgZ5pUGA
-rw-------  1 root root     1.9M Jun 26 14:07 magick-1zxmD-tkqiJ92
-rw-------  1 root root     1.3M Jun 21 18:24 magick-1zyZSFsKdfDyF
-rw-------  1 root root     128M Jun 26 14:29 magick-1zyqBzTsFUOv4
-rw-------  1 root root     1.9M Jun 26 14:15 magick-1zzA5iJNJSLDK
-rw-------  1 root root     1.9M Jun 26 13:46 magick-1zzFkCeN41DBu
-rw-------  1 root root     2.4M Jun 26 13:29 magick-1zzRUwG7NXsBK

[VojtechVitek]

[pkieltyka]

interesting.. what is taking up the disk space? is gographics/imagick not cleaning up after itself (in /tmp) while its active, or is it if it crashes for whatever reason, its garbage is left over (I think this is probably the case). A bunch of crashes throughout the day would certainly consume all of the disk.

As a first measure, we should check if gographics/imagick has a method to give us a path to the temproary directory and its filename prefix, then we can sweep the temp directory when the process starts to clean this up if it restarts by crashing.

[VojtechVitek]

We don't know yet... but it was happening periodically. Imagick was leaving TONS OF temporary files in /tmp, see above. Most of the files were rather small, but some of them were up to 128M.

[pkieltyka]

the 128M files are very weird... but, you will see files in there while imgry is running and resizes are happening. If you cd /tmp and ls -la every second, you will see a new set of files constantly. However, for the most part, imagick should be doing all of this in memory instead of temporary files, there hits a point where the imagemagick api will start using temporary files. One idea would be to mount a ram disk and tell imagick to use that directory for its temporary files. It should speed things up too.

[VojtechVitek]

We can try mounting ramfs to container's /tmp. But we need to make sure that Imagick cleans up after itself. Seems like it doesn't right now.

[c2h5oh]

It's not as crazy as it sounds - animated gif gets decompressed into a set of 24 bits per pixel bitmaps before processing: 5 second gif, 15 frames per second, 800 x 600 px: 5 * 15 * 800 * 600 * 24 / 8 = 108 000 000B = 103MB and that's just the buffer to load it. you'll need at least one more to process it (sometimes more)