Attempt to stabilize the ingest pipeline

[htpvu]

work on this ticket encapsulate the effort made to stabilize the ingest pipeline which has been issue prone and fail often.

• [x] Capture ActiveMQ messages to a DLQ (merged)

• [ ] Troubleshoot and address the resource requirements by:

• • [x] increasing the resources available to the container and
• • [x] updating the Houdini PHP code to delete the large temporary files being left behind in the filesystem. (merged)
• • [x] craft an Imagemagick policy file that limits the resources used by Imagemagick. (in progress) Bethany will ensure that imagemagick is paying attention to this file and that it’s useful.

• [ ] Replacing PHP Houdini with a Golang rewrite to see if imagemagick is the issue itself (evaluate 1-2c and see if this is necessary)

• [ ] switch from imagemagick to libvips. (evaluate 1-2d and see if this is necessary)(deferred)

• • [x] Go from the result of a/b/c/d?

• [x] Enable observability into activeMQ DLQ for developers (merged)

• • [x] see also the Alpaca PR (pending review and merge)

this card can be closed when the changes described above are deployed to TEST

[emetsger]

Release v0.6.0 has been made which addresses some of the tasks in the issue. After it's deployed, we can test ingest with problematic migrations to see if stability is improved.

https://github.com/jhu-idc/idc-isle-dc/releases/tag/v0.6.0