tomcrane
commented
1 year ago After starting work on this I'm inclined to leave this issue alone. The data is more useful than I thought, and:
There's a problem with the second para above; if you switched this on and re-ran a sync operation, you would often not get any new request and response bodies to log because the DDS wouldn't have anything to sync with the DLCS. It doesn't create sync batches unless there are things to change. So you'd still need to see the original exchange with the DLCS, which is lost.
A better approach would be to do some annual maintenance and, say, remove DLCS API body data older than 1 year (or whatever).
We started with clean databases when we moved this to PostgreSQL in the cloud 2 yrs ago. These two instances are not huge:
SELECT pg_size_pretty(pg_database_size('ddsinstrumentation'))
571 MB
SELECT pg_size_pretty(pg_database_size('dds'))
709 MB
Currently, the request and response bodies between DDS and DLCS are stored in the dds_instrumentation database and can be viewed via the dashboard. While that's nice, it's a lot of storage (verbose JSON payloads) for only very rare utility.
DDS should not store these by default, but we can add a flag that will store them when a job is triggered from the dashboard (rather than background processes). This means they still have their diagnostic usefulness, but you have to re-run the job from the dashboard to see the payloads.