improve ingest-core audit logging

[amnonkhen]

It is currently difficult to understand what is going in ingest-core. When we notice some problems in the operation of ingest-core such as:

• errors in the logs
• increased memory consumption we have no way of investigating and figuring out which usage scenarios cause these problems.

We need to add more information to the log so that it includes:

• [ ] http access logs, i.e. originating ip, url, https status, duration
• [ ] ingest-core flow calls, i.e. controller > service > repo - add request identifier to track log messages related to the same request (session id)
• [ ] track project/submission - add relevant ids to log messages

once the data is in the logs, it would be useful to have a dashboard that shows

• [ ] api call duration trend
• [ ] error counts - server and user
• [ ] api call https status daily trend
• [ ] originating ip distribution, new vs old, geography

possible tools/technologies:

spring boot logs http access to stdout

pros:

• no 3rd party to add
• can be implemented today, easy to implement cons:
• major affect on logs size
• duration not logged
• output in "access log format", need parsing in loki/graphana

spring actuator -> prometheus

Have prometheus poll the spring actuator and extract http requests traceinfo. See example pros: http request duration is there output is structured

cons:

• new 3rd party dependencies

[amnonkhen]

@jacobwindsor please review & revise as needed @clairerye what do you think? please prioritise this

[amnonkhen]

@jacobwindsor I remember we discussed the need for this. Can you please update what this was related to?