Mobility writer has high CPU usage + several bugs which might lead to data loss

[Piiit]

The high cpu is mainly due to ORM and object creation, de/serialization issues...

The intermediate solution before we migrate to the new infrastructure is to install it twice (or even more often, if necessary) and have a load balancer in front.

The problem is, that the logging is minimal and no alerting / monitoring is in place. I need to create something small and fast for this job, otherwise I would have to fly blindly...

The whole thing is not little work, seeing already the end of this tool near... I will therefore first try to improve the writer itself for a short time, as follows:

• [x] migrate ci / cd to github actions (for a faster feedback cycle)
• [x] fix the push record method or a relevant related method a little (just enough to reduce some memory footprint)
• [x] add some minimal direct logging to the console for manual testing

... and if that is not enough (CPU still above 90%), go with the following minimal tasks:

• [x] Add a simple logging to see if something goes wrong
• [x] Fix the n+1 issue in stations fetching, which retrieves all stations but fetches then each metadata record one by one
• [x] Fix the poor performance for latest record timestamp, which fetches all measurement tables in a 3 queries and combines the result in Java (not on the DB)
• [x] Update dependencies, maybe a newer hibernate or geotools stack has better performance
• [x] Let spring boot manage transactions (maybe it is better to not always open and close connections)... hopefully it improves connection pooling
• [x] add http request parameters to the logging output + split http request paths
• [x] reduce provenance UUID calls, should just be fetched ones per data collector execution
• [x] heartbeat checks
• [x] add origin to each call, similar as it is for web components now
• [x] fix stacktrace mimetype, should be plain text not json
• [x] fix noi-techpark/bdp-core#137
• [x] fix wrong latest record time
• [x] Add a Grafana dashboard / alerting on top of that logging output

... if all those efforts above are not enough or if kibana reveals issues:

• [x] fix huge push record calls
• [x] fix huge sync station calls
• [x] fix wrong station state updates
• [x] fix station state multiple calls if syncStation gets split into chunks
• [x] fix varchar(n) vs text on string measurements (some inserts are too long, causing transaction aborts)
• [x] fix a concurrency issue while inserting strings vs numbers in transactions (switch to application based entity managers)
• [x] fix Hibernate Lob recognition of "postgres/text" to String conversion
• [x] fix duplicate key violations, check if other measurements of that transaction get inserted

  Duplicates in POST payloads resulted in a complete loss of that measurement chunk, same for duplicate station syncs

Moved to new issues

• noi-techpark/bdp-core#250
• noi-techpark/bdp-commons#511
  • (See Burndown)

    
    NOT NEEDED ANYMORE... CPU ISSUE SEEMS SOLVED WITH RECENT CHANGES

• [ ] Install the writer a second time
• [ ] Introduce a AWS load balancer OR just have two endpoints and configure some DC for the second one

[Piiit]

@rcavaliere Finally with the new logging capabilities (now on production), I found one culprit:

The echarging plugs are a huge push to the writer, that is, every 5 minutes we push something that takes 30-60 minutes to get written to the DB.

[Piiit]

@dulvui @rcavaliere Finally done... now lets check DCs