darrylmendillo
commented
4 years ago Update
Stephen Kratowicz reported an HL7 performance issue from 0745 - late afternoon, lining up with our dropouts.
Cause
- HL7 performance dropped from 10s/insert - 20m/insert
- This may have caused DB blocking
Next Steps:
Compare expected messages sent from App Dev over one day with messages observed by DS
Observation:
Grafana triggered an alert from missing ventnotifymssql.micro for 15 minutes.
Troubleshooting:
Investigation of logs from the nomad ui confirmed the accuracy of grafana logs. Initial observations showed that all microservices were working as intended, with reduced upstream data. Vent microservice nomad job was restarted, with results remaining the same after restart.
Conclusion:
Upstream is reduced and was identified as the root cause.
Next Steps:
Contact Rutvij from app dev to troubleshoot upstream issues.