search

scandihealth / lpr3-docs

https://scandihealth.github.io/lpr3-docs/
MIT License
11 stars 7 forks source link

Errors: Failed to Invoke Service Endpoint #308

Closed Marc-Petersen closed 5 years ago

Marc-Petersen commented 5 years ago

We're currently logging a lot of communication errors when submitting CDAs to SDS endpoint, indicating failure to reach endpoint. This is not for all submission, but a lot.

Also, the submission rate is very low. About 200-300 msgs/hour, where we normally reach 100.000-200.000 msgs/hour.

I'll post further, if I see a pattern in the failed submissions

TueCN commented 5 years ago

Hi, I assume your issue is regarding PROD. In the future please state what environment the issue applies to (preferably in the title of the issue).

We're currently logging a lot of communication errors when submitting CDAs

What errors? Can you be more specific? SOAP faults? Time outs?
We see a lot of errors in our log today regarding Medcom headeren mangler. (the medcom header element is not found in the SOAP envelope).
Can you verify that the requests that got rejected did in fact contain a medcom:Header XML child element in the SOAP Header XML element?

Also, the submission rate is very low. About 200-300 msgs/hour

We have identified an issue that can slowly degrade performance over time, until servers are restarted. We restarted one of the failing servers this morning and are in the process of testing a fix and will include the fix in the next hotfix.

Are you still experiencing throughput issues currently? (after 10 today)

kdjohansson commented 5 years ago

@TueCN Is this also currently affecting performance in TEST?

TueCN commented 5 years ago

@kdjohansson in theory it could, but due to the much much smaller workload the TEST system is under, it is much less likely to happen.

From what I can tell, TEST is currently operating normally.

Marc-Petersen commented 5 years ago

Yes. PROD env. Currently throughput is normal again. The errors also seems to be gone. Some time after 10 am.

They were CommError: WEBSERVICEINVOCATIONFAILURE and errors indicating missing ACKs. And multiple other errors following. We have no indications of missing MedCom headers.

TueCN commented 5 years ago

RESOLUTION Fixed a bug that could lead to performance degradation over time when the service is receiving a high amount of requests concurrently.

You may experience intermittent errors/timeouts on some web service requests while we restart the LPR3 web service servers daily, until a hotfix is deployed (which should happen one of the coming days)

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.