search

nextgenhealthcare / connect

The swiss army knife of healthcare integration.
Other
868 stars 265 forks source link

[BUG] Loading statistics and deploying channels slow on service restart #6186

Closed thorst closed 2 months ago

thorst commented 2 months ago

Describe the bug I have a python script that will clear the statistics, undeploy the channels and then restart the services. When it comes back up it will hang on the loading statistic screen for an unacceptable amount of time (more than 5 minutes) before it finally loads.

To Reproduce I was told previously this was because of statistics, so I added the step to clear current and lifetime stats. I think the issue is that we have 300k pending transactions (the vendor was hacked, so we are just queueing). We use postgres so the database should be able to handle a lot of pending transaction (in my head anyway).

We have 305 channels deployed, they get deployed on system startup. Heap, ram, disk, everything looks perfect.

Running on redhat.

Expected behavior I believe that logging in, deploying channels, and rendering stats should happen quicker, I'm not sure why it is taking so long. Only thing that I know we COULD be doing is that we set channel writers dependent on channel readers being deployed. We dont do that currently, but Ive started discussing it with the team that its something we probably should be doing.

Actual behavior I think while its loading stats, it's still in the process of starting everything up, and the fact that we have a lot of high-volume channels, like ADT, it prolongs it even further. The channels I believe are deployed, but just not showing because gathering stats is expensive.

Mirth Version: 4.2.0 openjdk version "22-ea" 2024-03-19 OpenJDK Runtime Environment (Red_Hat-22.0.0.0.36-1) (build 22-ea+36) OpenJDK 64-Bit Server VM (Red_Hat-22.0.0.0.36-1) (build 22-ea+36, mixed mode, sharing) Red Hat Enterprise Linux release 8.9 (Ootpa)

ab-mg-23 commented 2 months ago

I'll start with a question: What suggested a potentially 300+ thread software solution would have a fast startup/restart time? Threads aren't cheap.

thorst commented 2 months ago

I believe there is room for improvement possibly, but the primary issue was that we were having a disk array issue causing io errors and slowing everything down.

thorst commented 2 months ago

I'll start with a question: What suggested a potentially 300+ thread software solution would have a fast startup/restart time? Threads aren't cheap.

Is there a "max" recommended number of interfaces per server? Our env runs with low cpu/ram, but perhaps we have too many interfaces on this server?

pacmano1 commented 2 months ago

There isn't that I know because it wildly depends on what your channels do, message volume, back end config, etc. Before I wrote all code as multi-tenant, I have had engines with 1500 channels.