zevenet / zlb

ZEVENET becomes SKUDONET and RELIANOID
Other
143 stars 29 forks source link

High 100% CPU usage #62

Closed JanuszU closed 4 years ago

JanuszU commented 4 years ago

It is a new installation of Zevenet CE v5.10.1 we have moved our farms configuration from an old version CE v4.0.2 - on that old system the load was very low

Two of our 11 farms tend to use all the CPU power after some hours of normal operation. These are https farms. In that full loaded state the requests are still servised but 100% load is no good. Restarting the farms returns the load to about 10% max. I can see no errors in the logs Is there any symptom I can track to figure out the cause ?

emiliocampos-zevenet commented 4 years ago

It could be related to zproxy, could you try to ensure you are running latest version of this product as follow?

apt-get update

apt get install zproxy

Once latest version is installed please restart all the http farms in web GUI and try again

Let us know if this solves the problem.

Thanks!

El vie., 17 ene. 2020 15:24, JanuszU notifications@github.com escribió:

It is a new installation of Zevenet CE v5.10.1 we have moved our farms configuration from an old version CE v4.0.2 - on that old system the load was very low

Two of our 11 farms tend to use all the CPU power after some hours of normal operation. These are https farms. In that full loaded state the requests are still servised but 100% load is no good. Restarting the farms returns the load to about 10% max. I can see no errors in the logs Is there any symptom I can track to figure out the cause ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/zevenet/zlb/issues/62?email_source=notifications&email_token=AFBQEPDCWTA36XY7BLUKTQTQ6G5P7A5CNFSM4KIJBV7KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IG6LYLQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBQEPAKWOCHCL3QMPRXQ3DQ6G5P7ANCNFSM4KIJBV7A .

abdessamad-zevenet commented 4 years ago

You should set temporarily the LogLevel parameter to value 8 in the farm configuration file ( Debug mode ). Also, could you share the farms configuration files and logs using a supportsave (by private if you wish) and which TLS version does the clients use?

JanuszU commented 4 years ago

It will not help since I already have the newest version:

# apt-get install zproxy
Reading package lists... Done
Building dependency tree
Reading state information... Done
zproxy is already the newest version (0.1.3-5.10.1).
emiliocampos-zevenet commented 4 years ago

Thanks, then can you follow Abdessamad's indications?

El lun., 20 ene. 2020 20:54, JanuszU notifications@github.com escribió:

It will not help since I already have the newest version: apt-get install zproxy

Reading package lists... Done Building dependency tree Reading state information... Done zproxy is already the newest version (0.1.3-5.10.1).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/zevenet/zlb/issues/62?email_source=notifications&email_token=AFBQEPBNJOM2FFABDQT7L2DQ6X6Q5A5CNFSM4KIJBV7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJNVM2Q#issuecomment-576411242, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBQEPHTF2JKGETBRSA7MXLQ6X6Q5ANCNFSM4KIJBV7A .

JanuszU commented 4 years ago

Thanks for a hint, I will start with LogLevel 8

JanuszU commented 4 years ago

CPU got again 100%, restarting the farm helps unfortunately there is literally nothing in the logs (LogLevel 8)

please see attached farm config

redir132HttpsJira_proxy.cfg.txt

JanuszU commented 4 years ago

sorry, I found the zproxy logs they were erroneously routed now with loglevel 8 , there are a lot of them, in 300MB of logs today there is just one line with "Error": Jan 22 08:42:00 zen-lb-01 zproxy[15711]: (redir132HttpsJira,Jira-Portal,0) fd: 118:127 GET /s/2fd3e6ee935d33f7b645a2c860fb005a-T/nzwsam/805001/6411e0087192541a09d88223fb51a6a0/d8ee810cd11ff5c1276dddb9ada2b6c1/_/download/contextbatch/js/jira.rich.editor.api,jira.rich.editor,-_super,-jira.view.issue,-atl.general,-jira.global,-project.issue.navigator/batch.js?agile_global_admin_condition=true&jag=true&jaguser=true&jira.create.linked.issue=true&locale=pl-PL&richediton=true&user-logged-in=true&whisper-enabled=true HTTP/1.1 Error sending response IN#011buffer size: 20642#011Content-length: 0#011left: 0 header_sent: true chunk_size_left: 0 IO RESULT: ERROR CH= T

should I look for something specific ?

abdessamad-zevenet commented 4 years ago

Sorry I closed accidentally the issue,

The logs are sent to Syslog. The above error line mostly indicate that the user closed the SSL tunnel prematurely, are the farm users having issues using the service?.

Please write an email to sopport@zevenet.com and i will contact you in order to do deeper checks on the issue.