search

stacks-network / stacks-core

The Stacks blockchain implementation
https://docs.stacks.co
GNU General Public License v3.0
3.01k stars 673 forks source link

Performance Degradation in 2.1.0.0.3 #3671

Open CharlieC3 opened 1 year ago

CharlieC3 commented 1 year ago

Describe the bug Based on our metrics, we're seeing a large regression in HTTP response time performance when we upgraded from 2.1.0.0.2 to 2.1.0.0.3 for stacks blockchain nodes receiving HTTP traffic.

We upgraded our nodes around 1500ET on April 17th. CPU and memory usage has stayed the same before and after the upgrade.

HTTP GET response time increased for 200s and 404s. Some affected endpoints I've seen are:

HTTP POST response time increased for 200s and 400s. Some affected endpoints I've seen are:

Screenshot 2023-04-18 at 11 43 24 AM Screenshot 2023-04-18 at 11 43 50 AM

kantai commented 1 year ago

I'll look through our logs to try to see what's driving the performance issues here, but I'm very skeptical that this could be related to 2.1.0.0.3 -- the patch did not touch any codepaths used by follower nodes.

pavitthrap commented 1 year ago

Performance degradations went away after API release was downgraded. It seems like the issue did not emerge from the node upgrade.

More analysis needed.

AshtonStephens commented 1 year ago

First, @kantai you said:

I'll look through our logs to try to see what's driving the performance issues here, but I'm very skeptical that this could be related to 2.1.0.0.3 -- the patch did not touch any codepaths used by follower nodes.

Then @pavitthrap you said:

Performance degradations went away after API release was downgraded.

This sounds like the issue was caused by the API release upgrade. Am I misunderstanding something? Is it possible that the upgrade causes a memory issue or something else on the hosts that could tangentially cause large http times?

kantai commented 1 year ago

Pretty unlikely -- the stacks-blockchain-api itself isn't handling these requests: these are being handled by the stacks-node RPC handler. My best guess is that request patterns changed a little bit which led to a change in the performance of the node.

CharlieC3 commented 1 year ago

Also adding some further perspective, this issue does not mark the first of "poor HTTP performance" for the stacks-node. Rather it should be interpreted as a time when things went from poor to worse. Agreed with @kantai's statement that this may have been a result of a change in user behavior; it's unclear to me and given the issue's age we likely won't be able to extract any more contextual data from that time.