search

wpengine / faustjs

Faust.js™ - The Headless WordPress Framework
https://faustjs.org
Other
1.44k stars 131 forks source link

HEADLESS WORDPRESS CONNECTION LOST 524 on NEXTJS FRONTEND #907

Closed esilverman682 closed 2 years ago

esilverman682 commented 2 years ago

Applicable Versions

Steps To Reproduce

Screen Shot 2022-06-20 at 4 54 27 PM

Link to code example: https://newfaust.vercel.app

The current behavior

The expected behavior

theodesp commented 2 years ago

@esilverman682 It looks like the host is down there. Were you able to pinpoint why the connection fails? Is the kapstaging site hosted in WP Engine Atlas? We might also need more information about the Applicable Versions

esilverman682 commented 2 years ago

It is hosted on DigitalOcean with Cloudflare DNS +SSL pointed to the IP. The NEXTJs portion is on Vercel. Ive been trying to figure out the issue for some time now. It works well until one navigates on a few pages / urls. Then the backend begins to slo down and eventually gives an error code of 524 by cloudflare then it goes back up again on its own. memory limit is set high 512M along with post_max_size 512M. The server is Litespeed by the way.

On Tue, Jun 21, 2022 at 5:35 AM Theofanis Despoudis < @.***> wrote:

@esilverman682 https://github.com/esilverman682 It looks like the host is down there. Were you able to pinpoint why the connection fails? Is the kapstaging site hosted in WP Engine Atlas? We might also need more information about the Applicable Versions

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1161504955, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD6YULEYQMSGVYG7ZXVLVQGEHTANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

esilverman682 commented 2 years ago

I am willing to grant access if needed.

On Tue, Jun 21, 2022 at 5:35 AM Theofanis Despoudis < @.***> wrote:

@esilverman682 https://github.com/esilverman682 It looks like the host is down there. Were you able to pinpoint why the connection fails? Is the kapstaging site hosted in WP Engine Atlas? We might also need more information about the Applicable Versions

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1161504955, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD6YULEYQMSGVYG7ZXVLVQGEHTANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

theodesp commented 2 years ago

According to the docs 524 is timeout error. How long did you wait for this to show because it mentions by default it is 100 seconds? If the the backend begins to slow down maybe there is a GraphQL request that is very expensive to resolve.

Is it possible to replicate this flow using Chrome Recorder tool?

https://developer.chrome.com/docs/devtools/recorder/

You will have to enable recording as shown in the guide and perform the interactions. Once you hit the 524 take a screen shot and export the recording as JSON and share it here:

Screenshot 2022-06-22 at 10 59 22
esilverman682 commented 2 years ago

Will do.

On Wed, Jun 22, 2022 at 6:00 AM Theofanis Despoudis < @.***> wrote:

According to the docs 524 https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#524error is timeout error. How long did you wait for this to show because it mentions by default it is 100 seconds? If the the backend begins to slow down maybe there is a GraphQL request that is very expensive to resolve.

Is it possible to replicate this flow using Chrome Recorder tool?

https://developer.chrome.com/docs/devtools/recorder/

You will have to enable recording as shown in the guide and perform the interactions. Once you hit the 524 take a screen shot and export the recording as JSON and share it here:

[image: Screenshot 2022-06-22 at 10 59 22] https://user-images.githubusercontent.com/328805/175001868-9afe86fa-f3ae-49cb-b6fb-6d2b9c2ca384.png

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1162898821, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD66D5QWX5XURPSVGILDVQLP2LANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

theodesp commented 2 years ago

@esilverman682 any updates?

esilverman682 commented 2 years ago

TIMEOUT.zip See Attached.

theodesp commented 2 years ago

@esilverman682 I'm trying to reproduce with no success. I will try the workflow again today.

theodesp commented 2 years ago

@esilverman682 I tried several times to reproduce this with no success.

esilverman682 commented 2 years ago

I recreated the instance. it was working as expected and suddenly began to freeze the backend once again. Im using Custom Post Type + atlas content modeler mix with faust for the schema generation. I can add you to the github repo on the Nextjs frontend and to the Wordpress Backend if you need to get a better look at whats going on. Your help is much appreciated.

Front: https://model-eta.vercel.app/products-services/invoice-factoring Back: https://ada-kapitus.com/

On Wed, Jun 22, 2022 at 6:00 AM Theofanis Despoudis < @.***> wrote:

According to the docs 524 https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#524error is timeout error. How long did you wait for this to show because it mentions by default it is 100 seconds? If the the backend begins to slow down maybe there is a GraphQL request that is very expensive to resolve.

Is it possible to replicate this flow using Chrome Recorder tool?

https://developer.chrome.com/docs/devtools/recorder/

You will have to enable recording as shown in the guide and perform the interactions. Once you hit the 524 take a screen shot and export the recording as JSON and share it here:

[image: Screenshot 2022-06-22 at 10 59 22] https://user-images.githubusercontent.com/328805/175001868-9afe86fa-f3ae-49cb-b6fb-6d2b9c2ca384.png

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1162898821, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD66D5QWX5XURPSVGILDVQLP2LANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

theodesp commented 2 years ago

Hey @theodesp thank you. I was able to replicate the timeout issue using a some quick load testing.

❯ wrk -t12 -c400 -d30s https://model-eta.vercel.app/partner/sales-partners
Running 30s test @ https://model-eta.vercel.app/partner/sales-partners
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   343.40ms  236.10ms   1.97s    88.01%
    Req/Sec    57.78     27.43   160.00     61.77%
  20637 requests in 30.09s, 0.86GB read
  Socket errors: connect 158, read 0, write 0, timeout 13
Requests/sec:    685.81
Transfer/sec:     29.15MB
Screenshot 2022-07-27 at 11 13 27

However it is difficult to discern which component fails here.

It could be the Wordpress instance it could be the Vercel function or the network. We would need some sort of instrumentation here. You could check the logs to see which part times out first. Then you could try to use the Vercel Analytics https://vercel.com/analytics to see some insights.

If you identify that the issue is related to Wordpress then you will need to check the DigitalOcean host to see which part of the server fails. The DigitalOcean dashboard provides a monitoring tool for droplets -> https://docs.digitalocean.com/products/monitoring/

After that step you will need to check the Wordpress instance using an APM tool. For example you can use the free tier for NewRelic PHP Agent to capture traces within the server to pinpoint which part fails to deliver:

https://docs.newrelic.com/docs/apm/agents/php-agent/installation/install-php-agent-shared-hosting-service/

After that step you will know which part fails and we can provide insightful suggestions for resolving it.

esilverman682 commented 2 years ago

Hi Theofanis, Here are the results. I attached a new recording and some screenshots at the time when Wordpress Admin froze today. U went thru product-services-business-loans and other product links and this is when it all went downhill.

DIGITALOCEAN MONITORING- set for ( if CPU is above 70) No Alert notification was sent RELIC- https://onenr.io/07wkvpNxmQL username: @.** password: Xena1234 VERCEL- *No issues Score 100

On Wed, Jul 27, 2022 at 6:25 AM Theofanis Despoudis < @.***> wrote:

Hey @theodesp https://github.com/theodesp thank you. I was able to replicate the timeout issue using a some quick load testing.

❯ wrk -t12 -c400 -d30s https://model-eta.vercel.app/partner/sales-partners

Running 30s test @ https://model-eta.vercel.app/partner/sales-partners

12 threads and 400 connections

Thread Stats Avg Stdev Max +/- Stdev

Latency   343.40ms  236.10ms   1.97s    88.01%

Req/Sec    57.78     27.43   160.00     61.77%

20637 requests in 30.09s, 0.86GB read

Socket errors: connect 158, read 0, write 0, timeout 13

Requests/sec: 685.81

Transfer/sec: 29.15MB

[image: Screenshot 2022-07-27 at 11 13 27] https://user-images.githubusercontent.com/328805/181222936-c7cba347-e271-47b0-be52-e470beddc85a.png

However it is difficult to discern which component fails here.

It could be the Wordpress instance it could be the Vercel function or the network. We would need some sort of instrumentation here. You could check the logs to see which part times out first. Then you could try to use the Vercel Analytics https://vercel.com/analytics to see some insights.

If you identify that the issue is related to Wordpress then you will need to check the DigitalOcean host to see which part of the server fails. The DigitalOcean dashboard provides a monitoring tool for droplets -> https://docs.digitalocean.com/products/monitoring/

After that step you will need to check the Wordpress instance. For example you can use the free tier for NewRelic PHP Agent to capture traces within the server to pinpoint which part fails to deliver:

https://docs.newrelic.com/docs/apm/agents/php-agent/installation/install-php-agent-shared-hosting-service/

After that step you will know which part fails and we can provide suggestions for that.

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1196547531, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD62VVGPQOBJJNIVSW6DVWEFAXANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

esilverman682 commented 2 years ago

New Recording to accompany the Relic logs.

On Wed, Jul 27, 2022 at 6:25 AM Theofanis Despoudis < @.***> wrote:

Hey @theodesp https://github.com/theodesp thank you. I was able to replicate the timeout issue using a some quick load testing.

❯ wrk -t12 -c400 -d30s https://model-eta.vercel.app/partner/sales-partners

Running 30s test @ https://model-eta.vercel.app/partner/sales-partners

12 threads and 400 connections

Thread Stats Avg Stdev Max +/- Stdev

Latency   343.40ms  236.10ms   1.97s    88.01%

Req/Sec    57.78     27.43   160.00     61.77%

20637 requests in 30.09s, 0.86GB read

Socket errors: connect 158, read 0, write 0, timeout 13

Requests/sec: 685.81

Transfer/sec: 29.15MB

[image: Screenshot 2022-07-27 at 11 13 27] https://user-images.githubusercontent.com/328805/181222936-c7cba347-e271-47b0-be52-e470beddc85a.png

However it is difficult to discern which component fails here.

It could be the Wordpress instance it could be the Vercel function or the network. We would need some sort of instrumentation here. You could check the logs to see which part times out first. Then you could try to use the Vercel Analytics https://vercel.com/analytics to see some insights.

If you identify that the issue is related to Wordpress then you will need to check the DigitalOcean host to see which part of the server fails. The DigitalOcean dashboard provides a monitoring tool for droplets -> https://docs.digitalocean.com/products/monitoring/

After that step you will need to check the Wordpress instance. For example you can use the free tier for NewRelic PHP Agent to capture traces within the server to pinpoint which part fails to deliver:

https://docs.newrelic.com/docs/apm/agents/php-agent/installation/install-php-agent-shared-hosting-service/

After that step you will know which part fails and we can provide suggestions for that.

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1196547531, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD62VVGPQOBJJNIVSW6DVWEFAXANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

theodesp commented 2 years ago

Hey @esilverman682 could you post relevant tracing info from NewRelic here? I think you can also export them as JSON.

esilverman682 commented 2 years ago

Hi Theofanis,

I couldn't find any data on the traces section. Sorry Im a noob at this if you need to access directly I dont mind. Creds listed below.

RELIC https://onenr.io/07wkvpNxmQL username: @.*** password: Xena1234

Best

On Thu, Jul 28, 2022 at 7:41 AM Theofanis Despoudis < @.***> wrote:

Hey @esilverman682 https://github.com/esilverman682 could you post relevant tracing info from NewRelic here? I think you can also export them as JSON.

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1198028568, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD656OHDZURSOKUKRDX3VWJWVNANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

theodesp commented 2 years ago

Hey @esilverman682 you cannot share credentials here as Github masks emails and stuff. Maybe you can post some screenshots instead or share me on discord https://discord.com/channels/836253505944813629/984146897231032350

esilverman682 commented 2 years ago

Hi Theopanis.

What is your tag/username on discord so I can send creds?

On Fri, Jul 29, 2022 at 4:55 AM Theofanis Despoudis < @.***> wrote:

Hey @esilverman682 https://github.com/esilverman682 you cannot share credentials here as Github masks emails and staff. Maybe you can post some screenshots instead or share me on discord https://discord.com/channels/836253505944813629/984146897231032350

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1199037775, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD63OC4ZSZB5WJHQLRRTVWOL7JANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

theodesp commented 2 years ago

Theo Despoudis#1014. You can find me on the faust.js discord channel

esilverman682 commented 2 years ago

I sent over 3 images but need to split it up since email claims its too large of a file. Here is the first.

On Fri, Jul 29, 2022 at 4:55 AM Theofanis Despoudis < @.***> wrote:

Hey @esilverman682 https://github.com/esilverman682 you cannot share credentials here as Github masks emails and staff. Maybe you can post some screenshots instead or share me on discord https://discord.com/channels/836253505944813629/984146897231032350

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1199037775, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD63OC4ZSZB5WJHQLRRTVWOL7JANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

esilverman682 commented 2 years ago

Second.

On Fri, Jul 29, 2022 at 4:55 AM Theofanis Despoudis < @.***> wrote:

Hey @esilverman682 https://github.com/esilverman682 you cannot share credentials here as Github masks emails and staff. Maybe you can post some screenshots instead or share me on discord https://discord.com/channels/836253505944813629/984146897231032350

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1199037775, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD63OC4ZSZB5WJHQLRRTVWOL7JANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

esilverman682 commented 2 years ago

Hi Theofanis,

Just sent over a friend request to later send the creds.

Best, Wil

On Fri, Jul 29, 2022 at 8:29 AM Theofanis Despoudis < @.***> wrote:

Theo Despoudis#1014

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1199220396, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD6ZSFPXBBXRM4VI5BD3VWPFB7ANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

esilverman682 commented 2 years ago

StephenG#3896 on discord

On Fri, Jul 29, 2022 at 8:29 AM Theofanis Despoudis < @.***> wrote:

Theo Despoudis#1014

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1199220396, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD6ZSFPXBBXRM4VI5BD3VWPFB7ANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>

theodesp commented 2 years ago

Hey @esilverman682 I've sent you a message on Discord about the screenshots. Thank you.

josephfusco commented 2 years ago

Hey @esilverman682, I'm going to close this since this issue is more server related. Please feel free to open up another issue if you are still experiencing problems.

esilverman682 commented 2 years ago

Third

On Fri, Jul 29, 2022 at 4:55 AM Theofanis Despoudis < @.***> wrote:

Hey @esilverman682 https://github.com/esilverman682 you cannot share credentials here as Github masks emails and staff. Maybe you can post some screenshots instead or share me on discord https://discord.com/channels/836253505944813629/984146897231032350

— Reply to this email directly, view it on GitHub https://github.com/wpengine/faustjs/issues/907#issuecomment-1199037775, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKUD63OC4ZSZB5WJHQLRRTVWOL7JANCNFSM5ZKXLPKQ . You are receiving this because you were mentioned.Message ID: @.***>