planningcenter / developers

Planning Center API docs and support
https://developer.planning.center/docs/
85 stars 8 forks source link

Timeout issue with people api #1153

Closed glenhayes closed 9 months ago

glenhayes commented 9 months ago

Affected Product Which product does this bug affect? People

Describe the bug

We are getting Timeouts when do requests on the people/v2 api. This is happening for a couple of hours each day. Hoping you might be able to point us in the right direction for getting this resolved. We are a Planning Center approved 3rd party application.

To Reproduce

Expected behavior

Screenshots

Additional Context:

I have..

seven1m commented 9 months ago

Which API endpoint (i.e. full URL) are you seeing this on? What is the HTTP response code and error message you get from the server?

proefi commented 9 months ago

The callback url is https://servewireapp.com/pcoauth With the php curl command we are getting this error... Curl error: Could not resolve host: api.planningcenteronline.com

The hosting company didn't see any blocks on their end but they elevated the issue to their next tier tech team for review.

The strange thing is this is a rolling outage... the form works most of the day but times out for several hours once or twice a day. This has been occurring for the past few days.

glenhayes commented 9 months ago

The full url we are trying to reach is https://api.planningcenteronline.com/oauth/token to revalidate our token

seven1m commented 9 months ago

Oh man, that's no fun. Since we're not getting other reports of DNS resolution errors, I would suspect this is something inside your data center. The way I would debug an issue like this would be to set up a cron job on the server to run the following commands and output to a log:

#!/bin/bash

# check_dns.sh

echo ====================================
date

dig api.planningcenteronline.com
dig @8.8.8.8 api.planningcenteronline.com
dig @1.1.1.1 api.planningcenteronline.com

echo

You can put that in a crontab to run every 5 minutes and output to a log like this:

*/5 * * * * bash $HOME/check_dns.sh >> $HOME/check_dns.log 2>&1

The first dig call should show you what you are getting with your default DNS server.

The next two calls are Google and Cloudflare public DNS, respectively. I expect you are going to see resolution failure on the first call, but success on the other two.

If that is the case, you will need to investigate the settings of your default DNS server to see what it could be. Since our public IP addresses change frequently, it will be critical that your DNS server honor the TTL (Time to Live) value it gets from upstream servers.

I hope that helps!