500 internal server error on GET /passes/<norad-id> - Githubissues

redraw / satellite-passes-api

query next satellite passes 🛰️

https://sat.terrestre.ar

32 stars 4 forks source link

500 internal server error on GET /passes/<norad-id> #12

Closed viconnex closed 6 months ago

viconnex commented 6 months ago

Hello! Thank you for this project! I get 500 errors when calling the GET /passes/<norad-id> endpoint. Did you also notice this? Do you know what is happening? Or am I making a mistake in my query?

Below a sample of the performed query:

curl 'https://satellites.fly.dev/passes/25544?lat=-34.911221&lon=-57.9372988&limit=100&days=7&visible_only=false' \
  -H 'accept: application/json' \
  -H 'accept-language: fr,en-US;q=0.9,en;q=0.8,fr-FR;q=0.7,es;q=0.6,de;q=0.5,it;q=0.4' \
  -H 'cache-control: no-cache' \
  -H 'pragma: no-cache' \
  -H 'priority: u=1, i' \
  -H 'referer: https://satellites.fly.dev/docs/' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin'

Result:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or
    there is an error in the application.</p>

Thank you for your help!

redraw commented 6 months ago

Hi, thanks for alerting. Reading the logs and seems like the redis cache reached the daily limit 🥵

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] cache.set(cache_key, tle, CACHE_TIMEOUT)

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 1801, in set

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] return self.execute_command('SET', *pieces)

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 901, in execute_command

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] return self.parse_response(conn, command_name, **options)

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 915, in parse_response

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] response = connection.read_response()

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 747, in read_response

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] raise response

2024-04-24T14:58:35.835 app[9080e205a22087] gru [info] redis.exceptions.ResponseError: max daily request limit exceeded. Limit: 10000, Usage: 10000. See https://upstash.com/docs/redis/troubleshooting/max_daily_request_limit for details

2024-04-24T14:58:35.836 app[9080e205a22087] gru [info] 172.16.136.66 - - [24/Apr/2024:14:58:35 +0000] "GET /passes/25544?lat=-34.911221&lon=-57.9372988&limit=100&days=7&visible_only=false HTTP/1.1" 500 290 "https://satellites.fly.dev/docs/" "curl/8.4.0"

I guess I could skip cache writes if hitting this limit, I'll take a look.

redraw commented 6 months ago

I could add Cloudflare on front to cache cached requests, but that will require a domain change.

redraw commented 6 months ago

Done. Now use https://sat.terrestre.ar and let's wait until tomorrow to reset the daily quota. I blocked a heavy bot that was consuming the quota.

viconnex commented 6 months ago

Hello @redraw! Thanks for setting this up! However I get 403 Forbidden errors now 😅
I am blocked by the Cloudflare protection. The error message invites me to share my Cloudflare Ray ID: 88394dda2a0a9e8d. Does it help for debugging? Thanks! 🚀

redraw commented 6 months ago

@viconnex based on the Cloudflare Ray ID, you are THE one I've blocked 👀. You were flooding the server, exceeding the redis operations quota. I can unblock your IP if you can fix your code. Seems like you were hitting the /passes endpoint in a while True loop maybe?

Screenshot 2024-05-14 at 12 42 37

viconnex commented 6 months ago

LOL indeed! Sorry! My code calls the /passes endpoint every 10 seconds. What would be an acceptable delay for you?

redraw commented 6 months ago

No prob. Why every 10 seconds? passes don't change. Calculations tends to drift after 15 days of prediction, as those will require updated TLEs in the future. Every 1 day you should be fine.

redraw commented 6 months ago

Just unblocked your IP 👍

viconnex commented 5 months ago

Thank you! Yes you're right, I will change the code.