search

r-spacex / SpaceX-API

:rocket: Open Source REST API for SpaceX launch, rocket, core, capsule, starlink, launchpad, and landing pad data.
Apache License 2.0
10.5k stars 935 forks source link

Performance issue with /launches/all #247

Closed Tearth closed 5 years ago

Tearth commented 5 years ago

It's not the bug, but I just want to let you know about it. When trying to retrieve data from the https://api.spacexdata.com/v2/launches/all endpoint, it usually takes a few seconds (from 4 to 8) before the data is received, so quite long. I assume that it's because of the launches count, but maybe some sort of cache would be helpful with shortening waiting time?

Screenshot_1

jakewmeyer commented 5 years ago

Looks like we might be outgrowing the current VM hosting the project.

There's been a massive influx of traffic this month, and the nginx request logs ate up all the disk space, which caused an issue with Redis. Cleared out all the logs, and everything returned to normal. Also been wanting to cache on the edge more with Cloudflare, but this would have to likely be done with the new Workers product, because the shortest time you can cache is ~2 hours.

cloudflare

cloudflare2

Going to reach out to DigitalOcean and see if they can gift us a small Kubernetes cluster for the project going forward, or just a beefier VM in general.

However this doesn't address the slow database issue. The slowness likely has to do with the low tier MongoDB Atlas cluster everything runs on. The lowest tiers use shared CPU and RAM, so I don't have any insights into the cluster performance unfortunately.

Cheapest dedicated cluster is ~$60 a month, which is cost prohibitive for a project of this size. I'll reach out to MongoDB, not sure I've seen them sponsor things though.

Tearth commented 5 years ago

Thanx for the explanation, now I understand the situation. I'm not sure if I should close this issue, so you decide.

jakewmeyer commented 5 years ago

Closing since the immediate issue is resolved 👍