bad gateway - api seems unavailable

[jhpoelen]

Hi! Was just taking another stab at using the fishbase api and got an unexpected {"error":"route not found"} in response to http get for http://fishbase.ropensci.org/refrens?limit=1000&offset=0.

Is api down for maintenance?

See screenshot.

[sckott]

that works for me. is it still not working?

[jhpoelen]

Weird! Now it is working again. For me, this issue and #68 seem to suggest that the api starts acting funny when it is used heavily. Do you have any logs that point to a cpu getting burnt or memory getting full?

[jhpoelen]

(by burnt i mean 100% cpu usage)

[sckott]

yeah, we have been hitting 100% Cpu occasionally. We've talked about getting a bigger box. @cboettig should we do that

[jhpoelen]

Ok. 100% cpu usage spikes would be a reasonable explanation for #71 and perhaps even #68 .

At risk of sounding like a broken record: my use case is to use all data from endpoints like: species, refrens, fooditems, predats. Right now, I have to download all this in chunks of ~1000-5000 records, resulting in hundreds of requests. Instead, I was hoping that the api could provide a link to a file (pre-generated, e.g. refrens.json or refrens.csv) that contains all data. Serving these files is relatively cheap, having to generate the files in 1000-5000 record chunks on the fly seems a bit CPU intensive (as we see in this issue). This might save you from having to buy a bigger box. Curious to hear your thoughts.

[sckott]

Hmm, we are going through some changes #72 - so this is a good time to think about whether we want to provide dumps.

opened a new issue re: dumps #73

[jhpoelen]

Thanks! Looking forward to the outcomes.

[cboettig]

@jhpoelen thanks for the reports here, closing this since it looks like the remaining issue is now #73 instead of bad gateway. Meanwhile do ping us back here if you hit more bad gateway errors