search

socrata / soda-js

A Javascript-based library for accessing the SODA2 API.
175 stars 48 forks source link

Throttling question #39

Closed kylerummens closed 4 years ago

kylerummens commented 4 years ago

Your Getting Help page on dev.socrata.com says that questions should be made on stackoverflow and that issues should be submitted here, so I apologize for posting a question here. However, I have posted two questions on stackoverflow and gotten no responses.

I am trying to query different USAC E-rate databases using soda-js, but I feel like I am getting throttled. The [Socrata App Token documentation](Socrata App Token documentation) says:

Without an application, we can only track usage and perform throttling based on a few simple criteria, mainly source IP address. As such, requests that aren’t using an application token come from a shared pool via IP address. IP addresses that make too many requests during a given period may be subject to throttling.

The problem is, though, that I am passing in an app token with all my requests, but some of the SoQL requests that are fairly simple are taking up to 36 seconds to respond. I know some of the datasets have upwards of 70,000 rows, but am I wrong in assuming that a simple query shouldn't take 36 seconds? Does this mean I'm being throttled?

The Socrata Request ID returned with the request was 17b5a244e611eb41caf87bc3ac073dc7

chitang commented 4 years ago

The request was not throttled. A throttled request will only be rejected instead of taking longer to run. This is a group by query on 8M rows.

kylerummens commented 4 years ago

Ok. So, not being an expert in SQL, is there a way to make the http request quicker than 36 seconds when doing grouping? That seems like an unusual amount of time for a query, but then again I'm not familiar with SQL run times.