get a 500 status code for some requests

[sckott]

E.g.,

~$ http http://www.antweb.org/api/?genus=crematogaster

HTTP/1.0 500 Internal Server Error
Connection: close
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Wed, 19 Feb 2014 20:33:06 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3

http http://www.antweb.org/api/?genus=camponotus gives the same thing

[sckott]

But this format seems to work

http http://www.antweb.org/api/?rank=genus&name=crematogaster

~$ HTTP/1.1 200 OK
Connection: close
Content-Type: text/html; charset=UTF-8
Date: Wed, 19 Feb 2014 20:34:43 GMT
Server: Apache/2.2.15 (CentOS)
Transfer-Encoding: chunked
X-Powered-By: PHP/5.3.3

[{"genus":"(aenictinae)"},{"genus":"(amblyoponinae)"},{"genus":"(apidae)"},{"genus":"(attini)"},{"genus":"(basicerotini)"},{"genus":"(bethylidae)"},{"genus":"(braconidae)"},{"genus":"(cardiochilinae)"},{"genus":"(cerapachyinae)"},{"genus":"(cryptinae)"},{"genus":"(cynipidae)"},{"genus":"(dacetini)"},{"genus":"(diapriidae)"},{"genus":"(diaspididae)"},{"genus":"(dolichoderinae)"},{"genus":"(dorylinae)"},{"genus":"(ecitoninae)"},{"genus":"(ectatomminae)"},{"genus":"(eurytomidae)"},{"genus":"(figitidae)"},{"genus":"(formicidae)"},{"genus":"(formicinae)"},{"genus":"(hymenoptera)"},{"genus":"(mutillidae)"},{"genus":"(myrmicinae)"},{"genus":"(pompilidae)"},{"genus":"(ponerinae)"},{

...cutoff

p.s.. http is a python library on the cli

[sckott]

And the aw_data function uses the first format, with either /api/?genus=... or /api/?species=...

[karthik]

Looking into it now. The way the API is set up is tricky. Using rank = genus, name = "foo" will not allow you to also include a species name because you can't do: rank = genus, name = "foo", rank = species, name = "foo2" So I went with option A.

I'll check with @jaymollica on this but I know he's out of town this week.

[sckott]

Right, i realized you couldn't pass a eptithet when using the rank= format, but that at least seems to work when you're just passing genus and no epithet

[karthik]

Good idea, I'll implement that fix. But this is weird (that you get a response on the cli while I can't in R).

library(httr)
base_url <- "http://www.antweb.org/api/"
args <- plyr::compact(as.list(c(rank = "genus", name = "crematogaster")))
results <- GET(base_url, query = args)
results
Response [http://www.antweb.org/api/?rank=genus&name=crematogaster]
  Status: 500
  Content-type: text/html; charset=UTF-8

[sckott]

Would be nice if he would return application/json instead of text/html too

[karthik]

Both issues are now fixed by pagination. In AntWeb_0.5.9 or greater:

install_github("ropensci/AntWeb")
library(AntWeb)
crem <- aw_data(genus = "crematogaster")
Query contains 15341 results. First 1000 retrieved. Use the offset argument to retrieve more 

15341 results available for query. Downloading 1000
[Total results on the server]: 15341 
[Args]: 
genus = crematogaster 
limit = 1000 
[Dataset]: [1000 x 18] 
[Data preview] :
                                                       url catalogNumber     family  subfamily
1      http://antweb.org/api/v2/?occurrenceId=antweb:60025         60025 formicidae myrmicinae
2 http://antweb.org/api/v2/?occurrenceId=antweb:alas188691    alas188691 formicidae myrmicinae
          genus specificEpithet            scientific_name typeStatus stateProvince country dateIdentified
1 Crematogaster    curvispinosa crematogaster curvispinosa                  Heredia             1995-07-06
2 Crematogaster    curvispinosa crematogaster curvispinosa                  Heredia             2000-06-06
                         habitat minimumElevationInMeters geojson.type decimal_latitude decimal_longitude
1 Huertos plots Cordia alliodora                       50        point         10.43333         -84.01667
2       Sendero Jaguar Hojarasca                       50        point         10.42118         -84.02531
  decimalLatitude decimalLongitude
1            <NA>             <NA>
2            <NA>             <NA>
>

Now large requests wont return a code 500. If there are more than 1000 requests, it will automatically default to that if the user has not specified their own limit/offset.