Design patterns for API wrapper packages

jennybc commented 9 years ago

At the very least, this would make for an interesting discussion or panel.

I know there are a growing number of nuts-and-bolts packages that provide important functionality, such as httr, jsonlite, XML. I'm really trying to raise higher-level questions about what makes for a good package design and user interface.

Examples:

Authentication. Specifically, which user-facing functions and what sort of documentation are needed to help users authenticate in a way that facilities use in unattended scripts and that is secure.
Pagination. How (whether?) to help with page traversal and concatenating results.
Testing. Do you write tests against the actual API? How to finesse things w/r/t CRAN and, e.g., Travis? What about tests as authenticated user?
JSON/XML vs R lists. Best packages to translate between and attitudes re: translating early, late or not at all?
How much info to inflict on the user or to even expose. Specifically, how much metadata about the HTTP requests to retain and how to store/provide?

jcheng5 commented 9 years ago

Really good questions. Selfishly, I'd like these types of API wrappers to work really well in Shiny apps, especially for near-realtime scenarios--which begs for some kind of asynchronous and/or streaming style of API, at least at the HTTP request/response level. (cc @hadley)

cboettig commented 9 years ago

Awesome. However we do this, it would be great if it could also result in at least some rough notes that could evolve into webpage/vignette. Something that tackles the issues @Jennybc raises and others we might list would be a nice complement the httr best practices vignette.

Adding to Jenny's list:

strategies for caching to avoid repeated calls
strategies for error-handling that are programming-friendly
Summarizing / conveying metadata (e.g. column definitions, units)
to the JSON/XML vs R lists, I'd toss in data.frames as well, which I think are often more user friendly / R friendly than the others but also raise their own challenges.
Closely related to this is strategies for handling object classes sensibly (e.g. should you always/sometimes/never coerce logicals and dates into their corresponding R classes?)
The little nice things we forget to do, like declaring user-agent in our calls

hadley commented 9 years ago

I'm planning on writing a book on data ingest, and I'd be happy to include this as a chapter

karthik commented 9 years ago

:100: This would make for a good session early on in the event.

jordansread commented 9 years ago

:+1:

craigcitro commented 9 years ago

i'd love to be involved in this -- but i don't get in until thursday at lunchtime. would happily spend a good chunk of the time i'm there talking about interacting with APIs.

jennybc commented 9 years ago

This blog post I just found via R-bloggers is relevant to this discussion:

What’s the Point of an API?

So what is an API good for, and where does it actually exist? I’m starting to think that for many users, if there isn’t some equivalent of an “import from X” option in the tool they are using or environment they’re working in, then the API-for-X is not really doing much of a useful job for them…. And note further – this does not mean things like a pandas Remote Data Access plugin or a CRAN package for R (such as the World Bank Development Indicators package or any of the other data/API packages referenced from the rOpenSci packages list should be seen as extensions of the API. At worst, they should be seen as projections of the API into user environments. At best, it is those packages that should be seen as the actual API.

APIs for users – not programmers. That’s what I want from an API.

hafen commented 9 years ago

@jcheng5 I'd be interested to hear your thoughts / discuss what a streaming API might look like. I've been thinking about this recently with rbokeh. Visualization is a great use case for streaming and Bokeh has some cool capabilities along these lines (for example I would love to be able to replicate this live audio spectrogram example but served by R: https://www.youtube.com/watch?v=hjW4gL9hioY#t=48m15s).

jeroen commented 9 years ago

The jsonlite package implements async streaming of jsonlines data, which the format that is used by e.g. dat. See ?stream_in in jsonlite.

dani-lbnl commented 9 years ago

Looking forward to participating of this panel!

jcheng5 commented 9 years ago

@hafen @jeroenooms Just some random thoughts on streaming.

We should probably start by cataloging some popular/important streaming APIs and seeing what patterns of access and data format need to be supported.

To consume streaming APIs well from R, we will need a non-blocking, streaming, async HTTP client. We've talked about adding one to the httpuv package, as all the necessary infrastructure is already in there in the C/C++ code. That would provide a low level, maybe httr would be the right place to hang some high level streaming APIs on top.

Whatever abstractions we end up with will need to work in the context of event loops--either the one belonging to R/RGui/RStudio/etc., or the one that Shiny maintains for itself, or the same for httpuv, or any other environment within which we want the streaming APIs to work.

Once some of these greasy bits are in place, we can start working on the really fun/interesting stuff, like high level abstractions for manipulating and computing on streaming data--I have some half-formed thoughts in this area too.

jennybc commented 9 years ago

Hey people who have weighed in here! @craigcitro and I will try to rally those interested in this in the late morning, since he's leaving sometime before 1pm. I'm writing this at 9:50am. If you want to us to find you when we talk, reach out.

jordansread commented 9 years ago

I would like to join :+1:

noamross commented 9 years ago

Remote but will follow as I can!

On Fri, Mar 27, 2015 at 9:53 AM Jordan S Read notifications@github.com wrote:

I would like to join [image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/ropensci/unconf/issues/3#issuecomment-87005881.

gmbecker commented 9 years ago

I am interested in this as well.

On Fri, Mar 27, 2015 at 10:01 AM, Noam Ross notifications@github.com wrote:

Remote but will follow as I can!

On Fri, Mar 27, 2015 at 9:53 AM Jordan S Read notifications@github.com wrote:

I would like to join [image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/ropensci/unconf/issues/3#issuecomment-87005881.

— Reply to this email directly or view it on GitHub https://github.com/ropensci/unconf/issues/3#issuecomment-87009628.

Gabriel Becker, PhD Computational Biologist Bioinformatics and Computational Biology Genentech, Inc.

ropensci / unconf15

Design patterns for API wrapper packages #3