Design APIv3

[humitos]

We have been talking about creating a new APIv3 since APIv1 is deprecated (uses old non-maintained 3rd party libs) and APIv2 is not as useful as we want (we can't query anything by slug for example).

This issue is a way to collect data/info/thoughts and start writing a document with the design proposed before coding anything new.

I also raised the idea of using GraphQL in another issue, https://github.com/rtfd/readthedocs.org/issues/4182

Suggested documents/videos/etc:

• https://geemus.gitbooks.io/http-api-design/content/en/
• https://www.youtube.com/watch?v=JQYnFyG7A8c

Core, feel free to edit the list below. We can use cases here to build up our opinions on what apiv3 needs

What are the problems with apiv2?

I documented some of the problems with APIv2 in a note in the docs: https://docs.readthedocs.io/en/latest/api/v2.html

• No real authentication scheme. It's read-only.
• Not designed for slugs. It should be. I don't see how this could be fixed in v2 although the filtering is a good workaround.
• Also, related, impossible to find your project id now, slugs++
• Quite a few of the APIs aren't really designed for outside use (eg. search)
• Error reporting is a bit of a mess. While testing it out for documenting, I got a number of 500 and 502 (timeout) errors. This could be mostly be fixed without a v3.
• Relationships between API objects is not obvious. Return for a Project doesn't give me an easy way of querying API for versions. I have to hand construct a URL for versions. HTML URL relationship serializer would solve this problem

Decisions made by Core Team

We are going to split the work in different phases, aiming to keep it simple but start offering an API with useful features as soon as possible.

Phase I

• Access most of the objects by slug adding them to the URL: /api/v3/projects/pip/
  • there are some cases where it doesn't make sense (Build objects, for example)
  • Community: User, Project, Version and Build
  • Corporate: (Community) + Organization, Team and TeamMember
  • Return related objects --special care with performance: make sure it's just one query to the db
• Personalized object listing based on authenticated user
• Do not allow public listing for objects
• Basic Auth won't be supported
• Token based auth (linked directly to a User)
• No authentication required for detailed endpoints (specific Version)
• All non-readonly endpoints must force CSRF check
• All non-readonly endpoints must required authentication
• Kill APIv1 (#5293)
• Standarize datetime fields (#4864)

Next phases

• Token-Scope based auth
• Footer / Version's menu flyout endpoint
• Docsearch
• Upload built docs
• Access and actions on more objects (growing from what users are going to use): our dashboard could use our own API

Nice to have

• HATEAOS
• Core API
• Rate limit

Considerations

• Be careful about what to expose by filters --make sure fields are indexed
• Slugs are becoming mutable, so it could be a good idea to access by slug or id --return theid` on the detailed response

[stsewd]

Would this API also be read only?

[davidfischer]

Would this API also be read only?

I would say that it shouldn't be but if it helps to get it out the door it could be read-only to start and add write capability to the same endpoints later.

Adding write capabilities means having to do some form of authentication which the current API doesn't really do at all (it will use your session token if you make the call from the browser but that's about it). Getting auth right is hard and will require good amount of work and testing.

[davidfischer]

Also, related, impossible to find your project id now, slugs++

This is now not such a big deal! https://readthedocs.org/api/v2/project/?slug=pip

Relationships between API objects is not obvious. Return for a Project doesn't give me an easy way of querying API for versions. I have to hand construct a URL for versions. HTML URL relationship serializer would solve this problem

This is still true but it is at least better with the documentation and filtering that went live today. https://readthedocs.org/api/v2/version/?project__slug=pip

How will apiv3 be better?

I think a good idea here is to start just writing down the endpoints we think we need and what they should return. We know (I don't but I could if I looked hard enough) what API endpoints the build system uses. We know what API endpoints public documentation needs: the footer API.

Here's some thoughts to get the creative juices going:

• https://readthedocs.org/api/v3/project/ should maybe show your projects by default for a logged in user. For a logged out user, I'm not sure. I don't think paging through 100k projects is that useful.
• For builds, I'm 100% sure paging through all builds is worthless. You should have to request a project to get a list of builds. Personally I like endpoints like https://readthedocs.org/api/v3/project/pip/builds/ (edit: I forgot "project" in there)
• One of the nice aspects of GraphQL -- I'm no expert so maybe I'm wrong here -- is that it should be easy to request the versions and builds for a given project with a single request. This is possible for non-GraphQL too but if you include it by default all project querying is more expensive and if not then you need two API calls to get the same data.
• I think it would be great if the footer API didn't exist in v3 and we just used the version API.
• There's not currently a APIv2 for users. I'm not 100% sure there should be a v3 API for it.

[davidfischer]

Just to add a note based on a discussion with @agjohnson, I think APIv3 should require authentication. The lack of authentication in APIv1 and APIv2 makes upgrading/deprecation a huge pain. Currently, we don't have a good idea who is using the API and for what. On a similar note, the default for the API should probably be to show "my" projects and builds rather than all projects and all builds. I think this would make the API considerably more usable.

[davidfischer]

I think APIv3 should require authentication.

One drawback of this would be that if somebody needed to use the API during a build, they would need a way to expose their API key securely to the Read the Docs builder.

[stsewd]

A way to expose their api key can be with https://github.com/rtfd/readthedocs.org/issues/3992

[davidfischer]

A way to expose their api key can be with #3992

I was looking for this and I couldn't find it. Thanks.

[modelmat]

Couple of things I would definitely like to see:

• api for genindex.html file (e.g [http://discordpy.readthedocs.io/en/rewrite/genindex.html])
• possibly to return a specific object's representation, slightly possible with /search endpoint and with html parsing, but it would be nice for an api with this (possibly tied in with the genindex.html

[agjohnson]

We don't do anything special with generated index files. If anything this data will make its way into our search, but I don't know if it would warrant it's own api endpoint

[stsewd]

Request from #4649

API provides a feature to enable/disable build for specific branch

[stsewd]

Request from #4650

Query the build status of a version

[jaraco]

At a high level, ideally the API wouldn’t be a bolt-on consideration after the service is created, but would be the foundation on which a service is built, such that all actions available in the UI are also available in the API because the UI relies on the API to perform its operations. I realize such an approach may not be a good fit for this project in its current state, but it may be worth considering at this juncture. If you could build a v3 API that is designed to supply the complete suite of operations demanded by any client, it will help avoid cases where some basic features (like project creation) aren’t available through an API.

[stsewd]

Request from #2725

Create a project using the API

[humitos]

I update the description of this issue to write what we talked in our meeting last week and the decisions we made. I will start modifying the document proposed at #4863 soon.