Make limits configurable

[wrygiel]

Most retrieval APIs (which ones are those?) have limits set on the number of items the client is allowed to retrieve in a single request.

In the current draft versions of the specifications we have made these limits "hardcoded" (they are part of the API specification itself, example here).

However, we can make all such limits configurable. We have already applied this technique in ToRs API:

• Each EWP Host would be required to provide his own maximum number of allowed IDs, as you can see here and here.
• Clients would be required to either (a) parse and respect these limits (there will be present in the catalogue.xml file returned by the Registry Service), or (b) to issue a separate request for each ID (since <max-ids> cannot be lower than 1).

We have applied this technique to ToRs API only, but we can do this in all other APIs too. What do you think?

[mikesteez]

I think this is a cumbersome way of having a limit on search results and not very RESTful. I think we should agree on a minimal set of records that everyone should be able to return, such as 100. The GitHub API is quite nice and could be adopted https://developer.github.com/v3/search/

[wrygiel]

There are two topics here:

1. One is that of the limits for retrieval APIs (which return entities based on lists of IDs). There is no need to order nor paginate results of such APIs. But there is the need to limit the number of IDs passed.
2. The other are the search APIs. These APIs return IDs only, and they always return all matching IDs, regardless of their number, in order to facilitate replication (e.g. detect deleted items). If we introduce limits to these APIs, then we will need to design separate APIs for replication.

I am aware that our current proposals seem a bit "crude". I would love to see a more complex API built later on, but we must keep in mind that we don't have resources such as GitHub has. For some, the current APIs might be already quite complex to implement, as they are. I think we should start with simple ones, as long as they cover our use cases (and replication is among these usecases).

[wrygiel]

These APIs return IDs only, and they always return all matching IDs, regardless of their number, in order to facilitate replication

Just a clarification: We may (or even should) add a limit parameter for such APIs, but we need for the client to be able to set this limit to infinity (on a per-request basis). So its a different kind of limit.

[wrygiel]

I think we should agree on a minimal set of records that everyone should be able to return, such as 100.

We could agree on such limits among us. But I think we need to take future partners into account too.

[wrygiel]

If we introduce limits to these APIs, then we will need to design separate APIs for replication.

Which, of course, is also possible, if this is what everyone prefers. For example, for each entity we could have get+search and list APIs (get and search APIs would need to be merged in such case, because their results would be intertwined, probably). We will be able to speak more about these options once we have the models to work on.