Arrays of things

[achingbrain]

There are APIs that take and return arrays of things - Get all pins and Add an array of pins for example.

How big are we expecting these arrays to be? If unbounded, it would be better to accept/return ndjson to avoid having to a) hold these all in memory at once and b) wait for the request/response to be sent before we can start processing the message without needing streaming JSON parsers.

Pagination could be used to limit response sizes, but if you want to paginate requests you may need some way of holding state on the server if the pages relate to each other and you'd still need to buffer each request before you could start processing it in order to parse the JSON contained in the body.

If ndjson is used, some way of conveying error messages during a stream is required. The approach we agreed on for the IPFS HTTP API (though have yet to implement) was to wrap each JSON line in something like { result: ... } or { error: ... } so the caller could differentiate between an error and a valid request/response entity.

[lidel]

The default API should be very simple and I worry ndjson introduces significant complexity, which could impact adoption.

I like suggestion from #14 to cap responses at the spec level to around 1000 records, forcing client clients to use pagination on bigger sets.

Support for ndjson responses could be added as opt-in via Accept header at later stage, but I don't think its feasible for mvp.

[obo20]

for calls that deal with passing in an array of CIDs it would be nice to have a limit on the amount of CIDs that can be passed in at once. Specifically from the get all pins endpoint as postgres (what we use for our DB) can have some interesting performance issues when doing queries on arrays of IDs.

[lidel]

Explicit array size limits are proposed in https://github.com/ipfs/pinning-services-api-spec/pull/20