Expected QPS, latency, and data retention

[noonhub]

Providing frequent and quick access to an ever growing set of data will obviously put pressure on provider's infrastructure, especially for new or smaller providers which may not have as sophisticated technology stacks. I think that we should establish explicit limits and expectations around QPS, latency, and data retention.

QPS/Load

• What kind of QPS should providers expect on these APIs?
• As a provider, I expect we will want to enforce rate limits, what are acceptable limits?

Latency

• What is an acceptable latency?
• Should we consider asynchronous implementations for providers that cannot meet latency expectations easily?

Data Retention

• What is the expected data retention? Indefinite?

[oderby]

• What is the expected data retention? Indefinite?

Judging from the material released from last week's workshop, it seems like the thought is about 1 year. (On slide 11 of MDS-Presentation-Provider-Agency.pdf it states "API must ... Contain and store 1 year of Data")

Regardless, the specification as documented here on Github is ambiguous and should be clarified. Additionally, it would be nice to agree on response code served from Provider APIs when the request is for a time range outside of the required range (and clarity on how to handle requests for which the provider only has partial information).

[hunterowens]

Ref for #105, we should have an error code for "data no longer stored"

On one hand, would love to standardize, on the other than, different jurisdictions == different retention schedules.

[jsierles]

Is there a document we can refer to that clearly states the retention policy and expected response times for historical data in the Provider API?

Given the requirement of the new agency API, we're wondering if it makes sense to have a long retention period like one year. As providers grow, the arbitrary one-year cutoff can become increasingly difficult to support in real time.

For practical purposes today, 'd to know if currently we can set a shorter retention cut-off while this spec evolves and providers work on long term data warehousing strategies.

[schnuerle]

The SLA/latency and other expectations from an agency could be added to the #646 Policy Requirements feature in an upcoming release, and provider specific details like historic data retention could be added to #628 in the future.