worldbank / DECAT_Space2Stats

https://worldbank.github.io/DECAT_Space2Stats/
Other
1 stars 2 forks source link

Define user constraints for number of fields and AOI size #37

Closed zacharyDez closed 2 months ago

zacharyDez commented 2 months ago

We are improving the amount of data that the API can return, as pointed out by #35. However, there is an upper limit of data that the API should be expected to return. A user wanting the entire or almost the entire dataset should use the raw data directly.

@andresfchamorro @bpstewar; would you be able to define:

zacharyDez commented 2 months ago

Discussed with @alukach. An initial plan could be:

  1. Get the size of the response and throw a clearer message to users about it exceeding the allowed size and to try with a smaller area or number of fields.
  2. Redirect users to S3 signed url where response data would be stored when requests exceed the allowed size. For most clients (requests, httpx), the default behavior is to follow redirects. Hence, the experience would remain the same (hidden for end-users).

@alukach; let me know if I missed anything here.

andresfchamorro commented 2 months ago

For AOI size, I would say we should be able to handle any country's geometry. Is that feasible?

For number of fields, what you've shown (7/10) sounds reasonable. As long as we know what the limit is and have some recommendations for how to effectively loop and get more variables.

zacharyDez commented 2 months ago

@andresfchamorro; with the new approach using an S3 redirect, we do not need to define these constraints afterall. The experience remains the same from an end-users perspective.