camaraproject / PopulationDensityData

Repository to describe, develop, document and test the Population Density Data API family
Apache License 2.0
3 stars 5 forks source link

Discussion on API spec - how to handle big areas/time in the response #10

Closed jgarciahospital closed 6 months ago

jgarciahospital commented 8 months ago

Following the discussion on maximum time frame and area to cover, we've been analyzing the management of valid/useful requests and the corresponding response.

Considering the size of a common request, of 1 week and common areas of around 10Km2, the size of the response's content may be too big for being handled directly in the API REST body.

Proposal

Include a second option for a developer to be able to request a big area,that will receive as response containing a file link instead of a direct content response. That will allow developers to better manage big responses (<1MB) when asking for data of big areas or long time frame.

To discuss the best approach on how to manage both flexibility and speed of a direct response, with more complete data in a filed response.

sachinvodafone commented 8 months ago

I have some queries on this:

  1. Are there any restrictions on the common area, or can it be as large as 100 KM2?
  2. Additionally, if the area is substantial and the server takes a considerable amount of time to process the data, will the response be asynchronous?
  3. Does this apply specifically to large areas only and will clients receive real-time responses for smaller areas?
  4. And, how do we differentiate between small and large areas?
gregory1g commented 7 months ago

Does it make sense to separate mechanisms to deliver data for large areas and definitions of large areas? First is part of the API and must be agreed here, while the second one is part of implementation or even business strategy and can vary from MNO to MNO and from region to region and therefore unlikely can be defined on Camara level.

jgarciahospital commented 7 months ago

Update from last meeting 06/03 Minutes

It was proposed to separate two mechanisms for supporting separately "big" and "small" areas, but that would include complexity in developer to determine which mechanisms is indeed being used or required.

Final proposal is to analyze the implementation a unique asynchronous mechanisms for being able to handle both kind of scenarios, small scenarios where response will reach soon but also time and resource consuming scenarios where it may take even minutes to calculate the response.

That way, developer doesn't need to take care of which scenario is asking for.

jgarciahospital commented 7 months ago

Update from last meeting 20/03 Minutes

Proposal is to analyze the implementation a unique asynchronous mechanisms for being able to handle both kind of scenarios, small scenarios where response will reach soon but also time and resource consuming scenarios where it may take even minutes to calculate the response.

Agreed WF

→ Gregory: to analyze limiting the size of the request to avoid complexity in the response, at least to consider response size that can fit properly in a HTTP response.

→ Keep discussion in the issue offline.

sachinvodafone commented 6 months ago

In our opinion, the API should support both situations: when immediate information is needed (during flight too) and when the data will be used for future planning, such as a scheduled flight. For the former, a synchronous response is ideal, while for the latter, an asynchronous response works well. However, to ensure real-time information can be processed instantly during synchronous requests, the provider should impose limitations on the input values from the client.

jgarciahospital commented 6 months ago

Decision: Support async endpoint for bigger and smaller areas, also consider to define a size limitation (area, duration and granularity) (To be defined in issue #7). The response will be part of the API response body, not considering external file link. Only identified limitation is on the calculation time, not in the response size. Async mechanism to be discussed in #20, first code consideration in #21