I propose to have two configurable limits for defining the threshold of http 413 return (for dataselect):
1) max days per stream (a stream is always returned to the end user at once. limit required to allow timeouts)
2) max total number of stream days.
For configuring:
on 1) some nodes are quite slow, (e.g. orfeus 28min to deliver 100 days = 1G of 100Hz HHz data), thus reasonable limit could be 35 days (allowing users to request monthly intervals)
on 2): max total temp storage required is #endnode connections multiplied by the data size returned by maximum epoch request of a highly sampled channel. With strict global connection control, adequate storage and slow endpoints (but this may change), this number may actually not even need to be set. For the time being (no full connection control in situations of parallel requests to the federator, 1000 stream days may be adequate.
I propose to have two configurable limits for defining the threshold of http 413 return (for dataselect):
For configuring: on 1) some nodes are quite slow, (e.g. orfeus 28min to deliver 100 days = 1G of 100Hz HHz data), thus reasonable limit could be 35 days (allowing users to request monthly intervals) on 2): max total temp storage required is #endnode connections multiplied by the data size returned by maximum epoch request of a highly sampled channel. With strict global connection control, adequate storage and slow endpoints (but this may change), this number may actually not even need to be set. For the time being (no full connection control in situations of parallel requests to the federator, 1000 stream days may be adequate.