search

cycronix / cloudturbine

CloudTurbine - Data Streaming Middleware
Apache License 2.0
3 stars 4 forks source link

Feature Request: Add the ability to perform a sparse fetch with a given interval. #156

Open vxlk opened 5 months ago

vxlk commented 5 months ago

Given N Frames between 0, 10 seconds and an interval I between N:

I want to:

Be able to fetch every I'th frame in interval N, in one request.

Plain english:

Give me every 10th frame available, without me having to make a separate API request for each frame.

Use case:

Reducing the time to see a frame in our web app - given extremely large groups of data. We would like to "dither" or get a lower resolution version of the video in the app as fast as possible.

mjm-cycronix commented 5 months ago

Hi,

Short answer: for most cases, it would be comparable efficiency for your app to make separate API requests for each frame as it would be for the API to do this for you. In each case, the underlying code just cycles through time-stamped files for all IO.

Longer answer: the CT API automatically buffers and caches file requests, e.g. frames within zip files can be re-fetched without repeatedly unzipping them. If your app is making requests over a network (e.g. CTweb using HTTP), it could indeed save round-trip overhead to bundle up multiple frames on the server and send them in a single merged response. This would probably be a niche application that could be layered on the API vs building it in...

A thought: if your application generating the data could write both a full-rate CT "channel" along with a down-sampled separate channel (e.g. at 1/10 rate), on playback you could then choose either full or partial-rate data. The price would be small; e.g. 10% upfront overhead on bandwidth and storage...