search

dandi / dandi-infrastructure

A repository to collect docs/issues on DANDI project infrastructure
Apache License 2.0
0 stars 6 forks source link

estimate hot/warm/cold data access ratios within archive or even per dandiset #162

Open yarikoptic opened 1 year ago

yarikoptic commented 1 year ago

cold - over a week I guess

waxlamp commented 1 year ago

Could you say more about this, @yarikoptic? Perhaps give some examples of what you expect to see?

yarikoptic commented 1 year ago

I believe this request was in relationship to questionnairs from protocollabs while investigating data access schemes if data gets deposited to IPFS. It relates to typical scenarios on how people access data -- are the same files get re-accessed immediately, or typically just once and then not again for a while. I do not truly remember hot/warm separation idea here, but we can make it following that "Cold" - prior to first access in current period (day? week?) it was not accessed for a week. "Warm" - was accessed within last day, "Hot" - was accessed within last hour.

So e.g. for overall transfer of 1TB of data, we could have Cold: 900G, Warm: 90G, Hot: 10G thus showing that primarily data is accessed once and only small portion (1%) could benefit from some kind of faster caching layer to speed up access. But we could also see Cold: 10G, Warm: 90G, Hot: 900G suggesting that having some cache over overall slow storage backend could be of great benefit since most of the time data is immediately reaccessed.

Makes sense?