search

ipld / go-ipld-prime

Golang interfaces for the IPLD Data Model, with core Codecs included, IPLD Schemas support, and some handy functional transforms tools.
MIT License
132 stars 50 forks source link

Better handling of budgets in preloading #507

Open hannahhoward opened 1 year ago

hannahhoward commented 1 year ago

Goals

Budgets and preloading are an inherently challenging area, but we can probably do better than the current approach.

Right now, for a link budget of N, we can preload up to N^2 / 2 links I think (essentially, I'm imagining a depth N graph, with N links at each level. We would essentially preload (n-1)+(n-2)+(n-3)... as we walked down the left edge of the graph).

I wonder if it would make sense to treat the budget as simply seperate for preloading. A budget is given to the preloader with the same initial target, and when the budget is expended, we shift to just doing no preloading.

This would wait preloading towards the top of the graph, but perhaps that's what's most important?

rvagg commented 1 year ago

I think we should experiment in the preloader logic in Lassie and then bring that back here when we extract out the core of it. Keep track and if we max out the preloader budget then new Preload() calls are just noops.