ZenGround0
commented
1 year ago Spec
Concretely Im working on a tool that can take a snapshot and output information on the total snapshot size. This is mainly different from #9793 in that it accounts for message data and churn data. The tool will report:
1) total bytes in messages, headers, state trees 2) for state tree a breakdown by top level actor field of the size of the set equal to the union of all data blocks belonging to that field 3) For the purposes of this investigation I'll focus in on the market actor to begin with since we suspect it is the major contributor
With this information we can diagnose what went so right in nv19 and how to preemptively avoid the same problem happening in the future in other parts of the state tree.
Implementation
Rough sketch 1) Read CAR snapshots into a badger database so we can do random reads 2) Use / implement function for count size of subgraph 3) Traverse chain, count bytes of headers, count bytes of messages 4) When counting bytes of state tree traverse all actors in the same way as #9793. I'll start by doing this with market actor, others I'll just count at the top level. a) Track every cid in a big set, if already seen don't traverse and count state size b) If this is too much memory revisit and do something less simple, probably a badger datastore for the CID set.
raulk
jennijuju
arajasek
snissn
edited by @ZenGround0
User Story
I am a user who develops the filecoin protocol or software involved in running or using the protocol.
1) I want to understand what is happening with filecoin state. I want to see the current breakdown of byte usage by the protocol to understand if there are dangerous trends that need to be addressed and get ideas for how to fix them.
2) I also want to be able to go back and analyze how previous changes have impacted snapshot size / protocol data usage.
3) I can visualize how new changes I have developed impact snapshot size / protocol data usage.
Acceptance Criteria
I can run a command or a small set of commands that gives me two things provided with a filecoin snapshot:
1) data on the breakdown of state usage of a chain snapshot broken down into a) state tree b) message / headers c) state churn. Further breakdown of state and churn by i) actor type ii) actor type field and further breakdown of messages by message type
2) a visualization of this data for quick inspection. This visualization will be a DiskInventoryX style treemap breakdown of the above data.
Techincal Breakdowns
Simplifying assumptions (we can revisit these)
Bonus Cake visualization suggestion from andy: https://askubuntu.com/questions/73160/how-do-i-find-the-amount-of-free-space-on-my-hard-drive. This is a good possible alternative to the python based treemap