swirlds-automation
commented
1 year ago This issue is old, but I think it's still relevant. It would be really nice if we had a tool that could give us a breakdown of where all data being written to the signed state file and originated. This is a little complex, since we also need to account for the database files written by virtual map.
There are two ways we could go about this.
1) an offline tool 2) collect statistics online as we are writing the state
I actually think option 2 might be straightforward to accomplish. But there are a lot of design questions that need to be answered first. author:cody-littley, createdAt:2022-11-03T20:30:31Z, updatedAt=2022-11-03T20:30:31Z
poulok
Recently the platform team has added a statistic to the MainNetStats CSV file to allow DevOps to monitor the SignedState.swh growth.
From the time between 2021-02-04 to 2021-03-11 (35 days) the SignedState.swh file has grown 6MB. This growth rate is roughly 172KB per day. The growth according the the chart looks to be linear.
After discussing with @lpetrovic05 there does seem to be some rudimentary tools to inspect state files.
I suggest we evaluate putting time into polishing those tools so we can compare the state files from the corresponding dates above to determine where the majority of the growth exists and if it needs to be optimized.
Kibana Link&_a=(description:'',filters:!(),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),panels:!((embeddableConfig:(),gridData:(h:15,i:'1',w:24,x:0,y:0),id:f65676b0-2968-11e9-8f19-a925a729ed40,panelIndex:'1',type:visualization,version:'6.6.0'),(embeddableConfig:(),gridData:(h:15,i:'2',w:24,x:24,y:0),id:'9a681b90-2974-11e9-8f19-a925a729ed40',panelIndex:'2',type:visualization,version:'6.6.0'),(embeddableConfig:(),gridData:(h:15,i:'3',w:24,x:0,y:15),id:e3684f40-2974-11e9-8f19-a925a729ed40,panelIndex:'3',type:visualization,version:'6.6.0'),(embeddableConfig:(),gridData:(h:15,i:'4',w:24,x:24,y:15),id:'14190fd0-2975-11e9-8f19-a925a729ed40',panelIndex:'4',type:visualization,version:'6.6.0'),(embeddableConfig:(),gridData:(h:15,i:'5',w:24,x:0,y:30),id:'3501d6e0-6254-11eb-8cee-adc17dc434c6',panelIndex:'5',type:visualization,version:'6.8.9'),(embeddableConfig:(),gridData:(h:15,i:'6',w:24,x:24,y:30),id:'7b88dcb0-67c4-11eb-8cee-adc17dc434c6',panelIndex:'6',type:visualization,version:'6.8.9'),(embeddableConfig:(),gridData:(h:15,i:'7',w:24,x:0,y:45),id:'8f6c1c10-67c4-11eb-8cee-adc17dc434c6',panelIndex:'7',type:visualization,version:'6.8.9')),query:(language:lucene,query:''),timeRestore:!f,title:'mainnet%20important',viewMode:view))