Data collection: docker stats and host-proc

[0xFugue]

• this PR implements two data collection frameworks: data collection via docker stats and data collection via host /proc/

• users can now chose among three data collection frameworks: cadvisor, host-proc, container-proc.

• can collect data on wakunodes inside dockers, wakunodes in the host, or any combination thereof

• docker stat collection is at container granularity. /proc collection is at per-wakunode granularity.

• collects CPU, Memory, Network and Block IO data at regular intervals (1s default)

• can handle multiple wakunodes per container

• requires sudo for block i/o measurements

[0xFugue]

run.sh is indeed getting messy: we are doing quite a few things outside kurtosis and also have 3(+1) data collection pathways. Once we zero in on one collection framework and move things to kurtosis, I suspect almost 2/3rd will go away.

We currently record the inputs to the cpu utilisation calculations ( instantaneous per-user/system user time and sys time in clock ticks). The actual cpu utilisation will involve taking two samples of sys-wide cpu usage and per-process cpu usage, and use the difference to compute the fraction of cpu used by the process wrt total sys-wide cpu used; this computation is currently done in the analysis module, after the runs are over. The TODO: is to see if I can push the computations into the measurement if they aren't heavy.

[Daimakaimura]

I see you that in order to to monitor container level processes you are accessing the /proc files in the host and finding the mapping of the PIDs running in the container right? Then as far as I understand this should be the same to running the shell probe within the containers. Since we are using the same /proc files then the metrics should be the same, hence no need for the container sampling approach whatsoever. Have you tested this at scale? At what number of containers/nodes the 1s sampling cannot be sustained anymore? The specs of the machine you are running it as a Threadripper with 32 core with 192G of RAM right? I assume that for every second you are getting 1 sample for every node, so in a 60s simulation with 100 nodes (no matter the number of containers) the output will have 6000 samples right? Have you verified you are actually getting the right number of samples at the end? If this works as I understand I m totally up for ditching the other method (even breaking the containers isolation).

[0xFugue]

Yes, I am mapping waku pids to containers and pick up the most stats from the host /proc . I pushed host-proc upto 100 nodes on my laptop and 256 on Corey's machine (one node per container; 512 crashes the system). Each epoch sticks to sub-second collection deadlines so far, for up to 256 nodes. Yes, the log has data from every node for each epoch. Each epoch currently costs us ~20ms. So for 10k nodes, I am cautiously optimistic that an epoch would cost us around 1sec. Adding some statistical buffer, 2 sec sampling interval for 10k seems very realistic; but we do need real data to support these machinations of course. I have not (yet) pushed it to 1000s range with multi-nodes: waiting for Corey's machine to be back online.

But I would like you to continue with the container-proc approach; I think /proc inside the container probably does better accounting. So let's continue to do the 3 + docker stats collections, and decide on the suitable way once we have all the data-collectors running on the same machine. I would like to make sure that all collections give out similar data for few hundred nodes in mono-containers first before zero'ng in on any approach.

[Daimakaimura]

My part is done and we should verify that both provide similar results, which they should as we are gathering the same data. If they not match, there is a mistake somewhere in either implementation. But regarding the deliverable framework, I dont think we need both approaches gathering metrics. Just let s make out life a bit easier. Of course, this is also assuming that your method doesnt scale worse which I highly doubt. Do you have the csv output of any of those runs you mentioned?

[0xFugue]

We are on the same page: we need to keep only one measurement framework for the final deliverable. I agree that my framework keeps time, scales well, has low overhead, and 'seems' to collect the correct data; but I just would like more evidence to bolster the final claim before stopping the other efforts. And it is not clear if container /proc/1/ and host /proc// will be in sync: I'll look at the pending PR and lets discuss after that.

For now, I just set up couple of 512/1000 node (multi-node) runs, and will merge once they are successful.

[0xFugue]

The host-proc can clock 2000/8 (#wakunodes/#containers) on Corey's machine with avg data collection time per epoch at ~450 msecs; each epoch has data from 2000 wakunodes. That gives us more than ample headspace to grow. Merging.