dstats/host-proc analysis

[0xFugue]

Issue 121 (#121 ) lists the tasks undertaken in this PR.

Please run the PR on all measurement infras (cadvisor, container-proc, dstats and host-proc). In all the runs, the combined comparison plots are generated under wakurtosis_log/analysis.pdf

dstats and host-proc have a hoard of other plots as well, feel free to explore then. All the plots are selectable, and can be generated by adding the appropriate entries to config.json lists. the analysis generates all the plots requested, and puts the combined, comparison plots under wakurtosis_log/analysis.pdf.

solves #121 #122 , #119 , #110

[Daimakaimura]

@0xFugue I am getting similar results to @AlbertoSoutullo:

• cadvisor works
• container-proc works
• dstats gives an error:

Simulation ended with code 0 Results in ./wakurtosis_logs Simulation took 26 + 61 = 87 secs dstats: WLS finished: stopping the docker monitor 93040 Dumping Kurtosis logs dstats: copying the dstats data Successfully copied 6.14kB to /home/daimakaimura/Projects/AnalysisMasterMerge/wakurtosis_logs Successfully copied 644kB to /home/daimakaimura/Projects/AnalysisMasterMerge/wakurtosis_logs python: can't open file '/analysis/src/hproc.py': [Errno 2] No such file or directory Error response from daemon: Could not find the file /analysis/output-dstats-compare.pdf in container dstats Done.

• host-prockeps waiting forever in host-proc: signalling the monitor

[0xFugue]

This PR is indeed huge and I cannot go into detail about everything...

But I can leave here some comments:

* Clean lines that are not used unless they act as some kind of "documentation"

some of the comments are alternate ways of doing the same thing, but not yet completely ruled out. so kinda serves as revisit tag.

* Some WLS logs are being printed, I think this should not be displayed in `run.sh`

the output are now suppressed

* The data should be similar, just out of curiosity, why did you build from the beggining the plotting part?

I assume you mean the columns in the df? the columns are n't similar for two reasons : 0) docker stats params are fixed, host-proc is fully custom made and is more flexible. 1) the separators are chosen so that they minimise pre-processing in respective data frames. the final, pre-processed file is very, very similar, barring the custom columns like ss/ds tack size etc. let me know if I misunderstood you.

* `cAdvisor`, `container-proc` and `dstats` works fine for me, but `host-proc` gets stuck in `signalling the monitor`. Then when I cancel that, I got a `sudo: a password is required`. Was asking for the sudo password intentional?

Let me give a longer reply to this in Jordi's thread: he has the same issue. In short, it is a documentation issue.

Then, I am already noticing some differences between the different analysis... And we are talkinb about 10 nodes 1 minute simulation... Lets see how we deal with this.

this is indeed a cause for concern. let me check dig more info on how and where the fleet nodes collect data.

[0xFugue]

on dstats, have u rebuilt the analysis-module? hproc.py is added to the docker in this PR and is now part of the new docker build.

on host-proc, the issue is with the asynchronous sudo: the sudo prompt is tied to current terminal, but the actual sudo process is n background; this means a background process (host-proc-helper.py) is waiting on sudo while the foreground chugs along until it wait on wls. now, wls will not start host-proc-helper is ready. so the whole sim is waiting forever on a password prompt!

I now have modified the run.sh to make sure that the sudo prompt is now synchronous: that is, the run will now wait synchronously for password and will be stuck on the password prompt until it gets the password (or timeout, depending on local sudo config). and then will fork a process to run the host-proc-helper.

[Daimakaimura]

No, forgot to run the build step. Now dstats works but host-proc doesnt seem to be working (I have pulled your last chagnes). This is what I get:

./monitoring/host-proc/host-proc-helper.sh: 40: kill: No such process

host-proc: updating the owner of logs: 1000, 1000

Also, I think it would be nice if you could follow the same plotting style / colours that we already have. Just copy the styling from the analysis module.

[0xFugue]

am glad dstats works fine now, cheers.

so the host-proc-helper is failing in ur machine . could you send me the log at monitoring/host-proc/stats/docker-proc-log.out and monitoring/host-proc/stats/docker-proc.out?

[Daimakaimura]

I just realised going through your PR, why is there is another analysis.py script in /monitoring/host-proc? Shouldnt that be inside the analysis module? Also, regarding the the plotting, I've already mentioned to keep the same styling we have at the moment but can you please also make sure that by default you output the same metrics than in the cadvisor and container-proccases? I just would like to have the same plot (at least by default) whenever we run either branch.

[Daimakaimura]

I spent some time with the PR by now and I m still having trouble following what is going on. I think we should focus on making sure that dstats is properly integrated with cadvisor and container-proc, that the plots are also alike and that we outputting the same data (ie adding propagation time and injection times into dstats). Once this is done, we can switch to host-proc

[0xFugue]

that analysis is the original hproc. removed it and pushed the changes. propagation time computation is reused/same across all our analysis: idea of the compare plot is to compare things that differ. if u check, cadvisor does NOT plot it either i reckon.

the plots are done at the base class and it doesn't matter which metric framework the data is pulled from. the data format vagaries are handled at the child class, so The default is at wakurtosis/analysis.pdf, and it is the same plot cadvisor puts in the same place.

[AlbertoSoutullo]

I am still having issues with host-proc:

I put the password, but looks like it gets locked there.

[0xFugue]

from the output, both host-proc and WLS are now running side-by-side (as they should) and host-proc is waiting for WLS to finish. How long did u set the sim time for? the progress of host-proc is logged at monitoring/host-proc/stats/docker-proc-log.out. Can you share it, cheers.

[AlbertoSoutullo]

from the output, both host-proc and WLS are now running side-by-side (as they should) and host-proc is waiting for WLS to finish. How long did u set the sim time for? the progress of host-proc is logged at monitoring/host-proc/stats/docker-proc-log.out. Can you share it, cheers.

Are you enabling the venv before calling your scripts?

[0xFugue]

u need to run the build.sh before. that should work, let me know if it doesn't.

[AlbertoSoutullo]

u need to run the build.sh before. that should work, let me know if it doesn't.

I did.

Where are you activating that venv?

monitoring/host-proc/host-proc-helper.sh. python3 binary from the venv is directly used to invoke the host-proc-helper.py.

[AlbertoSoutullo]

u need to run the build.sh before. that should work, let me know if it doesn't.

I did.

Where are you activating that venv?

monitoring/host-proc/host-proc-helper.sh. python3 binary from the venv is directly used to invoke the host-proc-helper.py.

I detected the problem. We assume that the user has python3-venv. We should add this to build.sh, in the apt-get install line.

Other than that, LGTM

[0xFugue]

I detected the problem. We assume that the user has python3-venv. We should add this to build.sh, in the apt-get install line.

Other than that, LGTM

great. good point for non-Ubuntu/*NIX sys', let me add it to build.sh.