Analysis multinode refactor

AlbertoSoutullo commented 1 year ago

I am updating my analysis container with the following changes:

You can set now which metrics you want to get from prometheus, as well as some plotting options like title, labels, convert bytes to mbytes and so on
- Also, you can group several matrics in the same subplot
Metrics needed for running a standard simulation are now enabled by default in .toml files and also config.json
Added disk usage to previous plot
Added summary as title to plot
build.sh now also builds analysis container
analysiscontainer only needs one mount point
Added more tests to analysis.
analysis README.md updated
Now you can select which script to run from the container, depending on which analysis we want to use
Cleaned and refactored code a little bit

Looks like lots of changes, but most of them are test files and config file changes.

Important information: Thanks (a lot) to @vpavlin , we know that cAdvisor is behaving correctly when it comes to metrics like 'container_fs_reads_bytes_total' or 'container_fs_writes_bytes_total'.

From his explanation: "cAdvisor gets reads_bytes and writes_bytes from cgroup file io.stat, which is empty if there are no fs reads or writes by any process. Once anything is written or read, the io.stat gets proper content (potentially with 0 for either of the metrics) and cadvisor starts reporting the metric for the particular container. Second part of this is that even if there is some write (like directory creation or touch on file), it may not be actually written to disk as long as it fits into memory page (8kB), and you need sync to happen for the actual disk write to take place."

I ran accross this because my simulations were small, and sometimes I indeed ran accross that problem. It shouldn't happen in big simulations (or with more protocols activated). We assume that the value of those metrics, if we get an empty return, is zero, since the node did little-to none disk usage.

I encourage you to check the same in your analysis.

Daimakaimura commented 1 year ago

Execution and analysis work fine in my machine after running build.sh and run.sh cadivsor with the default .json config. To run the corresponding analysis I did:

docker run --network "host" -v $(pwd)/wakurtosis_logs:/simulation_data/ --add-host=host.docker.internal:host-gateway analysis ./src/main.py -p 32803

Which runs the analysis script and generates the analysis.pdf figure inside ./wakurtosis_logs/:

analysis.pdf

I've tried to run the tests and I am not sure if it is working:

docker run --network "host" -v $(pwd)/wakurtosis_logs:/simulation_data/ --add-host=host.docker.internal:host-gateway analysis -m unittest discover -s tests -p "*.py"% 

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Other than that, all good.

AlbertoSoutullo commented 1 year ago

Execution and analysis work fine in my machine after running build.sh and run.sh cadivsor with the default .json config. To run the corresponding analysis I did:

docker run --network "host" -v $(pwd)/wakurtosis_logs:/simulation_data/ --add-host=host.docker.internal:host-gateway analysis ./src/main.py -p 32803

Which runs the analysis script and generates the analysis.pdf figure inside ./wakurtosis_logs/:

analysis.pdf

I've tried to run the tests and I am not sure if it is working:
docker run --network "host" -v $(pwd)/wakurtosis_logs:/simulation_data/ --add-host=host.docker.internal:host-gateway analysis -m unittest discover -s tests -p "*.py"% 

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Other than that, all good.

Uh that is weird. Can it be that there is a % at the end of your line?

It should otuput something like:

docker run --network "host" -v $(pwd)/wakurtosis_logs:/simulation_data/ --add-host=host.docker.internal:host-gateway analysis -m unittest discover -s tests -p "*.py"
[09-05-2023 11:30:32.059] [WLS-ANALYSIS] Started
.......[09-05-2023 11:30:33.081] [WLS-ANALYSIS] 3 of 3 messages delivered. Lost: 0. Delivery rate 100.0

.
.
.
.
..............test1 41234
[09-05-2023 11:30:33.096] [WLS-ANALYSIS] Loaded 1 messages.
.......[09-05-2023 11:30:33.101] [WLS-ANALYSIS] Loaded nodes topics from toml files
.
----------------------------------------------------------------------
Ran 33 tests in 0.021s

OK

We are running a docker image, it is impossible that we are getting different test outputs...

Daimakaimura commented 1 year ago

Oh dear. Yeah, that was the problem. Now it works:

docker run --network "host" -v $(pwd)/wakurtosis_logs:/simulation_data/ --add-host=host.docker.internal:host-gateway analysis -m unittest discover -s tests -p "*.py" 
[09-05-2023 11:41:20.310] [WLS-ANALYSIS] Started
.......[09-05-2023 11:41:21.167] [WLS-ANALYSIS] 3 of 3 messages delivered. Lost: 0. Delivery rate 100.0

.
.
.
.
..............test1 41234
[09-05-2023 11:41:21.182] [WLS-ANALYSIS] Loaded 1 messages.
.......[09-05-2023 11:41:21.186] [WLS-ANALYSIS] Loaded nodes topics from toml files
.
----------------------------------------------------------------------
Ran 33 tests in 0.020s

OK

Daimakaimura commented 1 year ago

Oh dear. Yeah, that was the problem. Now it works:

docker run --network "host" -v $(pwd)/wakurtosis_logs:/simulation_data/ --add-host=host.docker.internal:host-gateway analysis -m unittest discover -s tests -p "*.py" 
[09-05-2023 11:41:20.310] [WLS-ANALYSIS] Started
.......[09-05-2023 11:41:21.167] [WLS-ANALYSIS] 3 of 3 messages delivered. Lost: 0. Delivery rate 100.0

.
.
.
.
..............test1 41234
[09-05-2023 11:41:21.182] [WLS-ANALYSIS] Loaded 1 messages.
.......[09-05-2023 11:41:21.186] [WLS-ANALYSIS] Loaded nodes topics from toml files
.
----------------------------------------------------------------------
Ran 33 tests in 0.020s

OK

vacp2p / wakurtosis

Analysis multinode refactor #113