New computations and data

Resolves #69, resolves #68, resolves #67, resolves #62, resolves #53, resolves #47, resolves #27, and works on #22

Fixes #81, fixes #74

Todos:

[x] Split the pull request in more pull requests (perform a new pull request starting from the current state of the code without applying fixes, then commits the fixes mapping them to the comment I left in this pull request):
- Commons and Configuration Updates: analysers/commons/commons.py, analysers/conf/analysers/camunda.analysers.yml
- Experiments Processes Metrics: all the experiments metrics belonging to processes
- Experiments Constructs Metrics: all the experiments metrics belonging to constructs
- Trials Processes Metrics: all the trials metrics belonging to processes
- Trials Constructs Metrics: all the trials metrics belonging to constructs
[x] Some question referring to #22:
- We should discuss about the computation of combined standard deviation: http://www.inside-r.org/packages/cran/fishmethods/docs/combinevar. Note: refer to #83
- 95% Confidence Interval: For the metrics for which we compute the simple average, this should be the simple 95% confidence interval, e.g., faban_total_ops_mean, total_mean, writes_mean, reads_mean, size_mean, number_of_construct_instances_mean, number_of_process_instances_mean, execution_time_mean, throughput_mean, cpu_integral_mean, ram_integral_mean, faban_stat_mean, faban_duration_mean, faban_metric_value_mean, faban_mix_mean, faban_failures_mean, faban_successes_mean. We should discuss about other metrics (especially the Faban ones). Note: the 95% CI is always computed when we compute the mean, and is it saved in the schema with two values named <metric_name>_ci095_min and <metric_name>_ci095_max
[x] Referring to https://github.com/benchflow/data-transformers/issues/6#issuecomment-164890622, do we have a DESC order? Note: Refer to https://github.com/benchflow/benchflow/issues/32
[x] Map the comment in this pull request to new commits in the new pull requests opened according to the previous point, so that we can track which ones and where are fixed
[x] Missing the Faban response times metrics at experiment level
[x] Missing the Faban metrics on run info: transformer (duration), experiment (normal statistics on duration, metric_value [keeping the unit], passes [boolean AND])
[x] Read commit history and check everything
[x] Enable back Cassandra tests
[x] Add test for all the functions
[x] https://www.codacy.com/app/BenchFlow/analysers/pullRequest?prid=250923
[x] https://travis-ci.org/benchflow/analysers/builds/135278743
[x] https://github.com/benchflow/collectors/issues/65
[x] Where do you compute the percentiles on metrics like the throughput?
[x] Remove Alluxio from this pull, then perform a new pull just for Alluxio used in the right parts of the code
[x] Check that all the metrics we defined are here and correctly computed and tested
[x] File: analysers/commons/commons.py:L30-44
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why 0, 101?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme because numpy records percentiles from 0 to 100, so we miss percentile 100 without doing that range
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Lets discuss this face to face.
[x] File: analysers/commons/commons.py:L138-177
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> We should have Alluxio as well here
[x] File: analysers/commons/commons.py:L138-177
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> You should store both the test statistic and the p-value. http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.stats.levene.html#scipy.stats.levene
[x] File: analysers/commons/commons.py:L138-177
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> If you have at least two samples, you can perform the test. Check at least that before applying the computation.
[x] File: analysers/experiments/constructDuration.py:L1-79
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Select the distinct couple of construct_name AND construct_type, because you already have all the information on the actual existing couple, then you can avoid the next O(N^2) loop and just iterate linearly over the couple.
[x] File: analysers/experiments/constructDuration.py:L1-79
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> This should come from Alluxio as well
[x] File: analysers/experiments/cpu.py:L62-122
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> From host
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Done
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme the number of active cores I already compute at trial level, so at exp level I can take it from the trial table
[x] File: analysers/experiments/cpu.py:L62-122
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> From host
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Done
[x] File: analysers/experiments/cpu.py:L62-122
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> From the right table.
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Done
[x] File: analysers/experiments/executionTime.py:L1-72
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why do you filter twice? Just filter one and cache the result if you need it twice
[x] File: analysers/experiments/executionTime.py:L1-72
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Where is the total execution time independent from the actual process_definition_id?
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> :-1:
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme It was computed at trial level, it's labelled "all", so it's included here
[x] File: analysers/experiments/numberOfConstructInstances.py:L1-75
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> The same comments as per constructDuration.py
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> :-1:
[x] File: analysers/experiments/numberOfProcessInstances.py:L18-43
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Where is the metric for all the process instance?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme In the trial tables we have process_definition_id = all for the stats on all process instances. The loop in this function will work on that too. This applies for the other parts you pointed this out too
[x] File: analysers/experiments/processDuration.py:L14-44
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Where is the metric for all the process instance?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme At trial level it was computed and labelled "all", so now it's included at exp level.
[x] File: analysers/experiments/processDuration.py:L14-44
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Remove
[x] File: analysers/experiments/throughput.py:L18-43
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Where is the metric for all the process instance?
[x] File: analysers/trials/constructDuration.py:L1-98
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> The same comments as per constructDuration.py, checking before the loop for type and construct that are None
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme The reason why I check afterwards is because I can't save None types to the trial table, as those are part of the Primary Key. I do however want to select processes with type or name as None in the computations either way. Leaving a comment in the code about this
[x] File: analysers/trials/cpu.py:L50-68
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Remove
[x] File: analysers/trials/faban.py:L1-82
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why do you need to iterate over section after iterating over operation?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme I'm computing separately per section, those are different from the operations.
[x] File: analysers/trials/numberOfConstructInstances.py:L1-66
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> The same comments as per constructDuration.py, checking before the loop for type and construct that are None
[x] File: analysers/trials/numberOfConstructInstances.py:L1-66
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> You definitely need Alluxio because you apply different computations in different Script over the same data.
[x] File: analysers/trials/processDuration.py:L13-60
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why do you filter twice, in two different ways?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Not needed, fixed it
[x] File: analysers/trials/ram.py:L15-59
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why do you need a try....catch?
[x] File: analysers/trials/ram.py:L15-59
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Where do you define dataIntegral?
[x] File: analysers/trials/ram.py:L15-59
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Remove
[x] File: analysers/experiments/constructDuration.py:L1-85
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Set then at the beginning of the for, and then always use and refer to them
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Done
[x] File: analysers/experiments/constructDuration.py:L1-85
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> This check should be at the beginning, what happens if it is Unspecified for the computation on https://github.com/benchflow/analysers/pull/75/files#diff-dc119ca3b3977aad5d2b2e2434d76438R22
[x] File: analysers/experiments/constructDuration.py:L1-85
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Move the ttl setting: hours=1 in the STS configuration file.
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme This is a Python type, so writing it in the conf could create confusion. I might just remove this option later on, because it may not be needed. Leaving it as is for now, opening an issue.
[x] File: analysers/experiments/executionTime.py:L1-74
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why you also have a similar operation on line https://github.com/benchflow/analysers/pull/75/files#diff-d34221bb2ed6e9d12b428bb0b94b721aR28?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme computing mode is different, so I prep the RDD differently (mode we use Spark for everything, not numpy)
[x] General Comment
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> @Cerfoglg check in the first comment, the items that are done so that we can then check them together. I checked up to analysers/experiments/faban.py excluded.
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> @Cerfoglg Review completed, all the comments are in https://github.com/benchflow/analysers/pull/75#issue-158514448 (all of them, do not trust the completed label). Most of the times I left a comment only the first time I noticed something to improve, you should apply it in all the other scripts where something similar can be improved. I did not check that everything mentioned in the related issues is actually completed, for that we need to discuss in person. After your fix all of them ping me and we setup a meeting to close this giant pull request.
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme It should be all ready now, save for the cassandra tests, as there seem to be a problem with the connector, similarly to what was happening with the data transformers. For now, Cassandra tests are disabled. See issue #81
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> @Cerfoglg deal with https://github.com/benchflow/analysers/issues/81 before merging this pull request. Try to use --net="host" for both the cassandra and the spark container and other ideas you might have.
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> :+1:
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Fixed the build issue
[x] File: analysers/experiments/faban.py:L1-176
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Just map once and then take min and max
[x] File: analysers/experiments/faban.py:L1-176
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> This should be passes as part of the configuration
[x] File: analysers/experiments/faban.py:L1-176
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why the following is not executed in different Threads?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Since we use Spark so heavily using multiple threads doesn't matter, since when the Spark context is called it will not do any extra parallelization.
[x] File: analysers/experiments/numberOfConstructInstances.py:L1-81
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> This is often present in the code: why do you have two very similar filter + map here and on line https://github.com/benchflow/analysers/pull/75/files#diff-ff083bb8138c0743d316956959d94012R28?
[x] File: analysers/experiments/ram.py:L62-83
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> This is already set as parameter
[x] File: analysers/trials/constructDuration.py:L1-99
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why do you need a different RDD just for the mode computation?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme I use Spark for the mode, unlike the others that use numpy (numpy gives you a single mode value, I want all of them)
[x] File: analysers/trials/constructDuration.py:L1-99
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> I guess this part should be before the computation of metrics
[x] File: analysers/trials/executionTime.py:L1-96
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> You can compute smallest and largest in a single for
[x] File: analysers/trials/executionTime.py:L1-96
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Assign it to the first value of data outside the loop, then loop from the second value on. The same for the code computing largest
[x] File: analysers/trials/executionTime.py:L1-96
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why just None and not (None, None)?
[x] File: analysers/trials/faban.py:L1-82
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Just use the host and operation couples and add the aggregate one for all the operations.
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme I would need to go over the aggregate ones for all ops and sections if I do that. I can keep the looping like this, because all hosts have the same ops and sections, so I don't end up wasting time in this case.
[x] File: analysers/trials/faban.py:L1-82
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Use cassandraKeyspace
[x] File: analysers/trials/numberOfConstructInstances.py:L1-67
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> First check, then filter
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Same reason as the other times
[x] File: analysers/trials/throughput.py:L39-69
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> You should just use the execution time and the number of process instance metrics, not compute then again here
[x] File: analysers/trials/throughput.py:L74-98
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why do you need to select to_ignore?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme I need to filter processes to ignore
[x] File: analysers/experiments/cpu.py:L15-31
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why do you need this function? Why don't you refer to the container properties where you have (if less than the maximum) the number of cores? You should refer to the host only if you don't have the number of core information in the container properties.
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme I do, if the container properties doesn't have the number of cores I take it from the host. But keep in mind that Docker API returns a list of all cores on the machine regardless, with unused cores showing up with 0 usage. If I want to keep track of the indexes I need to save the whole array, and thus I need to know the number of cores on the host machine.
[x] File: analysers/trials/cpu.py:L23-31
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why try..except?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme For safety
[x] File: analysers/trials/cpu.py:L39-65
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> You have the same function also in analysers/experiments/cpu.py, so it should go in commons
[x] File: analysers/trials/executionTime.py:L15-41
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Add comments in the following code so that I can understand better what is going on. Moreover, assign the first value of the data array to both largest and smallest and then iterate the data array from the second element to update the min and max.
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme This is fixed
[x] File: analysers/trials/ram.py:L15-38
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Add comments in the following code so that I can understand better what is going on. I don't get for example the meaning of maxMemory < 1.
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Docker writes the max memory as 0 or -1 when unlimited memory is available to the container, thus we need to go to the host properties to see the memory. Commenting to specify.
[x] File: analysers/trials/throughput.py:L13-44
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Don't we have the execution time of each single process? We should maybe refer to that one here.
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme We are computing execution time with the start time and end time difference, so for singular processes it would be the time between the first instance of a process starting, and the end time of the last. Not sure if that means anything. It's best to use the overall execution time, so we know the amount of processes of a given type over the entire execution.
[x] File: analysers/commons/commons.py:L159-166
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> Why not 2?
<img border=0 src='https://avatars.githubusercontent.com/u/8819678?v=3' height=16 width=16'> @VincenzoFerme Can give a NaN when lower than 3, so will need at least 3 to have something meaningful here.
<img border=0 src='https://avatars.githubusercontent.com/u/7163641?v=3' height=16 width=16'> It should work with at least 2

benchflow / analysers

New computations and data #75