search

crossminer / scava

https://eclipse.org/scava/
Eclipse Public License 2.0
18 stars 13 forks source link

Selecting the right metrics for a complete dashboard #425

Closed borisbaldassari closed 4 years ago

borisbaldassari commented 4 years ago

We have an on-going discussion through emails to select the correct set of metrics in the analysis. As of today most graphics show empty data, we want to select only the metrics required to have a nice and complete dashboard while preserving the performance of the analysis.

The reference dashboard is the one maintained by Konstantinos: \ http://83.212.75.210/

The list of metrics currently selected (for the next round of analyses) can be found at \ https://github.com/crossminer/scava-scripts/blob/master/scava_create_task.pl

tdegueul commented 4 years ago

Regarding Rascal metrics:

I hope this is helpful, please hit me with any other question.

creat89 commented 4 years ago

I cannot completely comment on this aspect as I do not use the dashboard. Maybe @mhow2 could have more recommendations.

In all the cases, regarding NLP tools that I know that they have a dashboard and that you have not selected, I could recommend:

I might be wrong, but I think that the number of visualizations available is greater than the dashboards available. However, that doesn't mean that all the visualizations would generate nice graphs. And, as well, I do not know well the difference between the dashboards and the visualize. For example, I'm not sure there is severity dashboard, which for projects using Bugzilla could generate nice graphs with the different severity levels, e.g.:

borisbaldassari commented 4 years ago

Hi @tdegueul , all,

For dependencies, the dependencies dashboard apparently shows the dependencies as intended (though there are only Maven dependencies in the analyzed projects). All the other dependencies metrics do not have an associated visualization. I can create visualizations for the ones you picked in the script. Ok?

Well, no: the dashboard you link to contains only 3 projects (the 2 defaults + ee4jjersey). maybe it's because of the nature of the analysed projects, and we would have more data with different projects? In that case we'll just drop this very dashboard if it's not useful for our usecase. Thanks for the tip. Regarding the other visualisations, if it brings more insights and results from your component/data provider i'd be interested in showing them up. So, if you think they would provide some useful information, please create the associated vizs.. Thanks. :-)

Regarding API evolution, I think there is no visualization associated with trans.rascal.api.changedMethods yet (@valeriocos?). I would suggest to turn on the following metrics that have associated visualizations and show the evolution of API breaking and non-breaking changes over time for the analyzed projects: trans.rascal.api.numberOfChanges / trans.rascal.api.numberOfBreakingChanges.

Ok, so I'll add the numberOfChanges* metrics and remove the changeMethods one. Thanks. :-)

Thanks a lot for the quick and useful response Thomas, have a wonderful evening/day! :-)

borisbaldassari commented 4 years ago

Hi @creat89 , all,

In all the cases, regarding NLP tools that I know that they have a dashboard and that you have not selected, I could recommend:

Great! Thanks! :-) Hum.. Just to make sure that we're all aligned here: if we add these metrics, will the show up in a dashboard? @valeriocos do you know that?

I might be wrong, but I think that the number of visualizations available is greater than the dashboards available. However, that doesn't mean that all the visualizations would generate nice graphs. And, as well, I do not know well the difference between the dashboards and the visualize. For example, I'm not sure there is severity dashboard, which for projects using Bugzilla could generate nice graphs with the different severity levels, e.g.:

Euh.. I'm not sure to understand that. Of course not all metrics will produce a good viz. We have to select them according to that criterion (among others) I guess. AFAIU if we think that these metrics are worth it we could add them to the dashboard (and the analysis). I can add them to the list of computed metrics, @valeriocos WDYT? Can you add these to the dashboards?

Thank you all, have a good day! :-)

borisbaldassari commented 4 years ago

@valeriocos Following up on one of your (very useful) emails ;-)

Just to make sure that we're on the same page, and I'm not missing something.. Do you mean that if we add all these metrics we have a complete dashboard? (on top of the code-related metrics discussed with Thomas of course)

METRIC PROVIDER IDS

numberOfBreakingChanges.historic LCC-Java-Quartiles.historic DIT-Java-Quartiles.historic AIF-Java-Quartiles.historic MHF-Java.historic PF-Java.historic commitsovertime.CommitsOverTimeHistoricMetricProvider emotions.EmotionsHistoricMetricProvider sentiment.SentimentHistoricMetricProvider severity.SeverityHistoricMetricProvider severitybugstatus.SeverityBugStatusHistoricMetricProvider severityresponsetime.SeverityResponseTimeHistoricMetricProvider severitysentiment.SeveritySentimentHistoricMetricProvider sentiment.DocumentationSentimentHistoricMetricProvider severitysentiment.SeveritySentimentHistoricMetricProvider docker.dependencies docker.smells puppet.dependencies numberMavenDependencies.historic numberOSGiBundleDependencies.historic topics.TopicsHistoricMetricProvider users.UsersHistoricMetricProvider bugs.BugsHistoricMetricProvider status.StatusHistoricMetricProvider newbugs.NewBugsHistoricMetricProvider topics.TopicsHistoricMetricProvider users.UsersHistoricMetricProvider bugs.BugsHistoricMetricProvider status.StatusHistoricMetricProvider newbugs.NewBugsHistoricMetricProvider churnToday.historic

BTW they do not look like usual metric IDs. how is it that we have different IDs for the metrics? I don't get it. I guess I can map them to metrics in the instance/metrics.json .. right?

valeriocos commented 4 years ago

Sorry for the late reply! I missed this issue.

from https://github.com/crossminer/scava/issues/425#issuecomment-552392231

Regarding API evolution, I think there is no visualization associated with trans.rascal.api.changedMethods yet (@valeriocos?). I would suggest to turn on the following metrics that have associated visualizations and show the evolution of API breaking and non-breaking changes over time for the analyzed projects: trans.rascal.api.numberOfChanges / trans.rascal.api.numberOfBreakingChanges.

trans.rascal.api.changedMethods is not shown in the dashboard.

from https://github.com/crossminer/scava/issues/425#issuecomment-553026478

Great! Thanks! :-) Hum.. Just to make sure that we're all aligned here: if we add these metrics, will the show up in a dashboard? @valeriocos do you know that?

The metrics will be shown in the debug dashboard if they are exposed via visualization endpoints. If you want to add one or more visualizations to a specific dashboard, we can work together to add them. Do you have a deadline for this?

from https://github.com/crossminer/scava/issues/425#issuecomment-553026478

Euh.. I'm not sure to understand that. Of course not all metrics will produce a good viz. We have to select them according to that criterion (among others) I guess. AFAIU if we think that these metrics are worth it we could add them to the dashboard (and the analysis). I can add them to the list of computed metrics, @valeriocos WDYT? Can you add these to the dashboards?

Creating a dashboard (or adding visualizations to it) with metrics exposed via visualization endpoints is pretty easy. @borisbaldassari if you agree, we could have a call to explain how to create a dashboard/visualization and save it to Kibana (@tdegueul and I had a call of this kind in the past). Then I can take care of export it and include it to the docker-compose, WDYT?

from https://github.com/crossminer/scava/issues/425#issuecomment-553039072

BTW they do not look like usual metric IDs. how is it that we have different IDs for the metrics? I don't get it. I guess I can map them to metrics in the instance/metrics.json .. right?

I took these names from the project page within the SCAVA UI. For instance, if you access http://crossminer.bitergia.net:5601/#/project/configure/technologyepf, and click on metrics (within the grey box at the very end of the page), you will see the list of metrics.

mhow2 commented 4 years ago

BTW they do not look like usual metric IDs. how is it that we have different IDs for the metrics? I don't get it. I guess I can map them to metrics in the instance/metrics.json .. right?

Those are metric providers (who produces metrics, which in turn get others Ids) and the way they are displayed in the UI is by keeping only the last "dot".

org.eclipse.scava.metricprovider.historic.bugs.bugs.BugsHistoricMetricProvider is displayed as bugs.BugsHistoricMetricProvider for the sake of readability. BTW If you hoover the metric as follow, it will reveal the full MP ID: image

borisbaldassari commented 4 years ago

Hiho @valeriocos

Regarding API evolution, I think there is no visualization associated with trans.rascal.api.changedMethods yet (@valeriocos?). I would suggest to turn on the following metrics that have associated visualizations and show the evolution of API breaking and non-breaking changes over time for the analyzed projects: trans.rascal.api.numberOfChanges / trans.rascal.api.numberOfBreakingChanges. trans.rascal.api.changedMethods is not shown in the dashboard.

Removed it from the list.

Hum.. Just to make sure that we're all aligned here: if we add these metrics, will the show up in a dashboard? @valeriocos do you know that? The metrics will be shown in the debug dashboard if they are exposed via visualization endpoints. If you want to add one or more visualizations to a specific dashboard, we can work together to add them. Do you have a deadline for this?

The deadline is the one we all have, so I guess it is set to our next meeting. I'd be interested to learn how to do that however, so I don't bother you each and every time.. ;-)

Creating a dashboard (or adding visualizations to it) with metrics exposed via visualization endpoints is pretty easy. @borisbaldassari if you agree, we could have a call to explain how to create a dashboard/visualization and save it to Kibana (@tdegueul and I had a call of this kind in the past). Then I can take care of export it and include it to the docker-compose, WDYT?

Same as before: yes, I'm interested. We'll need to find some time for that, next slot for me is next friday. Would that fit your schedule?

I took these names from the project page within the SCAVA UI. For instance, if you access http://crossminer.bitergia.net:5601/#/project/configure/technologyepf, and click on metrics (within the grey box at the very end of the page), you will see the list of metrics.

Done, I've updated the list of metrics in the task creation script with the full list of full IDs. Thanks to @mhow2 to point me to the right direction too! ;-)

valeriocos commented 4 years ago

Hi @borisbaldassari , next Friday is perfect (and even the days before in case your schedule changes), thanks.

borisbaldassari commented 4 years ago

We consider that the current set of metrics, as described in https://github.com/crossminer/scava-scripts/blob/master/scava_create_task.pl, will fill up the dashboard with nice graphics. We can close the issue, thank you all!