lordofthejars
commented
6 years ago The idea of this comment is to enumerate metrics, put my comments on them, and also ask questions about when and how should this be done. I will notice that my concerns are purely from the point of view of QA so I can be missing some part of the importance of why these metrics are important, but I prefer to just ask it or question them instead of saying nothing.
a) Manual initiated from OSIO pipelines screens
For this metric the only thing that I see that can be useful is in case of knowing the preferences of the user and for example to know if they rerun builds manually instead of webhook, but in terms of QA I don’t see now the benefits.
I am not sure if Jenkins provides this data.
b) Number of times, 'view log' been clicked
What is the important thing of knowing if user has clicked to see the log of the build or not? Most of the time you click on view the log not because there is a failure but because you want to know how the build is going on and what it is doing/in which stage it is.
I am not sure if Jenkins provides this data.
c) Able to determine if a build log shows success / failure. If Failure, store the contents of the failures
You can get this result from build result, it is not necessary to check anything in log. This seems to be a good metric but at the end the build can fail because we have added a regression from Build team or because the project is failing because it contains a flaky test, so at the end what we are having here is something like a flaky metric since we can see that something is failing constantly, then take a look what’s happening and see that it is something not related to build team, which means time to analyze each and every failure detected by the metrics.
I think this data can be retrieved using Rest API of Jenkins.
d) an instance is to be idled, is actually getting idled ?
This is something that might be interesting since I have find some delays and so on. What I don’t know if this is provided by OSO or fabric8 or something developed by us, so I will need some background about it.
e) an instance is to be unriddled, is actually getting unriddled ?
Same as before
f) How many times web hooks are being received, store the source for these webhooks
Exactly the same as point a) In this case it can be used to do some statistics but in terms of QA I don’t see much benefits.
krishnapaparaju
pradeepto
joshuawilson
aslakknutsen
We would need to collect these metrics and feed to Zabbix. These metrics would help to understand operational side of things with clear visibility into quality of OSIO user experience.
[ ] Manual initiated from OSIO pipelines screens (& check if build got started with success)
[ ] Number of times, 'view log' been clicked (& check if build log show up with success)
[ ] Able to determine if a build log shows success / failure. If Failure, store the contents of the failures
[ ] An instance is to be idled, is actually getting idled ?
[ ] An instance is to be unidled, is actually getting unidled ?
[ ] How many times web hooks are being received, store the source for these webhooks (& check if build / required action got started with success)