support for build data tracking and analytics

[morungos]

I'm loving NCI, but I have a small suggestion which I think would add some great value without too much pain. What I'd really love would be the ability to chart, over builds and/or time, data points from builds, such as number of tests, percentage of code/branches/statements covered, and so on. I already have that data, generated in a JSON file by the build process. What I don't have, is a simply way to persist it, in the way that Jenkins does (for example).

In an ideal world, I'd be able to specify a path for a JSON (maybe XML also) which is generated during whatever commands get run, and which could more or less get thrown into the DB for that build. Then, it'd not be a horrible task to do some d3 charts for tracking metrics. I'd be happy to assist with contributions here, once we have data we can persist.

It would be easier in a Jenkins-type system, where the data isn't persisted anywhere but in the file system, so a simpler alternative would be to expose the build number in an environment variable when running commands. Then I could persist the output myself, and even do the analytics. So, maybe it's as simple as binding up NCI_PROJECT and NCI_BUILD when executing the project commands.

The second is probably the easier and more flexible solution, now I think on it. All sorts of interesting power is possible if the build commands could have some knowledge of the build number.

Anyway, thanks for the cool framework -- it already has helped us find code issues.

[vladimir-polyakov]

Hello, Stuart. Thanks for your attention, ideas and suggestions. I think, it is good place for plugin that can manage statistics, gather it, add ability to filter it and other functionality. But we're working on a new nci ui right now and I think it'll be not efficiently to add some functionality with presenting here (charts and e.t.c).

What about exposing build number and project. I think it's easy to implement, but it's question for nci core, @okv, what do you think?

[okv]

Hi, there. I'm glad to hear that nci is useful for you.

When I'm thinking about nci functionality, charts and statistics look like cool and exciting things, but from other hand it's hard start working on it because of many reasons (persistence, visualization, etc). This issue good point to start.

@morungos generating statistic data during build process and storing it in files may work in particular build process but it's not suitable as general solution (e.g. when we are building on remote nodes via ssh or docker, workspace with files also will be remote). Solution mentioned by @vladimir-polyakov (separate plugin for managing statistics) should work, but such plugin needs to be designed properly.

All sorts of interesting power is possible if the build commands could have some knowledge of the build number.

Absolutely agree with that, so I'll try to provide bunch of environment variables in the nearest future.

@morungos could you tell us more (maybe provide some examples) about current statistic data aggregated by you? which tools are you using? Maybe it'll help (as one of use cases) us to design statistic plugin.

[morungos]

The environment variables would be a huge step forward for me.

The statistics I'm thinking of:

• Number of tests taken/passed/failed (a cumulative line chart)
• Percentage of the code base covered by coverage tests (I'm using istanbul)

I used to use the Jenkins chart plugin for Java-based projects, and that might be a good start: https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin -- a chart turned out to be a very good way of seeing the impact of code and test changes, a small change could result in a significant loss of code coverage, for example.

Don't think this needs to be high tech. If we could have some way to have a table of a set of numeric values by build number, the config file can just specify which should be charted. You could make it a bit more flexible via some angular-expressions magic to deeply pull values from JSON files. In the simplest version, expecting this to be done by the user's build scripts would be fine. It only really needs something like NCI_BUILD to be set and that could write static HTML+d3+data which does the rest.

[okv]

Thanks for your information. I'll let you know when build environment variables will be implemented. I also will think about extending nci api for being able to store arbitrary data to db via plugins (or provide some specific statistic functionality by nci core), but that will require more time.

[okv]

Some basic environment variables provided since nci 1.0.7.

[morungos]

Very nice and many thanks. This will make it much easier for us to generate some nice charts. I'll keep you in the loop with it and if I can make anything useful as a pull request, I will.

[okv]

Glad to help, will be waiting for your news

On 9 Dec 2016, at 22:30, Stuart Watt notifications@github.com wrote:

Very nice and many thanks. This will make it much easier for us to generate some nice charts. I'll keep you in the loop with it and if I can make anything useful as a pull request, I will.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.