code coverage everywhere (js repos uses coveralls rather than codecov)

[victorb]

@ipfs/javascript-team why we're using coveralls and what's the must-have features there? Thinking we can just use cobertura coverage tools directly in jenkins, as long as we have a separate status check for the coverage and expose badges. This will remove the need of adding another secret and might be better as we have more control over when to reject something if coverage is under X etc.

[victorb]

Would be a relatively simple change to activate everwhere via Jenkins:

• Install cobertura plugin in Jenkins
• Make aegir output in cobertura (istanbul + nyc already support it, just adding another reporter)
• Run yarn coverage
• Execute coverage collection
• Output results into status check

[achingbrain]

Will it report coverage changes on PRs to the PR itself? That's something coveralls does that's quite useful.

[victorb]

@achingbrain I think a better idea is to report the code coverage via the status checks rather than via comments, to reduce noise. Idea is that "comments are for humans, status checks are for robots"

It'll look something like this:

Note the "continuous-integration/jenkins/code-coverage" line, which would then link to the coverage report. Then it's also easy for us to setup Github PRs to not be mergable if code coverage dropped (by X) with the PR applied to master

[achingbrain]

Yes, that's what I meant 😄

[magik6k]

Side note on status checks: it would be really useful to have a function for setting the statuses in jenkins-libs. It would allow go-ipfs job to set some statuses early for quick tests.

[victorb]

@magik6k now there is! Available in all pipelines:

githubNotify description: 'Linting in progress',  status: 'PENDING', context: 'continuous-integration/jenkins/linting'

Automatically infers sha ref and more, full docs are here: https://github.com/jenkinsci/pipeline-githubnotify-step-plugin

[victorb]

Should look into if cobertura supports diff-coverage, similar to this: https://codecov.io/gh/ipfs/go-ipfs/compare/ecb4095099d104d94f73c6f180e641e702824726...d8e6139c7546f61ba4608a9315fb4d614d8fd1e7/diff

Also to note is that go-projects seems to be using codecov rather than coveralls.

[Kubuxu]

Coverage for me is a bit more than just a status icon and metric to maximise In case of go-ipfs I can take a look diff coverage as @VictorBjelkholm pointed out.

I can also look at given file coverage with just one of two tests suits (unit tests or coverage, https://codecov.io/gh/ipfs/go-ipfs/src/2162f7e681b87e6b322712870f9f883bf9f9ec40/pin/set.go , top right of the file viewer to toggle one of them).

It reports also two statuses. One for global coverage change and the other for coverage of the diff itself. Both are useful in their own rights.

[victorb]

@Kubuxu thanks for the feedback here and IRC! Let's go with codecov for everything since that's what seems to cover everyone's need.

[victorb]

Started looking into codecov and two issues became apparent:

1) Need to have a codecov.yml in every repository. Would be nice to avoid having sync files across all repositories

2) Every project has it's unique token, so we can't just store one token and use that to upload for all projects. I'm not sure how to solve this. If you're doing public builds with either CircleCI or TravisCI, you don't need the token, so I've written a email to the support asking if we can get Jenkins to work without token as well.

[Kubuxu]

@VictorBjelkholm there is global config that can be used instead of codecov.yml but either way, we will run into problems with differences between JS and Go (different levels of required coverage as an example).

[victorb]

@Kubuxu seems to only be organization-wide and there is no way of controlling it via the API.

I think the biggest issue that is stopping this from moving forward, is that I still haven't found a good solution to how we can avoid the project-specific tokens.

[victorb]

The codecov support is unresponsive and I'm out of ideas so this is currently blocked until I figure out a way forward.

One idea was to write some tiny service that maps Github Repository => CodeCov token and call that service from Jenkins. Still haven't made sure if we can get the CodeCoverage token from the API, without that we're screwed.

[victorb]

Ok, found a way of getting this to work. We'll request the upload token in each build via a user account access token, then use that for the upload.

[Kubuxu]

@VictorBjelkholm do we have a way to handle secrets in Jenkins securely? I think currently there are ways for people with write access to the repo (not main branch) to extract some secrets.

[victorb]

@Kubuxu Guess it depends on what you mean. I think this has been mentioned before but I'll reiterate. We have a few protections in place for people to not be able to run whatever commands they want:

Changes from outside repositories won't run on CI unless approved by someone with write access to the repository.

Most projects (except go-ipfs and ipfs-companion I think) get their pipeline from ipfs/jenkins-libs#master, which only infrastructure, admin and ci teams have write access to. All changes happens via PRs as well. go-ipfs and ipfs-companion should move their pipelines there as well, as otherwise they miss this protection. Everyone with write access to go-ipfs can change the pipeline and jenkins will run it, as write access to a GitHub repository means trusted user.

Access to files and data outside of the pipeline needs to manually approved by a Jenkins administrator, same with arbitrary groovy-code, if the signature hasn't been approved before, it needs manual approval before running, even for write-access contributors.

This quarter we have plans on trying to setup more protections though, including having a security audit of the CI setup.

Happy to hear any suggestions on how we can make it better!