[CI] 1: important_packages for private code ; 2: faster CI using self hosted VM's

timotheecour commented 4 years ago

The main idea of this proposal is to provide a http callback mechanism to run CI tests. This has 2 applications that would use the same underlying approach.

extend important_packages to private projects

This would extend the scope of important_packages (implementation for nimble wide CI, as described in https://github.com/nim-lang/Nim/issues/8638) to also include private (non shared) code. The tests for private code would run in remote machines hosted by owner of private repository.

pkg "cligen", "nim c -o:cligenn -r cligen.nim" # a regular public package
pkg("myPrivatePkg", callback = "https://mycompany.com/nimcihandler") # a private package

the test result information that be reported back to nim's CI would be up to the remote package, including nothing at all (still useful for remote code to trigger their own remote notifications), or just succeed/fail, or timing info/logs. All that's needed is to host a service that listens on that callback url https://mycompany.com/nimcihandler ; hosting details are up to remote service, but could reuse open source nimble packages to make it easier to, say, host privately on AWS.

faster CI using self hosted VM's

this is basically a particular case of the previous point, where the code being run doesn't have to be private, and thus could include (parts of) nim's own test suite, eg to offload some of the work (or matrix of options/cpus/OS, eg nim cpp, linux i386 etc) to self hosted VM's under our control. This would allow increasing throughput of CI and making CI run faster, improving the PR experience / doing more thorough testing (eg enabling https://github.com/nim-lang/Nim/pull/13348 faster).

The syntax would be the same:

pkg "cligen", "nimble test") # a regular public package
pkg("myPrivatePkg", "nimble testRemote", callback = "https://mycompany.com/nimcihandler") # a private package
pkg("nim", "koch runCI", callback = "https://nim-lang.org/nimcihandler") # self hosted by https://nim-lang.org/

CI async callback protocol

when the nim CI runs important_packages:

if a callback is not specified, it runs the code as usual
else, it calls the callback via async http POST request (using newHttpClient + related apis), which triggers running the private project's tests on the specified remote server, and passing arguments such as the specified nim hash for the PR being tested, eg:
```
https://mycompany.com/nimcihandler?nimurl=https://github.com/nim-lang/Nim/commit/42dad3a836f7eed860f300e68b33d4c0b39bd1f4&nimci_sessionid=12345
# or rather, passing arguments as json payload
```
It then await's on the response (with a timeout) to tell whether private project's CI succeeded, with optionally other json output summarizing the test results, eg test output / running time etc. These shall be gathered in the final results for nim's own CI. The exact specifics could be debated, eg websocket would work too.

security concerns against malicious PR's

this callback approach could be made secure against malicious Nim PR's (eg ones that would cause nim compiler to start fork bombs, dump private code on some listening website, or cause other harm on the remote VM) by running the pkg CI in a sandbox; there are projects to do exactly that IIRC, including in Nim to some limited extent (eg nim playground https://play.nim-lang.org/)

The sandboxing logic to run the CI callback is up to the remote service to implement, but it could itself be based on an open source nimble project that would take care of implementing all those details once, so that a private could be as simple as:

# Package myPrivatePkg
version     = "0.1.2"
description = "some private code"
# Deps
requires    "nim >= 1.0.0"
requires    "nimcisandbox >= 1.0"

from pkg/nimcisandbox import execRemoteTest
task testRemote, "run remote CI tests":
  execRemoteTest(...) # runs tests on callback url, in a sandbox

note

an equivalent idea (just presented differently) is to move the callback logic as follows: myPrivatePkg has a public package counterpart hosted on github, eg https://github.com/mycompany/myproj which

pkg "cligen", "nimble test") # a regular public package
pkg("myPrivatePkgPublicCounterpart")

myPrivatePkgPublicCounterpart.nimble itself does the httpcallback POST request to a specified url, eg:

from pkg/nimcisandbox import execRemoteTest
task test, "run remote CI tests":
  execRemoteTest("https://mycompany.com/nimcihandler", getEnv("NIM_PR_HASH" )) # runs tests on callback url, in a sandbox

but this is less good as it requires a public interface for each private code being tested.

Araq commented 4 years ago

The benefits are not clear enough for me. What's the advantage over running a patched testament on a custom CI server?

timotheecour commented 4 years ago

What's the advantage over running a patched testament on a custom CI server?

the point is to trigger CI on every PR tested (as we currently do, ie every PR push triggers running tests for each important_packages), so that each registered package (private or public) can still be tested on every PR push. If someone merely runs a patched testament on a custom CI server, he'd have no way to trigger his patched testament whenever there's a push to an upstream nim PR; likewise a PR author would have no way to tell whether their PR broke some registered private code

Araq commented 3 years ago

The CIs are fast enough. It not an issue if you create 4 PRs of high quality instead of 30 PRs that nobody has the time and willingness to review.

nim-lang / RFCs