search

librecores / librecores-ci-jenkins-server

LibreCores Continuous Integration
https://ci.librecores.org/
MIT License
36 stars 8 forks source link

Easier LibreCores CI Slave Setup #24

Open imphil opened 8 years ago

imphil commented 8 years ago

Overall Goal

For LibreCores CI, we want to give users the ability to easily hook their machines into the CI system to run CI on their own project.

Why?

prerequisite: User "activates CI" for his project on librecores: essentially a web page on LibreCores (CI) where CI is enabled for a given project.

  1. User downloads a installable package from the LibreCores CI page
  2. User installs this package on his/her machine. The installation results in a daemon being installed.
  3. As part of the installation process, the user registers this node to build "CI for project XYZ on LibreCores"
  4. Jenkins master registers required build triggers with Git or librecores (details to be figured out)

Now whenever a build trigger happens, the node builds & tests the code, and uploads the results (most likely no build artifacts for license reasons) to the Jenkins master.

Constraints

oleg-nenashev commented 8 years ago

Which packages do we need? RPM/deb only?

imphil commented 8 years ago

I guess for a first version, just a script would be sufficient. While I in general don't like the curl https://example.com/install.sh | sh way of doing things, its a sufficiently simple starting point

Houkime commented 6 years ago

I think the "prerequisite" of having a project on Librecores is a bit too restrictive. For dev tests and such it probably would be better to do it in a way:

  1. User fetches a specialized mostly-preconfigured "LCCI-slave" Docker container. (Yep, no real installer needed)
  2. Special Jenkins plugin provides GUI for slave, in which user types "ci.librecores.org" or some specific IP as a desired master and can also paste a link to his project page if he wants to restrict usage or deploy only specific tools. It also should be possible to restrict max resources usage. Restriction is optional, a man can just donate librecores a temporary slave with comfortable for him configs if he wants to.
  3. Then the plugin regularly polls Jenkins master for jobs over https and does them.

It is also very convenient for development as you can have LCCI master and LCCI slave on the same machine in separate Docker containers and test really any possible interaction scenario with ease.

Houkime commented 6 years ago

It also might prove beneficial to use Tor instead of https for master-slave connection:

Pros:

  1. No need for certificate authorities or self-signing stuff. "It just works"
  2. Increased security... like... I don't even know what can be safer nowadays.
  3. Tor is specially designed to punch through NATs, local censorship and whatever goes in the way.
  4. User can hide his IP so LCCI master can't communicate without him polling even if it wants to.

Cons:

  1. Slower speed and increased time awaiting for tunnel. However it shouldn't be a real problem for LCCI tasks where files to send are small, and on-node calculation time is long.
wallento commented 6 years ago

@imphil Actually that is what our command line tool is supposed to do. It currently requires the user to provide a yaml file that is supposed to be downloaded from LCCI. The command line tool itself is a python tool that then does the docker stuff in background. It is not a daemon but a wrapper around docker. The yaml file contains the credentials, plus it must be changed to point to the supposed tools installation path. I still need to write a README for it.

wallento commented 6 years ago

Regarding communication it uses plain https to talk to LCCI, correct @oleg-nenashev (method 4)?

wallento commented 6 years ago

@Houkime This is kind of a build in feature of jenkins. What it would need is the ability to change the remote URL which is currently not possible, but can be easily added.