Remote runners

[exsilium]

Hi! I Wanted to understand if currently strider CD can be ran in a distributed model where the CD server (or coordinator) is hosted separately from the army of runners? I see there are some repositories concerning queens and drones but I haven't seen any comprehensive amount of documentation about the usecase or how to use them - they don't seem to be strider plugins (or are they?)

In short, I'd like to provision a strider cd which is connected to my repositories and instead of simple runner running the process on the same host, I'd like to offload the actual testing and possible deployment to remote server(s) where the results are fed back to the central strider installation.

Thanks!

[niallo]

Hi there.

Strider comes bundled with a simple worker implementation, but runners are pluggable. Check out the new Docker runner: https://github.com/Strider-CD/strider-docker-runner.

This can distributed jobs over the network.

[exsilium]

Thanks for your reply. I did take a look at strider-docker-runner but I guess I failed to mention that the remote "servers" or endpoints are extremely light-weight (in many cases embedded environments with node.js stack)

Without knowing the details of Docker containers I'm not sure if that would be something that would work for me. I will continue to investigate.

Bamboo remote agents fail also because of the Java requirements Gitlab-CI has gcr but forces the use of Gitlab

[niallo]

You can always modify an existing runner to work in your super lightweight environments. Sounds like you have very custom requirements.

On Tue, Sep 16, 2014 at 8:02 AM, Sten Feldman notifications@github.com wrote:

Thanks for your reply. I did take a look at strider-docker-runner but I guess I failed to mention that the remote "servers" or endpoints are extremely light-weight (in many cases embedded environments with node.js stack)

Without knowing the details of Docker containers I'm not sure if that would be something that would work for me. I will continue to investigate.

Bamboo remote agents fail also because of the Java requirements Gitlab-CI has gcr but forces the use of Gitlab

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/571#issuecomment-55756535.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[exsilium]

I'm aware of that, I'm really assessing to understand if a problem I'm having is already solved or not. My original question was fueled by the repos strider-remote-worker & strider-remote-drone as according to the repo documentation it seemed the thinking which I was looking for but there lacked detailed information about it.

Just for a background concerning the custom requirements part - there are more than 3 million Raspberry Pi's sold with Intel also entering the market with Intel Galileo and Galileo2 boards. This is excluding the countless other ARM based boards previously out there that basically are microservers running a Linux stack. Somehow I think that it will be only a matter of time when people need a good way of testing and updating their thermostats in a new way :)

[niallo]

If it's just a Linux stack then it should be able to run docker. Docker integration is really where we've seen the interest, rather than a custom thing.

On Tuesday, September 16, 2014, Sten Feldman notifications@github.com wrote:

I'm aware of that, I'm really assessing to understand if a problem I'm having is already solved or not. My original question was fueled by the repos strider-remote-worker & strider-remote-drone as according to the repo documentation it seemed the thinking which I was looking for but there lacked detailed information about it.

Just for a background concerning the custom requirements part - there are more than 3 million Raspberry Pi's sold with Intel also entering the market with Intel Galileo and Galileo2 boards. This is excluding the countless other ARM based boards previously out there that basically are microservers running a Linux stack. Somehow I think that it will be only a matter of time when people need a good way of testing and updating their thermostats in a new way :)

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/571#issuecomment-55759913.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[kfatehi]

@exsilium the ARM devices use case is a valid one that i too share. A remote test is not hard to achieve. In theory one could take strider-ssh-deploy and modify it slightly for use in the test phase. If this will suit your requirements i am happy to take a look.

Unfortunately i dont know the state of the other plugins you mention, perhaps their authors would, or @jaredly maybe knows about them.

[niallo]

Looks like Docker already runs fine on Raspberry Pi: http://sc5.io/blog/2014/07/a-private-raspberry-pi-cloud-with-arm-docker/

On Tue, Sep 16, 2014 at 10:25 AM, Keyvan Fatehi notifications@github.com wrote:

Reopened #571 https://github.com/Strider-CD/strider/issues/571.

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/571#event-166182159.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[kfatehi]

As awesome as that is, and as much as I love Docker and LXC, i am suspicious of this working or allowing some of the things an ARM device can do. Im thinking of GPIO and camera access... Although actually i take that back, you should be able to mount GPIO virtual directory as a volume. Camera is just a binary, i dont think you'd need to volume mount that. Super interesting, i will need to try it later on.

[niallo]

You can do all sorts of stuff with Docker privileged mode and setting individual Linux capabilities. GPIO, cameras, USB devices etc are just UNIX device files :) That's the beauty of it.

On Tue, Sep 16, 2014 at 10:46 AM, Keyvan Fatehi notifications@github.com wrote:

As awesome as that is, and as much as I love Docker and LXC, i am suspicious of this working or allowing some of the things an ARM device can do. Im thinking of GPIO and camera access... Although actually i take that back, you should be able to mount GPIO virtual directory as a volume. Camera is just a binary, i dont think you'd need to volume mount that. Super interesting, i will need to try it later on.

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/571#issuecomment-55783117.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[exsilium]

@keyvanfatehi Glad to have found a fellow thinker! I experimented with the strider-ssh-deploy and although it works nicely it seems like a... hack... I think the closest I've come in terms of a complete solution at the moment is Gitlab-CI with unofficial runner 'gcr' that is a light-weight node.js runner. The not so good side is that Gitlab-CI enforces Gitlab which itself forces you to host the code and both are quite heavy applications. Ruby deployment with bundled gems takes 750MB :D This is not a problem per se but I would prefer much more if Strider-CD (which beautifully hooks into github and bitbucket and does only what it should) would have this thin remote runner capability. But yes, through the SSH deploy plugin basically one can do anything.

@niallo Thanks for the link! I was aware that resin.io had ported Docker but still need to understand how it actually performs. Will try it out.

I do get the beauty of Docker but it still has it's limits (think how to get it to run on Arduino Yun, although the SSH deploy plugin trick would work here as well). If StriderCD would define a clear Server->Client remote runner API, it would make implementing different thin runners much more trivial - just write a runner on different technology stack. I understand this is available and doable today but it requires much more thought into implementing the server and the client, especially if one is at all concerned about basic security and making the Server side plugin more generic.

Using the terminology in strider-remote-worker the Queen would be the server with a documented API versions and the Drones or thin-runners can register to the server via predefined API-key, registering capabilities and then you can bind the drones to a specific project or via category of capabilities :)

[knownasilya]

Using the terminology in strider-remote-worker the Queen would be the server with a documented API versions and the Drones or thin-runners can register to the server via predefined API-key, registering capabilities and then you can bind the drones to a specific project or via category of capabilities :)

Sounds like a nice idea, @exsilium! I would love something like this as well. Maybe @jaredly can give us some insight into what's possible with strider-remote-worker and strider-remote-drone.

[airtonix]

@niallo one of the most common requirements for remote runners is hetrogenous execution environments, or put in simple terms: my striderCI is in a docker container, my remote runners should be able to execute jobs on windows/mac/linux/solaris/etc.