search

nanobox-io / nanobox

The ideal platform for developers
https://nanobox.io
MIT License
1.6k stars 89 forks source link

Nanobox server not running on Shippable CI #427

Open mbrodala opened 7 years ago

mbrodala commented 7 years ago

Currently launching Nanobox in a CI environment (thus using the CI bootstrap script) fails:

Error   : nanobox-server service start was successful but the service is not running
Context :

Any idea how to get more output for this?

lyondhill commented 7 years ago

this happens when a ci does not start services the same way as a native linux machine would. shippable is a testable CI.

mbrodala commented 7 years ago

See this build log for details. And the repository source.

lyondhill commented 7 years ago

So the problem I have found so far is that on shippable instead of running inside a vm like other CI's on shippable runs inside a docker container. then in that container they clone the git repo inside /root/src/. This creates 2 problems: 1: nanobox is expecting a standard service manager (systemd, upstart, etc). But nanobox cannot start its server 2: Shippable grants access from inside the container to the docker sock file which is good. but nanobox trys to mount the code folder into the docker container ( -v :/app) the problem with this is that in shippable they just clone the nanobox code folder inside the container and it doesnt really exist on the host (technically it does exist but under the docker container id) so when the docker server on the host gets the request to mount /root/src/ that folder doesnt exist on the host.

mbrodala commented 7 years ago

I'm not that much into Docker yet but do you see a way to have Nanobox' containers directly access directories of Shippable's build container? Do you see any other way to solve this?

As for the startup issue I'm not sure if there is anything I can do. Can't the Nanobox server be started in background as regular binary?

mbrodala commented 7 years ago

@lyondhill I've created a Shippable support ticket, maybe you can watch this ticket and join the discussion once more detailled questions show up.

mbrodala commented 7 years ago

Do you have any other ideas how this could be solved given the options to customize the build?

mbrodala commented 7 years ago

@lyondhill Do you think issue 2 can be fixed as described here?

pre_ci_boot:
  options: "-v /app:/app"

This mounts the directory /app on the host to the /app directory on the container. By copying data/code to the /app directory in the container in a CI task, we make those files available in the /app directory on the host. The /app directory can be replaced by any other directory.

Then later cp -R . /app.

mbrodala commented 7 years ago

Is there any way to launch the Nanobox server without a service manager?

danhunsaker commented 7 years ago

Actually, yes, there is. As root: nanobox server

mbrodala commented 7 years ago

I just tried that and the process keeps hanging afterwards:

nanobox server --trace  

   0.00/10.72MB [                                          0.00%]

   ...

   10.72/10.72MB [****************************************  100.00%]
2017/06/13 07:12:52 <nil>

I guess there is no forking into background if the server is started this way? But I also cannot simply do a nanobox server & to start it in the background since then the following Nanobox commands wouldn't work as long as the server is not yet started (which takes a considerable amount of time). And waiting for the background process also wouldn't help since it never finishes (or shouldn't) unless it's an error what I'm seeing here.

Any idea how to get this working?

danhunsaker commented 7 years ago

Looks like it's actually trying to run nanobox-update first thing. Maybe run that manually ahead of time? Once that closes, you should be able to start the server in the background just fine, I believe.

mbrodala commented 7 years ago

That didn't work out well and yes, the issue is obvious:

nanobox-update --trace  
/root/a52197e6-7908-4ce6-a7ef-3f5c1ee0273e.sh: line 73: nanobox-update: command not found

I'll try to manually download this binary.

mbrodala commented 7 years ago

After loading the nanobox-update binary and making it executable the behavior is unchanged except that apparently the update is now performed twice. Again, nanobox server hangs at the end:

nanobox-update --trace 

   0.00/15.10MB [                                          0.00%]

   ...

   15.10/15.10MB [****************************************  100.00%]

nanobox server --trace

   0.00/10.72MB [                                          0.00%]

   ...

   10.72/10.72MB [****************************************  100.00%]
2017/06/13 10:36:15 <nil>
danhunsaker commented 7 years ago

Interesting size difference, there, too...

As to hanging at the end, that would be the server actually running. I'm not sure what it's actually downloading, at this point, though, or why. One of the staff members will have to weigh in to get much further.

mbrodala commented 7 years ago

Alright, I hope a staff member chimes in then. Thanks so far.

mbrodala commented 7 years ago

As mentioned here it should be possible to basically add another volume mount to the Shippable build container, copy everything into that directory and thus make it accessible on the host. If another container then volume mounts that directory, it should be able to access the code.

mbrodala commented 7 years ago

I've now asked Shippable if they could implement a feature to completely override the build container command (currently docker run). This way I should be able to use Nanobox to run the build itself.

mbrodala commented 7 years ago

Currently I'm trying to install Nanobox on the Shippable build host instead and mount it into the build container. However currently this fails:

# bash -c "$(curl -fsSL https://s3.amazonaws.com/tools.nanobox.io/bootstrap/ci.sh)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

            Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 15.0M  100 15.0M    0     0  60.7M      0 --:--:-- --:--:-- --:--:-- 60.8M
--------------------------------------------------------------------------------
+ ERROR:
+ Nanobox is designed to run as a standard user (non root)
+ Please run all nanobox commands as a non privilage user
--------------------------------------------------------------------------------
Nanobox is ready to go!

# nanobox version
Using nanobox with docker-machine requires tools that appear to not be available on your system.
vboxmanage
View these requirements at docs.nanobox.io/install/requirements/docker-machine/

There are various issues:

  1. Running Nanobox as root should be allowed on CI systems.
  2. The bootstrap script happily continues even if an error occurs in the middle, it should stop and fail hard instead.
danhunsaker commented 7 years ago

It should be working as root in CI mode. Wonder if the CI bootstrap sets CI mode first or not. It'll have to do that before anything else.

I think that's part of why it's trying to use VirtualBox, too.

mbrodala commented 7 years ago

Small update: the error is caused by the CI bootstrap script which runs nanobox config set provider native before nanobox config set ci-mode true. If the latter is the first invocation of Nanobox, that part works just fine.

danhunsaker commented 7 years ago

Indeed. This confirms my suspicion, above.

PR submitted to fix this in nanobox-io/bootstrap#27...

mbrodala commented 7 years ago

Thanks to that fix I was able to successfully install Nanobox on the build host machine:

build_pre_ci
  bash -c "$(curl -fsSL https://s3.amazonaws.com/tools.nanobox.io/bootstrap/ci.sh)"
  nanobox version
  Nanobox version 2.1.2 (eea68d66ac4e86987e8a15215bd2a06a)

However, currently I am unable to properly mount the Nanobox binary into the build container. Neither mounting it as single file (using -v ~/../usr/local/bin/nanobox:/usr/local/bin/nanobox) nor mounting the whole parent folder (using -v /usr/local/bin:/usr/local/bin/host) works. The Nanobox binary always appears as directory in the build container ...

mbrodala commented 7 years ago

Yesterday I tried the same though a simple test and it worked just fine:

$ docker run -it --rm -v $(which nanobox):/usr/local/bin/nanobox php bash
# ls -l /usr/local/bin/
total 38764
-rwxrwxr-x 1 root root       118 Jul  3 21:22 docker-php-entrypoint
-rwxrwxr-x 1 root root      1276 Jul  3 21:22 docker-php-ext-configure
-rwxrwxr-x 1 root root      2288 Jul  3 21:22 docker-php-ext-enable
-rwxrwxr-x 1 root root      2223 Jul  3 21:22 docker-php-ext-install
-rwxrwxr-x 3 root root       587 Jul  3 21:22 docker-php-source
-rwxr-xr-x 1 root staff 15841242 Jun 15 21:34 nanobox
-rwxr-xr-x 2 root staff      817 Jul  3 21:27 pear
-rwxr-xr-x 2 root staff      838 Jul  3 21:27 peardev
-rwxr-xr-x 2 root staff      754 Jul  3 21:27 pecl
lrwxrwxrwx 1 root staff        9 Jul  3 21:27 phar -> phar.phar
-rwxr-xr-x 2 root staff    14829 Jul  3 21:27 phar.phar
-rwxr-xr-x 2 root staff 11828056 Jul  3 21:27 php
-rwxr-xr-x 2 root staff     2620 Jul  3 21:27 php-config
-rwxr-xr-x 2 root staff 11959520 Jul  3 21:27 phpdbg
-rwxr-xr-x 2 root staff     4547 Jul  3 21:27 phpize
# nanobox

CONFIGURE NANOBOX
---------------------------------------------------------------
Please answer the following questions so we can customize your
nanobox configuration. Feel free to update your config at any
time by running: 'nanobox configure'

(Learn more at : https://docs.nanobox.io/local-config/configure-nanobox/)

How would you like to run nanobox?
  a) Inside a lightweight VM
  b) Via Docker Native

  Note : Mac users, we strongly recommend choosing (a) until Docker Native
         resolves an issue causing slow speeds : http://bit.ly/2jYFfWQ

Answer: ^C

Since this still does not work on Shippable, I'll ask over there.

mbrodala commented 7 years ago

See https://github.com/Shippable/support/issues/3707

mbrodala commented 7 years ago

That didn't work out. In the end I managed to get the Nanobox CLI to exist on the build host but that didn't make a difference.

mbrodala commented 7 years ago

BTW I also just tried CircleCI but again, server startup fails:

sudo nanobox server

   0.48/10.73MB [*                                         4.48%]   1.46/10.73MB [*****                               2.41/10.73MB [********        2.89/10.73MB [**********                                 3.37/10.73   3.85/10.73MB [**************            4.33/10.73MB [****************                          40.36%]
   4.81/10.73MB [******   5.28/10.73MB [*******************                 5.7   6.24/10.73MB [****************   6.72/10.73MB [*************************                 6   7.20/10.73MB    7.69/10.73MB [**************************   8.63/10.73MB [*********   9.11/10.73MB [*********************************      9.59/1   10.07/10.73MB [*      10.73/10.73MB [****************************************  100.00%]
2017/07/11 07:56:57 <nil>
pablorsk commented 6 years ago

I have the same problem, on Bitbucket Pipelines:

+ bash -c "$(curl -fsSL https://s3.amazonaws.com/tools.nanobox.io/bootstrap/ci.sh)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  9.7M  100  9.7M    0     0  53.3M      0 --:--:-- --:--:-- --:--:-- 53.6M
Successfully set 'ci-mode'
Successfully set 'provider'
Nanobox is ready to go!

+ nanobox -v
Nanobox Version 2.3.0-180117T2039 (4b3d14f)

+ nanobox config set ci-mode true
Successfully set 'ci-mode'

+ nanobox remote add apicultor

✓ You've successfully logged in

✓ Codebase linked to apicultor

+ nanobox deploy

Error   : nanobox-server service start was successful but the service is not running
Context : 

Skipping cache upload for failed step
Searching for test report files in directories named [test-results, failsafe-reports, test-reports, surefire-reports] down to a depth of 4

Also I try with 

nanobox server --verbose &
sleep 30
nanobox deploy
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Error   : nanobox-server service start was successful but the service is not running
Context :

:cry:

mbrodala commented 6 years ago

@pablorsk Thanks for checking, that was on my TODO list but now I am sure that it won't ever work either.

seanculver commented 6 years ago

@mbrodala I've successfully ran CI with circleci using my nanobox thanks to using this Gist from @brandoncc - https://gist.github.com/brandoncc/e680d39490c434b059486fa08fc089c6

Here is my elixir version of the same thing and even has an auto deploy workflow built in. https://gist.github.com/seanculver/91bd62f31e2feec589e0f91644caf0df

I haven't figured out caching of the box yet but I'll update my gist with the caching stuff, it added a few minutes to my build but I'll see if I can't get it back to where it was.

Hope this helps!

What Brandon said was it might be the machine: true flag that kicks it out of a docker container and allows us to run our own docker container via nanobox, this is just a hunch though.

mbrodala commented 6 years ago

@seanculver Thanks a lot for the hints. I can confirm that this setup is working fine. I'll look forward to even faster builds once caching is working. :-)

brandoncc commented 6 years ago

@seanculver @mbrodala I'm glad my config worked for you guys. Just an FYI, there may be an additional charge for using machine: true in the future, since it gives you direct OS access on a powerful VM. You can see more here. At the bottom of that section you will notice that there is also a docker_layer_caching config option, which may handle the caching properly (it may not cache nanobox properly, I don't know for sure). Unfortunately, it is a paid addon.

pablorsk commented 6 years ago

On Bitbucket Pipelines receive new errors :cry:

+ nanobox deploy apicultor-20
Failed to get D-Bus connection: Unknown error -1
Failed to get D-Bus connection: Unknown error -1
Failed to get D-Bus connection: Unknown error -1
Error   : nanobox-server service start was successful but the service is not running
Context :

And this is my bitbucket-pipelines.yml

arnaud-zg commented 5 years ago

Any news on this problem? I have the same issue on Gitlab Pipelines.

danhunsaker commented 5 years ago

Nanobox will never run on any host which doesn't provide access to Docker. It can't be Docker Remote, either, because of the need to mount things into the container. So you won't be able to run it on a lot of the CI services out there, and it's not a thing that can be fixed on Nanobox's end. It's a fundamental limitation of the technology.

brandoncc commented 5 years ago

If it helps anyone, I use this for circleci: https://gist.github.com/brandoncc/e680d39490c434b059486fa08fc089c6

The downside is it can't cache so it takes while to setup the docker image. A full test run takes about 15 minutes.