General feedback from an outsider

[eliast]

I'm giving a talk at AWS re:Invent on Singularity and have been working to setup a demo for the audience. The good news is that I'm almost there with a very compelling demo in my opinion but I did run into several issues while getting ready. I'd figured the feedback would be help you get the project to a better place for someone trying Singularity for the first time after a talk.

The goal was to run Singularity and launch two requests (docker containers): mysql and wordpress and show that it can survive a docker kill as well as showcasing an increase in the number of instances for the wordpress container. I did all of this using the Vagrant setup. Very nice @tpetr and thanks for all your help. I couldn't have done it without you.

But I ran into a few things along the way and I figured it'd be best for you guys to see even though I know you know most of them already.

• [ ] There's a test that times out during the vagrant up step. The hacky solution for now was to rm the test directory in SingularityService. I know you can run with -DskipTests but I really wasn't going to find out how to modify the vagrant provisioning step.

  ==> test: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project SingularityService: There was a timeout or other error in the fork -> [Help 1]

• [ ] I tried using the Docker container support and it's almost there. It doesn't support portMappings yet I believe. Therefore, I couldn't run mysql because I had no way of modifying the 3306 port since Singularity's mysql service runs on it. The hack was to modify Singularity to start on 3307.
• [ ] No Docker support on the UI yet.
• [ ] Since Docker in Singularity support doesn't support portMappings or linking I had to revert to running docker run commands directly via normal tasks.
• [ ] I used the UI to create a new request but had to learn the hard way that I couldn't use load balance support without Baragon setup.
• [ ] I tried submitting a task request but had to look at the source to figure out that the request had changed to contain a DeployInfo inside a DeployRequest. I'd recommend using something like Apiary to keep an up to date documentation of the API for those that can't read Java.
• [ ] As I moved away from Docker support, I had to figure out how to submit a normal task but failed to construct the proper JSON for it causing an invalid piece of data stored in your ZK. This caused an NPE that was forcing a System.exit. Singularity went in a loop and I couldn't figure out for a while why the UI was intermittently failing. That brings me to the fact that it's not obvious how to restart Singularity inside vagrant as well as finding the logs to realize there was an NPE. It won't be easy to a stranger on how to clear up the invalid task inside the ZK server.
• [ ] By default tasks are not running within a container that supports tailing or finding the output of a failed task. It was very hard to debug it. It took me a while to find that mesos-slave was logging everything to /tmp/... It'd be great to use whatever executor you have that support log access via the UI out of the box.
• [ ] Finally, I ran a container successfully, but then when I tried my second one, it failed. Not enough resources. I had to figure out how to add a resources file with cpus(*):8 so I can continue my experiment as well as how to restart mesos.
• [ ] Once I had figured out I could use $PORT0 to modify my docker run commands I was able to get everything working and I was blown away. I wanted more.
• [ ] I wanted to increase the number of wordpress container instances but it was not clear that you just post the same request to the same endpoint just change the instances. UI access would be cool or an endpoint that was PATCH|POST /api/requests/{id} instead of just POST /api/requests/ so I know I'm modifying an existing request not just creating a new one.

Now I just need to setup Baragon so I can load-balance between my two wordpress docker containers and everything will be awesome.

Overall, I'm blown away at what can be done without me having to modify a single line of code. Most of this it's simply out of the box configuration, but it will deter people from seeing how easy it's to use Singularity and Mesos to get an amazing PaaS for free! Keep up the good work guys.

/cc @wsorenson @tpetr @hgschmie @stevenschlansker

[gchomatas]

Thanks for the great and extensive feedback @eliast !

When I was preparing for MesosCon14 I run into similar problems when trying to use Singularity stand alone without the other HubSpot tools running on top. At that time I wrote the first version of the docs (needs a continuous curation :)) and changed vagrant to run both Singularity and mesos in the same machine. Tom has done many more enhancements to vagrant since then but it is true that we need to do more to make singularity dead easy to test/use for common cases.

Please feel free to update the docs with fixes, clarifications and since you are doing the exercise it would be fantastic if you find some time to write a how-to for other users that would like to run the scenario you are trying to demo. There is a section in docs named "Singularity_Deploy_Examples.md" that I didn't find the time to fill in . It is meant to include how-tos for running some common use cases. You could be the first to add a use case in there!

[stevenschlansker]

Yes thanks, this is great feedback. I'm going to split this out into separate issues and ask some followup questions.

[stevenschlansker]

@eliast

Finally, I ran a container successfully, but then when I tried my second one, it failed. Not enough resources. I had to figure out how to add a resources file with cpus(*):8 so I can continue my experiment as well as how to restart mesos.

I generally launch my tasks with e.g. cpus=0.1 since no task needs a dedicated CPU.

[stevenschlansker]

OK, I've filed issues for all except the above one which I believe can easily be fixed when you create your request. It's doubtful that we can fix all of these before your demo, but if there's a couple in particular you'd like (for example, I believe it would be relatively simple to get proper Docker integration working for you, and it's a lot more compelling with it working!) please mention it and hopefully we can find some time to get fixes in. I would close this mega-issue now but I can't :)

[eliast]

We have time for my demo! As long as we don't go backwards, I should be able to recreate what I have so far. I can see how portMappings is already supported in Mesos, but do you know how to add linking support? The basic test would be to get a mysql and 2 wordpress containers running. What do you say?

Thanks for all of the issue filing too!