Docker-ising the build process

rwb27 commented 7 years ago

It would be much better if the built STL files were automatically generated on each commit (or nightly, or some other schedule). I have a couple of batch files that do this presently, but it would be nice to go for something a bit more sophisticated (and cross-platform). If someone reading this knows about how to nicely automate builds with OpenSCAD, or how to integrate building with the GitHub API, I'd love to hear from you!

dborgesr commented 7 years ago

So i've been learning Docker recently and it might be possible to take advantage of this to automate this entire process. Are all your scripts up on github? If so when i have some time ill get started on making a Docker image to add on, this should allow you to pull the image and build the STLs trivially, likely can hook it up to Docker-hub to have automatic builds as well.

rwb27 commented 7 years ago

Thanks! That sounds like exactly what I want to do... My "scripts" are up, such as they are. They need a lot of improvement, but for the moment I have a batch file that runs on my Windows machine and compiles the larger STL files (the link below has the main body commented out, which should probably be undone, but you'll get the idea):

https://github.com/rwb27/openflexure_microscope/blob/master/rebuild_illumination.bat

I originally tried with Python, but couldn't get it to work, OpenSCAD just wouldn't function correctly in a subprocess.

If you can run OpenSCAD in your docker image from a bash script or something, that would be an enormous help - just pasting in the command line options from the script above would be a perfectly good test case.

rwb27 commented 7 years ago

In f805e5ed81615726e89c3060b3695e155c0c313d I've now got batch files for everything bar the illumination and lens insertion tools. This is progress! It would be nice to use something like make rather than just a bunch of batch files, and I'm starting to understand how to do that (I've not used make before for this). It should still be windows-compatible, because you can use make in MSYS, which comes bundled with Git for Windows. Other options do exist as well...

dborgesr commented 7 years ago

Hey, Could you give me a quick rundown of how to go from nothing to STL files? I'm looking to port this over to linux (and therefore docker) in the least painful process. What and how do the scripts work?

rwb27 commented 7 years ago

Sure. The scripts are very (painfully) simple. Essentially, you can build each STL file by compiling one OpenSCAD file, either by opening it in OpenSCAD and hitting "compile" or by running the OpenSCAD binary in command-line mode. The commands to compile all the relevant bits are listed in the batch files: they are in the root directory of the repository. The same commands should work as-is on Linux, provided you change the (currently hard-coded, sorry) OpenSCAD binary path to point it to the right place. There's some documentation on (using OpenSCAD from the command line)[https://en.wikibooks.org/wiki/OpenSCAD_User_Manual/Using_OpenSCAD_in_a_command_line_environment] though to be honest I'm not sure you even really need to read it in order to build things. It does however mention some examples of invoking it from make so it might be worth a skim.

If the only options you give to openscad are an output file (-o thing.stl) and an input file (last argument, thing.scad) then it will compile using the parameters in microscope_parameters.scad. I use command line arguments -D parameter=value to override some of these, and produce the different versions of the microscope (different stage heights, motor lugs, etc.). The only place where this is not sufficient is the optics module; currently, you pick the camera to use by including one of four different camera files, and OpenSCAD has no way to switch between them using command line arguments. I will need to pre-process the source to do this I think, which shouldn't be hard but I've put it off for another day. If I do that, I suspect the easiest way to do it is with sed and friends, I am relaxed about introducing that dependency!

NB compiling main_body.scad will take several minutes even on a relatively meaty machine.

I believe the only dependency it has is on OpenSCAD, which is available on Linux (though I'm ashamed to say I've yet to use it on that platform).

I hope that works - do please give me a shout if any of those instructions are lacking!

dborgesr commented 7 years ago

Sweet! ill start playing around w/ first dockarizing and running OpenSCAD, i'll likely expose a jupyter notebook from within the docker, so you can go to a browser window and compile everything from there (including selecting version etc).

rwb27 commented 7 years ago

cool :) does that mean you'd anticipate building stuff from a Python script? I played around with that briefly (there's some py files in the repository) but never got it to work, that may be a Windows thing though, I think it just didn't like being run in a subprocess. Picking versions/options in a browser would be really nice.

I have been thinking a bit more about make, and it quite probably is the right way to do things - I guess whatever interface does the building with OpenSCAD could just as easily run make. My thought was that each version of the microscope could be a target (versions just being particular combinations of variables). That target would then simply re-run make with said variables defined, which would then compile whatever's needed and copy it into the destination folder. I think it would even be possible to be cunning, and get make to cache the files that don't depend on particular variables (e.g. the gears are the same for all the versions). However, if you can get OpenSCAD working in docker, I can sort out the makefile!

rwb27 commented 7 years ago

OK, so I've had a stab at a makefile in 9e443826d8cbf142cdb23d9b320fa7524b7259ee and it now works, at least as well as the old batch files did (in that it can now rebuild everything, with the exception of the optics modules). It's not beautiful, but you can now just "make all" and, provided openscad is in your path, it should work.

Next step is probably to use variables in the makefile and/or create individual targets for the different microscope versions.

dborgesr commented 7 years ago

Hey, Just wanted to say i've been working to dockerize this whole thing and i stumbled on: https://github.com/SolidCode/SolidPython If you're able to move the Openscad code to that, its absolutely trivial to dockerize and edit and rebuild quickly using Jupyter. So far this definitely looks pretty doable, just taking some time.

rwb27 commented 7 years ago

Thanks - I have been considering some more Python-based solutions, my problem so far is that OpenSCAD already puts a lot of people off; adding Python into the mix will likely make that issue worse... Also, my quick skim of the docs for SolidPython suggests that it's a Python wrapper for OpenSCAD - and importantly its output is OpenSCAD code, not STL files. So you still need a working OpenSCAD installation, and you'd just be calling it from Python. So I think SolidPython might offer a solution for e.g. the Optics module, where it's necessary to tweak the source code to compile different versions, but I don't think it simplifies compilation, as you still need to have the OpenSCAD binaries available in order to render STL files.

Is there a reason that rebuilding through Jupyter is particularly advantageous vs just running make? I guess it gets a web interface that makes it easy to tweak parameters...

dborgesr commented 7 years ago

So, the reason to rebuild through jupyter is that its trivial to start a Docker container that has Openscad installed in it (https://hub.docker.com/r/wtnb75/openscad/), the issue is interacting with it in a meaningful way. So I'm almost done writing, essentially a python wrapper that is run when the docker container starts, so you can rebuild by say, running "docker run -it open_flexure_container rebuild_microscope.py -v output_data:output_data --foot_height=30 ..." adding the other handful of parameters. This works and essentially is a one liner that spits out STL files in a specific folder "-v output_data:output_data"

You can generate STL files quickly, but is more of a one-off and doesn't allow for editing.

Instead, if the whole thing is inside a Jupyter notebook (running SolidPython code, essentially a translation of the current OpenScad code), it can be exposed through a server that the container is running, so you can edit and rebuild, spitting out the STL in the same folder "-v output_data:output_data". The perk of this is that it becomes insanely easy to change the entire microscope quickly, simply change code and hit shift+enter to run and spit out STL file.

The reason why this actually increases the exposure is because once the docker container is set up, its a one line docker command to pull and run the image, followed by you simply going to say, http://localhost:8890 where you can edit and rebuild the microscrope trivially (into STL files which are output into that folder). https://hub.docker.com/r/jupyter/minimal-notebook/ i think is a decent example.

Once I get the files building in the docker container i'll put it up on Dockerhub. The only issue with makefile is that its largely a static deployment strategy, which will take lots of upkeep and its editability (special knowledge required) will be low, although it can be run pretty much anywhere.

rwb27 commented 7 years ago

OK, that makes a lot of sense. Currently all the makefile does is run a bunch of command-line OpenSCAD builds anyway, so that should be fine to do from a python script. The reason I've not done it from python so far is that it seems to crash on Windows, and I never figured out why - I guess Docker is designed to solve precisely this problem, making sure everyone has a consistent build environment.

I don't currently see an automated way to get the OpenSCAD files translated into SolidPython, but I think it's easy enough to find a simpler way to do the necessary pre-processing on the source files using Python, and possibly switch to SolidPython in the future if that looks promising. If you can get iPython to build anything using OpenSCAD in a docker container, that's the hard part done as far as I'm concerned; I can knock up the scripts to wrangle the source files fairly easily I think.

One last question, exposing my ignorance of Docker; how do you get files in and out of the container - will it be possible to automatically pull the source code in from the git repository, and spit out the STLs into a convenient build directory? I'll set it up and have a play, hopefully that will answer some of my questions!

dborgesr commented 7 years ago

Hey, So yeah it'll be super trivial, literally open to a web address, hit shift enter, and it will output the STL files to the directory you specify when running the container w/ the -v flag, I'll explain more when i get it all working and post it up. Question, you run openscad with multiple scad files (some as dependencies?) what's the command which is actually run? I'm having trouble running command line with multiple scads

rwb27 commented 7 years ago

Great :) Each time OpenSCAD is run, it's only processing one file - the dependencies are there because the file uses "include" or "use" statements, which mean it depends on the files I've listed. My hope is that make is smart enough to notice files have changed and recompile things accordingly, so making the dependencies explicit should save some time if e.g. I only change one file. For simplicity though, I've copied the actual command lines generated by make all below. For testing purposes, I would probably omit the main_body lines, as they can take some time (5-10 minutes). Building the lens tools is a nice simple start, followed perhaps by the gears or the feet (as those include more dependencies).

openscad -o builds/condenser_lens_tool.stl openscad/condenser_lens_tool.scad
openscad -o builds/tube_lens_tool.stl openscad/tube_lens_tool.scad
openscad -o builds/main_body_SS40.stl -D sample_z=40 -D big_stage=false -D motor                                                                                                                              _lugs=false openscad/main_body.scad
openscad -o builds/main_body_SS40-M.stl -D sample_z=40 -D big_stage=false -D mot                                                                                                                              or_lugs=true openscad/main_body.scad
openscad -o builds/main_body_LS65.stl -D sample_z=65 -D big_stage=true -D motor_lugs=false openscad/main_body.scad
openscad -o builds/main_body_LS65-M.stl -D sample_z=65 -D big_stage=true -D motor_lugs=true openscad/main_body.scad
openscad -o builds/main_body_LS75.stl -D sample_z=75 -D big_stage=true -D motor_lugs=false openscad/main_body.scad
openscad -o builds/main_body_LS75-M.stl -D sample_z=75 -D big_stage=true -D motor_lugs=true openscad/main_body.scad
openscad -o builds/illumination_and_rear_foot_SS40.stl -D sample_z=40 -D big_stage=false -D condenser=false openscad/illumination_and_rear_foot.scad
openscad -o builds/illumination_and_rear_foot_LS65.stl -D sample_z=65 -D big_stage=true -D condenser=false openscad/illumination_and_rear_foot.scad
openscad -o builds/illumination_and_rear_foot_LS65_condenser.stl -D sample_z=65 -D big_stage=true -D condenser=true openscad/illumination_and_rear_foot.scad
openscad -o builds/illumination_and_rear_foot_LS75_condenser.stl -D sample_z=75 -D big_stage=true -D condenser=true openscad/illumination_and_rear_foot.scad
openscad -o builds/illumination_and_rear_foot_LS75_condenser_tall.stl -D sample_z=75 -D big_stage=true -D condenser=true -D foot_height=26 openscad/illumination_and_rear_foot.scad

You can probably guess that the -D lines are what set the OpenSCAD variables. This works nicely for everything except the optics modules (for which I'll need to preprocess the source file to some degree).

dborgesr commented 7 years ago

Hey, So progress, I got the lens and tools compiling but i'm running into an issue w/ the main body and illumination stuff.

Parser error in line 142: syntax error Can't parse file 'openscad/main_body.scad'!

Parser error in line 150: syntax error Can't parse file 'openscad/illumination_and_rear_foot.scad'!

I think the problem might be the version of Openscad i'm using, because of the linux distro i'm using only has Openscad 2014, are you using the later version? I'm actually running this all on the raspberry pi itself... I thought it would be a good idea, and kinda cool, if it can actually build itself type of deal.

rwb27 commented 7 years ago

cool! That is indeed progress - but yes, I am using the latest version. I forget precisely what the changes were (around variables and things mostly) but I don't think it is trivial to make the files backwards-compatible...

I now have Docker running though :)

dborgesr commented 7 years ago

Hey, so attached is a Dockerfile i made which can be built and run for example like:

docker build -t test_scad . docker run --name open_flexure -tid -v "/Users/dborgesr/Documents/test_openscad:/input_data" test_scad

Also I included a python script which can replace the jupyter command when you're ready to make the container completely headless (instead of interacting w/ it through http://localhost i set the password to "openflexure" as you can see in the dockerfile)

openflexure.zip

that should help alot i think, i'll keep working on it soon

dborgesr commented 7 years ago

Hey quick note, in the above zip file ignore the python script... i uploaded the wrong one, the dockerfile is definitely useful, but i'll upload a new python script soon

rwb27 commented 7 years ago

Cool :) I've built it successfully so that all seems good - Unfortunately I'm going to have to wait for permission to run it, as I need a firewall rule to allow Docker to mount volumes - the joys of not being your own admin... Hopefully that will be sorted soon and I'll let you know how I get on :)

rwb27 commented 7 years ago

Hello, just thought I'd say I (finally) have the port open to allow file sharing with the VM. I had to tweak the Dockerfile (changing apt-get dist-upgrade to add the -yq flags) as it was failing to build otherwise, but it's now running :) I will need to figure out how to interact with it thought - "docker attach" doesn't seem to do anything...

dborgesr commented 7 years ago

ah, quick trick: docker exec -ti running_container_name bash

will run an interactable (-i) bash prompt within the image which will disconnect when you do.

Docker exec allows you to execute a command within a given running container, super useful

dborgesr commented 7 years ago

Oh! don't forget, its actually quite easy to install Docker on a Mac or Windows machine too, in case you dont want to deal w/ firewalls and such and just want to run everything local

rwb27 commented 7 years ago

Sure - I should just put it on my laptop; the problem is that my desktop (Windows machine) is so tightly managed, I can't connect to docker even when it's running on the same machine without special permission from IT which I've now obtained for some, but not all, ports. The command above does work though, so that is definitely progress!

rwb27 commented 7 years ago

Success :) After much faffing about, I've succeeded in connecting to my Docker install. Turns out the missing link was forwarding a port from my host machine to the container (-p 127.0.0.1:8090:80) as Docker for Windows doesn't do this by default. Perhaps for you on Linux it's different... Anyway, I guess now the ball's in my court to knock up a python script to do the building :)

rwb27 commented 6 years ago

Progress sincle I last updated here:

I can now build all the STL files with make. They come out, correctly named, in builds/. For now, I am still storing them in the git repo.
I have a build server, kindly provided by my university! It's not doing anything yet, but it exists :) This might mean I can just do nightly builds - but the idea of being able to do the build in Docker is still appealing for helping others get involved with development (I'm unlikely to be able to share SSH access to the build server outside of the University).

rwb27 commented 6 years ago

For my reference: while this repo is currently quite full of images, etc. it might be worth using a sparse checkout to download the SCAD files to the container - this will take an order of magnitude less time.

dborgesr commented 6 years ago

I should mention that the way I manage mine is by using different branches and only pulling the one i need, its served me well so far but keep in mind i'm actually working on deploying all this in a Resin.io image (which this project definitely will be incorporated)

rwb27 commented 5 years ago

I thought I'd add some comments in case they are helpful:

I've now got a halfway sensible build script that will retrieve the OpenSCAD files with a sparse checkout, and store builds in SHA1-named folders. That's progress...

https://gist.github.com/rwb27/5ab632a2d5ddf0949b052d2fd8754690

rwb27 / openflexure_microscope

Docker-ising the build process #22