search

JuliaCloud / JuliaBox

Juliabox continues to run, but this codebase is no longer current.
http://www.juliabox.org/
Other
185 stars 50 forks source link

Feature Request: Script Scheduling #227

Open quinnj opened 9 years ago

quinnj commented 9 years ago

I talked with Viral this morning about some stuff we're doing at Domo and integrating with Julia.

One feature we thought would be really useful would be to have some kind of scheduling functionality to run a script/notebook according to a daily/weekly schedule of sorts. An example use case for a company I'm working with here is:

-Every hour, they get a dump of raw, gzipped Google DFP data -Every hour, a Julia script could go fetch the latest file, validate the formatting, then push up into Domo

This saves the need for the company to install/manage Julia and helps get their data uploaded on a consistent basis.

Happy to help flesh the idea out some more. I wonder if, since people are logging in with Google, there'd be a way to utilize the Google Calendar API to handle the scheduling. Maybe more complicated than throwing something of our own together.

ViralBShah commented 9 years ago

I think the way to do this is as part of the API deployment that we were planning. The deployment model could be event driven, through API requests, or schedule driven.

jiahao commented 9 years ago

Sounds awesome!

quinnj commented 9 years ago

Actually, I was thinking about this some more the other day and I don't think we'd really need to be able to schedule scripts on the JuliaBox side if there was an API endpoint that would trigger the running of a script. That way, you could handle the scheduling where ever you wanted, cron, Domo, etc. and it would just involve hitting a https://juliabox.org/api/run/script1234. What's the status on a JuliaBox API?

ViralBShah commented 9 years ago

There is JuliaBox.jl, which will be renamed to something better, and I believe the rest of the stuff is largely ready, but needs a refactor to go into production.

@tanmaykm should really say more.

tanmaykm commented 9 years ago

@quinnj Yes, API can also be triggered externally by hitting the URL.

JuliaBox.jl provides the a REST API framework on the Julia side. It has a HTTPServer.jl based frontend, marshals requests over ZMQ to a separate Julia process that maps them on to Julia methods. Shall rename this to something appropriate, probably APIBox.jl suggested by @ViralBShah and @aviks.

JuliaBox APIs use JuliaBox.jl, but replaces the HTTP front with nginx+tornado. It also adds some queuing, and auto-scaling, and runs the Julia backend inside docker images. The API branch is at https://github.com/tanmaykm/JuliaBox/tree/api. It is largely working, but have been wanting to do a refactor of JuliaBox before merging this. I can probably setup a test server sometime next week.

For a local small volume setup, I think Julia + JuliaBox.jl in a docker container would work well. Here's one based on JuliaBox: https://github.com/tanmaykm/JuliaDockerImages/tree/master/pkgdists/juliabox. Do let me know if you would like to try this or the test server out.

quinnj commented 9 years ago

@tanmaykm excited to see the api service go live. I may have a go at playing with JuliaBox.jl, but I'd also be willing to stress-test the live version :)

tanmaykm commented 9 years ago

@quinnj It will be great to have you try this! Unfortunately I haven't had time to work on it yet, and I'll be travelling next week. I'll see if I can set it up over the weekend, otherwise I'll be probably be able to do this only after I'm back.

quinnj commented 9 years ago

I'd love to try it. Do you just want me to try JuliaBox.jl in general? Or is there a test server I can try on? julia.mit.edu?