Figure out exact requirements

[joelkuiper]

List of MUST features

• Features for patavi
  • Distributed design
  • [ ] Distributed workers
  • [ ] Distributed servers
  • [ ] < 10ms latency
  • [ ] Scalable to hundreds of concurrent users
  • [ ] Ability to chain different workers (define task topology)
  • [ ] No external brokers (e.g. Redis, RabbitMQ)
  • [ ] Persistent and failure tolerant (i.e. to disk)
  • Variable task duration:
  • [ ] < 500 ms
  • [ ] 500 ms - minutes
  • [ ] minutes - hour
  • [ ] hours - days
  • Features
  • [ ] Out of band updates
  • [ ] Ability to track state (i.e. accepted / rejected / processing /done / failed)
  • [ ] Multiple worker types (named services / channels) on single ip:port
  • [ ] Streaming updates (WebSockets, SSE, Comet, ...)
  • [ ] Ability to reconnect to update stream after session loss
  • [ ] REST API
  • [ ] Java/Clojure client
  • [ ] Javascript client
  • [ ] Preload R code & packages
  • [ ] Sandboxed R instances (forks / separate workspace / random uid+gid)
  • [ ] Binary results (images, code)
  • [ ] Multipart results (e.g. JSON/XML + images + archive of samples)
  • [ ] User authentication (via API keys or user/password)
  • [ ] Hot R code reload
  • [ ] Flexible back-end (not just R, but Python, shell, etc)
  • [ ] History of previous tasks

[gertvv]

This is the old list, but here you go:

• [x] Distributed workers
• [ ] Distributed server
• [x] < 10ms latency
• [x] Scalable to hundreds of concurrent users
• [x] Variable task duration
  • [x] < 500 ms
  • [x] 500 ms - minutes
  • [x] minutes - hour
  • [ ] hours - days
• [x] Out of band updates
• [x] Ability to track state (i.e. accepted / rejected / processing /done / failed)
• [ ] Multiple worker types (named services / channels) on single ip:port
• [ ] Streaming updates not sure how this differs from OOB
• [x] REST API
• [x] Sandboxed R instances (forks / separate workspace / random uid+gid)
• [x] Java/Clojure client
• [ ] Javascript client
• [ ] Binary results (images, code)
• [x] Multipart results (e.g. JSON/XML + images + archive of samples)
• [x] Preload R code & packages
• [x] User authentication (via API keys or user/password)
• [ ] Hot R code reload
• [ ] Persistent and failure tolerant (i.e. to disk)
• [ ] History of previous tasks
• [ ] No external brokers (i.e. Redis, RabbitMQ)

Notes

The R instances need some isolation, but probably not that much, as we control the code that goes on them.

The multi-part / binary data would be the easiest to compromise on. I do not have big problems with embedding such data with MIME encoding e.g.

The latency is not a very hard constraint, it could also be < 50ms (but for a no-op 10ms seems reasonable).

Multiple channes with a single broker is definitely a very nice to have.