Java API for Onyx - Githubissues

onyx-platform / onyx

Distributed, masterless, high performance, fault tolerant data processing

http://www.onyxplatform.org

Eclipse Public License 1.0

2.05k stars 204 forks source link

Java API for Onyx #1

Closed MichaelDrogalis closed 7 years ago

MichaelDrogalis commented 10 years ago

As of release 0.3.0, Clojure is the only supported language for Onyx. Java users can use the APIs that Clojure offers to tap some of the Onyx functionality, but this becomes problematic for areas such as lifecycle extensions that rely on implementations of multimethods.

Furthermore, EDN isn't the friendliest cross-language data format to send catalogs and workflows through. Part of this issue should explore options that Java users have on this front.

MichaelDrogalis commented 10 years ago

This one seems to be a blocker for a few folks - going into development for 0.4.0.

MichaelDrogalis commented 9 years ago

Here's a good first set of API functions that could be shim'ed for Java and merged back to master. There are more, but this is a start:

onyx.api functions:

connect
await-job-completion
shutdown
submit-job
start-distributed-coordinator
stop-distributed-coordinator
start-peers

Notes:

Many of these need to be polymorphic on the Coordinator - be it an in-memory or distributed Coordinator.
In places where we need to deal with exposing Clojure's data literals, we'll use java.util.Map and java.util.List. That means for submit-job, the function will received a List as the catalog, and Map as the workflow. The Java shim should send these objects through as-is to Clojure. We can disentangle them back to EDN data within Clojure itself.

prasincs commented 9 years ago

naming is tricky.. are you expecting some conversion from kebab case to camelcase too? Also, don't protocols already have corresponding Java interfaces?

MichaelDrogalis commented 9 years ago

We're shooting for something that will feel totally natural to a Java developer. Only a subset of those functions use protocols, and even then, they're sometimes parameterized with keywords. The more we can shield the average Java developer from Clojure while still exposing the full power of the API, the better shape we're in.

I don't have anything in particular set in mind, but that's the general idea.

aaelony commented 7 years ago

We have few Clojure people, but many Java devs. Some on our team have raised this issue as well, and this may become an issue.

MichaelDrogalis commented 7 years ago

A Java API is low on our priority list. Two viable paths to its completion are someone stepping up to work on this one with some guidance, or fund it's development.

MichaelDrogalis commented 7 years ago

Finally got this one done. Thanks community! https://github.com/onyx-platform/onyx-java