mesos / kafka

Apache Kafka on Apache Mesos
Apache License 2.0
414 stars 140 forks source link

Why MESOS_NATIVE_JAVA_LIBRARY? #126

Open vlombardi opened 9 years ago

vlombardi commented 9 years ago

Hi, I'm new to Mesos and Kafka and I have some newby question...

I can't understand why do I have to set MESOS_NATIVE_JAVA_LIBRARY environment variable to run Kafka on Mesos. Strictly speaking, I could run Kafka on one host and Mesos masters and slaves on different hosts (Zookeeper would do the coordinator to ask to about what is the current Master and where a broker task should be executed). So, I don't have to run Kafka on the same host as Mesos, so, why do Kafka on Meso need libmesos.so?

I've seen that other github "Kafka on Mesos" projects seem not to request to specify MESOS_NATIVE_JAVA_LIBRARY on the Kafka host, so I assume it could be not necessary, but your work is brilliant (simple, clean, elegant and well documented) and I'd prefer to use your one :)

So, why does mesos/kafka need MESOS_NATIVE_JAVA_LIBRARY to be set? Is there a way to run mesos/kafka on a host without mesos installed?

Thank you for your great work and support!

joestein commented 9 years ago

you need libmesos so that the scheduler can talk (The Driver) to the master and master to scheduler, there are java bindings that you can build to https://github.com/groupon/jesos if you want to remove the libmesos dependency

I don't see why you would ever run the scheduler on a machine not on mesos, if you are running in production you need the scheduler to be HA so popping it on something like marathon does that for you. Everything on Mesos!

Eventually we would move to http://mesos.apache.org/documentation/latest/scheduler-http-api/ as that stables and some binds for it are integrated and tested, etc