search

qubole / rubix

Cache File System optimized for columnar formats and object stores
Apache License 2.0
183 stars 74 forks source link

In split computation, do no assume master is localhost #32

Open shubhamtagra opened 8 years ago

shubhamtagra commented 7 years ago

This is required if we want Rubix to run on a separate cluster than the execution clustre

vrajat commented 7 years ago

Same for Tez ? Split computation is in an AM on a slave ?

On Fri, Jun 30, 2017 at 10:49 AM Shubham Tagra notifications@github.com wrote:

This is required if we want Rubix to run on a separate cluster than the execution clustre

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/qubole/rubix/issues/32#issuecomment-312177463, or mute the thread https://github.com/notifications/unsubscribe-auth/ABj_muaT1dijRcwOoNw8oi-vGBEHGSUeks5sJIVPgaJpZM4Kh8Mz .

shubhamtagra commented 7 years ago

Didnt know about that. Yes, in that case we will have to do this.

vrajat commented 7 years ago

@abhishekdas99 This is critical for support with Tez

shubhamtagra commented 7 years ago

We might be safe, Hadoop2 ClusterManager class is using "yarn.resourcemanager.webapp.address" config to figure out master address. As long as that is populated correctly, things should work fine for Tez.

For Presto too, "master.hostname" is used to find master name. So what was expected out of this issue has made its way into code as part of other changes and this is not just a testing exercise that things work fine.