CDH 5.3.1 - No container-executor

[kitplummer]

I'm not experienced enough with CDH/YARN to know what's really supposed to happen from the distro - but the 'container-executor' binary isn't there in the ${hadoop}/bin directory - or anywhere else. This causes the state to fail. Anyone else see this?

          ID: fix-executor-group
    Function: cmd.run
        Name: chown root /usr/lib/hadoop/bin/container-executor
      Result: False
     Comment: Command "chown root /usr/lib/hadoop/bin/container-executor" run
     Started: 14:33:50.911201
    Duration: 46.19 ms
     Changes:
              ----------
              pid:
                  10772
              retcode:
                  1
              stderr:
                  chown: cannot access `/usr/lib/hadoop/bin/container-executor': No such file or directory
              stdout:

----------
          ID: fix-executor-group
    Function: cmd.run
        Name: chgrp yarn /usr/lib/hadoop/bin/container-executor
      Result: False
     Comment: Command "chgrp yarn /usr/lib/hadoop/bin/container-executor" run
     Started: 14:33:50.957615
    Duration: 48.23 ms
     Changes:
              ----------
              pid:
                  10787
              retcode:
                  1
              stderr:
                  chgrp: cannot access `/usr/lib/hadoop/bin/container-executor': No such file or directory
              stdout:

----------
          ID: fix-executor-permissions
    Function: cmd.run
        Name: chmod 06050 /usr/lib/hadoop/bin/container-executor
      Result: False
     Comment: One or more requisite failed: {'hadoop.yarn.fix-executor-group': 'Command "chgrp yarn /usr/lib/hadoop/bin/container-executor" run'}
     Started:
    Duration:
     Changes:```

[kitplummer]

Same issue with CDH-5.0.0.

[sroegner]

Looks to me like Cloudera doesn't put the container-executor into their tarballs - it is in that path in all the HDP tarballs i checked. Not sure what can or should be done about this - is moving to one of the HDP versions an option?

[kitplummer]

Unfortunately it isn't an option.

I think the only thing would be to install Maven and build it. Yuck.

On Wed, Apr 15, 2015 at 2:07 PM, Steffen Roegner notifications@github.com wrote:

Looks to me like Cloudera doesn't put the container-executor into their tarballs - it is in that path in all the HDP tarballs i checked. Not sure what can or should be done about this - is moving to one of the HDP versions an option?

— Reply to this email directly or view it on GitHub https://github.com/saltstack-formulas/hadoop-formula/issues/21#issuecomment-93569606 .

[gravyboat]

@sroegner @kitplummer Is there anything unique to the Cloudera install we could check against? As much as I dislike it, we could add a pillar value to flag these as to whether we should run them or not based on that pillar value.

[kitplummer]

The existence of the binary I suppose. I'm not sure how necessary it is to YARN - especially considering it isn't part of the CDH distro. Maybe just flag to ignore if CDH-*?

On Wed, Apr 15, 2015 at 2:17 PM, Forrest notifications@github.com wrote:

@sroegner https://github.com/sroegner @kitplummer https://github.com/kitplummer Is there anything unique to the Cloudera install we could check against? As much as I dislike it, we could add a pillar value to flag these as to whether we should run them or not based on that pillar value.

— Reply to this email directly or view it on GitHub https://github.com/saltstack-formulas/hadoop-formula/issues/21#issuecomment-93571992 .

[gravyboat]

Hmm, I mean we could just add an unless that looks like this:

- unless: test ! -f /usr/lib/hadoop/bin/container-executor

That would return true if the file doesn't exist.

[kitplummer]

That'd work. :)

I can send a PR if we're OK with that. On Wed, Apr 15, 2015 at 2:34 PM, Forrest notifications@github.com wrote:

Hmm, I mean we could just add an unless that looks like this:

• unless: test ! -f /usr/lib/hadoop/bin/container-executor

That would return true if the file doesn't exist.

— Reply to this email directly or view it on GitHub https://github.com/saltstack-formulas/hadoop-formula/issues/21#issuecomment-93575777 .

[gravyboat]

@kitplummer Works for me!

[sroegner]

Good idea. The binary is only needed if yarn is (explicitly) configured to use it.

[gravyboat]

@sroegner Were you referring to the unless, or the binary check?

[sroegner]

@gravyboat I meant the unless condition

[gravyboat]

@sroegner Okay cool!

[gravyboat]

I'm closing this, it's resolved in #22. Thanks for the input!