Figure out why Hadoop is issuing "error importing function definition for `module`"

[GoogleCodeExporter]

Hadoop streaming issues 

2014-07-14 13:06.59 17504 [science.py]:272 /bin/sh: module: line 1: syntax 
error: unexpected end of file
/bin/sh: error importing function definition for `module'

when running streaming jobs where scripts depend on other scripts.  This does 
not stop code from executing but does result in warnings.

Find and fix.

Original issue reported on code.google.com by davehil...@gmail.com on 14 Jul 2014 at 6:18

[GoogleCodeExporter]

I was able to figure out(well it seems like a very good match) why we
are getting these messages in hadoop (when you look at the task logs)

stderr logs

sh: module: line 1: syntax error: unexpected end of file
sh: error importing function definition for `module'
sh: module: line 1: syntax error: unexpected end of file
sh: error importing function definition for `module'

It is because we have a multi-line environment variable(function) in the
environment.

See
http://stackoverflow.com/questions/19731808/error-exporting-shell-functions-from
-within-a-java-process

"
Hadoop Streaming, however, has a badly written Environment class that
tries to reimplement System.getenv() by naively parsing the output of env.

Since it doesn't handle multi-line variables, it destroys your functions.
"

When I type env at the command line I see this

module=() {  eval `/usr/bin/modulecmd bash $*`
}

Original comment by rd.alt.5...@gmail.com on 29 Sep 2014 at 8:04