YahooArchive / oozie

Oozie - workflow engine for Hadoop
http://yahoo.github.com/oozie/
Apache License 2.0
373 stars 160 forks source link

add support for an (Oozie) system share lib directory in HDFS for action binaries. #106

Closed tucu00 closed 14 years ago

tucu00 commented 14 years ago

This issue builds on top of GH-0027

The motivation for GH-0027 is to allow workflow apps to easily share a common set of JAR/SO libraries without having to bundle those JAR/SO libs in each workflow app lib/ directory.

The motivation for this issue is to allow the (Oozie) system to use set a default set of JAR/OS libraries in the workflow application path. A useful feature when a setup is defining a common stack (i.e. for pig, streaming, hive, sqoop, inputformats, UDFs libs, etc) and a workflow app just want to use the stack.

To maintain backward compatibility, by default this system defined 'share lib' directory is not included. Workflow jobs must set a flag in their job properties for the inclusion to happen.

A future version of workflow apps would revert the default behavior.

tucu00 commented 14 years ago

Closed by 40228888f52aa5a18b048d992316f536ba70e59c add support for an (Oozie) system share lib directory in HDFS for action binaries.

This issue builds on top of GH-0027

The motivation for GH-0027 is to allow workflow apps to easily share a common set of JAR/SO libraries without having to bundle those JAR/SO libs in each workflow app lib/ directory.

The motivation for this issue is to allow the (Oozie) system to use set a default set of JAR/OS libraries in the workflow application path. A useful feature when a setup is defining a common stack (i.e. for pig, streaming, hive, sqoop, inputformats, UDFs libs, etc) and a workflow app just want to use the stack.

To maintain backward compatibility, by default this system defined 'share lib' directory is not included. Workflow jobs must set a flag in their job properties for the inclusion to happen.

If the 'oozie.use.system.libpath' property is specified in the job properties, the system lib specified in the configuration property 'oozie.service.WorkflowAppService.system.libpath' is added to application path.