search

Netflix / PigPen

Map-Reduce for Clojure
Apache License 2.0
566 stars 55 forks source link

allow custom properties to be passed to FlowConnector when creating a flow #161

Open fs111 opened 9 years ago

fs111 commented 9 years ago

It is a common pattern for Cascading users to supply a Properties object to the FlowConnector to control many aspects of a flow. It would be great if PigPen allowed for that as well:

http://docs.cascading.org/cascading/2.7/javadoc/cascading-core/cascading/flow/FlowConnector.html

mbossenbroek commented 9 years ago

You can pass a user-defined FlowConnector to PigPen and have it use that to construct the flow: https://github.com/Netflix/PigPen/blob/2caa566a8a1f5c98cc1a5a65fc961c20ce066dfb/pigpen-cascading/src/main/clojure/pigpen/cascading.clj#L42

Is that sufficient or would you like to see an overload that takes a properties map and a query and constructs the default HadoopFlowConnector using those properties?

fs111 commented 9 years ago

The overloaded variant feels more natural to a Cascading user, I'd say