search

nathanmarz / cascalog

Data processing on Hadoop without the hassle.
Other
1.38k stars 179 forks source link

Broken vector arguments #217

Closed r0man closed 10 years ago

r0man commented 10 years ago

Hello everyone,

I think there is something broken when using vectors as arguments within a Cascalog query. This query used to work in 1.10.2 but fails with 2.0.0:

(let [source [{:a 1} {:a 2}]]
  (?<- (stdout)
   [?m ?a]
   (source :> ?m)
   (get-in ?m [:a] :> ?a)))

In Cascalog 2.0.0 I get a java.lang.IllegalArgumentException exception, with the appended stacktrace. I added a test case for this problem over here:

https://github.com/r0man/cascalog/blob/broken-args/cascalog-core/test/cascalog/api_test.clj#L1020

Roman

13/12/01 21:02:22 INFO util.HadoopUtil: using default application jar, may cause class not found exceptions on the cluster
13/12/01 21:02:22 INFO planner.HadoopPlanner: using application jar: /home/roman/.m2/repository/cascading/cascading-hadoop/2.2.0/cascading-hadoop-2.2.0.jar
13/12/01 21:02:22 INFO flow.Flow: [] starting
13/12/01 21:02:22 INFO flow.Flow: []  source: MemorySourceTap["MemorySourceScheme[[UNKNOWN]->[ALL]]"]["/080c0620-1cf1-470e-b128-ddb8f807c2f2"]
13/12/01 21:02:22 INFO flow.Flow: []  sink: StdoutTap["SequenceFile[[UNKNOWN]->['?m', '?a']]"]["/tmp/temp810292291642204319938226188950606"]
13/12/01 21:02:22 INFO flow.Flow: []  parallel execution is enabled: false
13/12/01 21:02:22 INFO flow.Flow: []  starting jobs: 1
13/12/01 21:02:22 INFO flow.Flow: []  allocating threads: 1
13/12/01 21:02:22 INFO flow.FlowStep: [] starting step: (1/1) ...1642204319938226188950606
13/12/01 21:02:22 INFO flow.FlowStep: [] submitted hadoop job: job_local_0290
13/12/01 21:02:22 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@11024072
13/12/01 21:02:22 INFO mapred.MapTask: numReduceTasks: 0
13/12/01 21:02:22 INFO hadoop.FlowMapper: cascading version: 2.2.0
13/12/01 21:02:22 INFO hadoop.FlowMapper: child jvm opts: -Xmx200m
13/12/01 21:02:22 INFO hadoop.FlowMapper: sourcing from: MemorySourceTap["MemorySourceScheme[[UNKNOWN]->[ALL]]"]["/080c0620-1cf1-470e-b128-ddb8f807c2f2"]
13/12/01 21:02:22 INFO hadoop.FlowMapper: sinking to: StdoutTap["SequenceFile[[UNKNOWN]->['?m', '?a']]"]["/tmp/temp810292291642204319938226188950606"]
13/12/01 21:02:22 ERROR stream.TrapHandler: caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [ccb0d6e9-7637-4b37-93c...][sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)] operator Each failed executing operation
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascalog.ops.KryoInsert.operate(KryoInsert.java:34)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:73)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:34)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascading.operation.Identity$2.operate(Identity.java:137)
    at cascading.operation.Identity.operate(Identity.java:150)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.SourceStage.map(SourceStage.java:102)
    at cascading.flow.stream.SourceStage.run(SourceStage.java:58)
    at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:127)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Keyword
    at clojure.lang.RT.seqFrom(RT.java:505)
    at clojure.lang.RT.seq(RT.java:486)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$reduce1.invoke(core.clj:890)
    at clojure.core$get_in.invoke(core.clj:5553)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at cascalog.logic.def$eval17441$fn__17442.doInvoke(NO_SOURCE_FILE:51)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at cascalog.ClojureCascadingBase.applyFunction(ClojureCascadingBase.java:74)
    at cascalog.ClojureMap.operate(ClojureMap.java:35)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    ... 22 more
13/12/01 21:02:22 ERROR stream.TrapHandler: caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [ccb0d6e9-7637-4b37-93c...][sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)] operator Each failed executing operation
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascalog.ops.KryoInsert.operate(KryoInsert.java:34)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:73)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:34)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascading.operation.Identity$2.operate(Identity.java:137)
    at cascading.operation.Identity.operate(Identity.java:150)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.SourceStage.map(SourceStage.java:102)
    at cascading.flow.stream.SourceStage.run(SourceStage.java:58)
    at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:127)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Keyword
    at clojure.lang.RT.seqFrom(RT.java:505)
    at clojure.lang.RT.seq(RT.java:486)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$reduce1.invoke(core.clj:890)
    at clojure.core$get_in.invoke(core.clj:5553)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at cascalog.logic.def$eval17441$fn__17442.doInvoke(NO_SOURCE_FILE:51)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at cascalog.ClojureCascadingBase.applyFunction(ClojureCascadingBase.java:74)
    at cascalog.ClojureMap.operate(ClojureMap.java:35)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    ... 22 more
13/12/01 21:02:22 ERROR stream.TrapHandler: caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [ccb0d6e9-7637-4b37-93c...][sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)] operator Each failed executing operation
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascalog.ops.KryoInsert.operate(KryoInsert.java:34)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:73)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:34)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascading.operation.Identity$2.operate(Identity.java:137)
    at cascading.operation.Identity.operate(Identity.java:150)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.SourceStage.map(SourceStage.java:102)
    at cascading.flow.stream.SourceStage.run(SourceStage.java:58)
    at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:127)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Keyword
    at clojure.lang.RT.seqFrom(RT.java:505)
    at clojure.lang.RT.seq(RT.java:486)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$reduce1.invoke(core.clj:890)
    at clojure.core$get_in.invoke(core.clj:5553)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at cascalog.logic.def$eval17441$fn__17442.doInvoke(NO_SOURCE_FILE:51)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at cascalog.ClojureCascadingBase.applyFunction(ClojureCascadingBase.java:74)
    at cascalog.ClojureMap.operate(ClojureMap.java:35)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    ... 22 more
13/12/01 21:02:22 ERROR stream.TrapHandler: caught Throwable, no trap available, rethrowing
cascading.pipe.OperatorException: [ccb0d6e9-7637-4b37-93c...][sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)] operator Each failed executing operation
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascalog.ops.KryoInsert.operate(KryoInsert.java:34)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:73)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:34)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascading.operation.Identity$2.operate(Identity.java:137)
    at cascading.operation.Identity.operate(Identity.java:150)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.SourceStage.map(SourceStage.java:102)
    at cascading.flow.stream.SourceStage.run(SourceStage.java:58)
    at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:127)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Keyword
    at clojure.lang.RT.seqFrom(RT.java:505)
    at clojure.lang.RT.seq(RT.java:486)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$reduce1.invoke(core.clj:890)
    at clojure.core$get_in.invoke(core.clj:5553)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at cascalog.logic.def$eval17441$fn__17442.doInvoke(NO_SOURCE_FILE:51)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at cascalog.ClojureCascadingBase.applyFunction(ClojureCascadingBase.java:74)
    at cascalog.ClojureMap.operate(ClojureMap.java:35)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    ... 22 more
13/12/01 21:02:22 ERROR stream.SourceStage: caught throwable
cascading.pipe.OperatorException: [ccb0d6e9-7637-4b37-93c...][sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)] operator Each failed executing operation
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascalog.ops.KryoInsert.operate(KryoInsert.java:34)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:73)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:34)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascading.operation.Identity$2.operate(Identity.java:137)
    at cascading.operation.Identity.operate(Identity.java:150)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.SourceStage.map(SourceStage.java:102)
    at cascading.flow.stream.SourceStage.run(SourceStage.java:58)
    at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:127)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Keyword
    at clojure.lang.RT.seqFrom(RT.java:505)
    at clojure.lang.RT.seq(RT.java:486)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$reduce1.invoke(core.clj:890)
    at clojure.core$get_in.invoke(core.clj:5553)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at cascalog.logic.def$eval17441$fn__17442.doInvoke(NO_SOURCE_FILE:51)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at cascalog.ClojureCascadingBase.applyFunction(ClojureCascadingBase.java:74)
    at cascalog.ClojureMap.operate(ClojureMap.java:35)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    ... 22 more
13/12/01 21:02:22 WARN mapred.LocalJobRunner: job_local_0290
cascading.pipe.OperatorException: [ccb0d6e9-7637-4b37-93c...][sun.reflect.GeneratedConstructorAccessor28.newInstance(Unknown Source)] operator Each failed executing operation
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascalog.ops.KryoInsert.operate(KryoInsert.java:34)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:73)
    at cascading.flow.stream.FilterEachStage.receive(FilterEachStage.java:34)
    at cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)
    at cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:119)
    at cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:107)
    at cascading.operation.Identity$2.operate(Identity.java:137)
    at cascading.operation.Identity.operate(Identity.java:150)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
    at cascading.flow.stream.SourceStage.map(SourceStage.java:102)
    at cascading.flow.stream.SourceStage.run(SourceStage.java:58)
    at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:127)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:214)
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Keyword
    at clojure.lang.RT.seqFrom(RT.java:505)
    at clojure.lang.RT.seq(RT.java:486)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$reduce1.invoke(core.clj:890)
    at clojure.core$get_in.invoke(core.clj:5553)
    at clojure.lang.AFn.applyToHelper(AFn.java:163)
    at clojure.lang.AFn.applyTo(AFn.java:151)
    at clojure.core$apply.invoke(core.clj:617)
    at cascalog.logic.def$eval17441$fn__17442.doInvoke(NO_SOURCE_FILE:51)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.lang.AFunction$1.doInvoke(AFunction.java:29)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at cascalog.ClojureCascadingBase.applyFunction(ClojureCascadingBase.java:74)
    at cascalog.ClojureMap.operate(ClojureMap.java:35)
    at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
    ... 22 more
13/12/01 21:02:22 WARN flow.FlowStep: [] task completion events identify failed tasks
13/12/01 21:02:22 WARN flow.FlowStep: [] task completion events count: 0
13/12/01 21:02:22 INFO flow.Flow: [] stopping all jobs
13/12/01 21:02:22 INFO flow.FlowStep: [] stopping: (1/1) ...1642204319938226188950606
13/12/01 21:02:22 INFO flow.Flow: [] stopped all jobs
13/12/01 21:02:22 INFO util.Hadoop18TapUtil: deleting temp path /tmp/temp810292291642204319938226188950606/_temporary
Quantisan commented 10 years ago

@sritchie quick question: is it a design choice to make all parameters be passed in via anonymous function, or is passing param directly and mixing with var still acceptable? e.g. ((get-in* [:a]) ?m :> ?out) vs (get-in ?m [:a] :> ?out)

sritchie commented 10 years ago

Hey, just saw this - it's acceptable to mix, and this example does need to pass.

The old syntax that's now gone was passing in higher-order parameters with the vector in the first position. That was a cascalog-only thing that didn't map into clojure.

Now, you can either pass in parameters directly via the vector, or create a closure. This is definitely a bug.

sorenmacbeth commented 10 years ago

I'm not sure of the fix exactly yet, but the source of the bug is here:

https://github.com/sorenmacbeth/cascalog/tree/develop/cascalog-core/src/clj/cascalog/cascading/operations.clj#L681

When the vector arg is added as an Insert, jacknife.seq/flatten is getting called on it.

r0man commented 10 years ago

Thanks for fixing this!

r0man commented 10 years ago

Can we get a new release? This fix was blocking our codebase to update to Cascalog 2. Now everything is cool and all tests are passing. Thanks again.

dollschasingmen commented 10 years ago

+1 on getting this into a new release -- we are in the same positon w/r/t to upgrading. thanks!

sorenmacbeth commented 10 years ago

I'm happy to do a soft 2.0.1 release, i.e., just do the release and pushing to clojars. the only thing that's been holding me back is that I haven't had time to write an annoucemt email and change log

On Monday, March 10, 2014, dollschasingmen notifications@github.com wrote:

+1 on getting this into a new release -- we are in the same positon w/r/t to upgrading. thanks!

Reply to this email directly or view it on GitHubhttps://github.com/nathanmarz/cascalog/issues/217#issuecomment-37249208 .

http://about.me/soren

dollschasingmen commented 10 years ago

at neat, thanks -- i think we are going to roll w/ 2.01-SNAPSHOT for now, but would be great if this was an official release. get-in is huge for us :)