search

GEOFBOT / flink

Mirror of Apache Flink. See `new-iterations-with-multiops` branch for Python API bulk iterations implementation
Apache License 2.0
0 stars 0 forks source link

The result of .flat_map cannot be directly used to close bulk iterations #1

Open GEOFBOT opened 8 years ago

GEOFBOT commented 8 years ago 
iteration = [...].flat_map(lambda x, c: [x, x])
iterator.close_with(iteration)

...causes...

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error.
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:524)
    at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
    at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:331)
    at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:775)
    at org.apache.flink.client.CliFrontend.run(CliFrontend.java:251)
    at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1003)
    at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1046)
Caused by: java.lang.RuntimeException: Error while creating the data flow plan for the program: Unknown operator or data set type: null
    at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:104)
    at org.apache.flink.api.java.operators.OperatorTranslation.translateBulkIteration(OperatorTranslation.java:196)
    at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:94)
    at org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:124)
    at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:88)
    at org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:124)
    at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:88)
    at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:63)
    at org.apache.flink.api.java.operators.OperatorTranslation.translateToPlan(OperatorTranslation.java:51)
    at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1035)
    at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1002)
    at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:58)
    at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896)
    at org.apache.flink.python.api.PythonPlanBinder.runPlan(PythonPlanBinder.java:150)
    at org.apache.flink.python.api.PythonPlanBinder.main(PythonPlanBinder.java:115)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509)
    ... 6 more
magsol commented 8 years ago

That's problematic. Do we need to explore other ways to perform the outer product? On Wed, Aug 3, 2016 at 19:53 Geoffrey Mon notifications@github.com wrote:

iteration = [...].flat_map(lambda x, c: [x, x]) iterator.close_with(iteration)

...causes...

The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:524) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:331) at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:775) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:251) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1003) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1046) Caused by: java.lang.RuntimeException: Error while creating the data flow plan for the program: Unknown operator or data set type: null at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:104) at org.apache.flink.api.java.operators.OperatorTranslation.translateBulkIteration(OperatorTranslation.java:196) at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:94) at org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:124) at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:88) at org.apache.flink.api.java.operators.OperatorTranslation.translateSingleInputOperator(OperatorTranslation.java:124) at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:88) at org.apache.flink.api.java.operators.OperatorTranslation.translate(OperatorTranslation.java:63) at org.apache.flink.api.java.operators.OperatorTranslation.translateToPlan(OperatorTranslation.java:51) at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1035) at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1002) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:58) at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:896) at org.apache.flink.python.api.PythonPlanBinder.runPlan(PythonPlanBinder.java:150) at org.apache.flink.python.api.PythonPlanBinder.main(PythonPlanBinder.java:115) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:509) ... 6 more

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/GEOFBOT/flink/issues/1, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIQ-Wib_z1nyi8bjP-VPs56Adl7vPddks5qcSnjgaJpZM4JcNeW .

iPhone'd

GEOFBOT commented 8 years ago

I can get around this issue by grouping by a dummy field and then using a group reduce function instead of a flat map. I made this issue so I know what to fix before I make a pull request for Python bulk iterations.

GEOFBOT commented 7 years ago

Note to self: as of today, still an issue