Python batch API example issues

VojtechBartos commented 5 years ago

Hi guys, i am playing with flink and would like to try python batch API. Unfortunately having some issues even with examples contained in the docker itself.

using docker-compose example from the docker hub https://hub.docker.com/_/flink/ (using docker flink:1.6.1)
using following Dockerfile to get python installed
```
FROM flink:1.6.1
```

RUN apt-get -y update && apt-get -y install python python-dev python-pip

3. connecting into the job manager container
4. executing following command 
```sh
/opt/flink/bin/pyflink.sh /opt/flink/examples/python/batch/WordCount.py

This is CLI output:

Starting execution of program
Failed to run plan: Job failed. (JobID: e1a88feaf33c99926bde821ca9f35de3)

The program didn't contain a Flink job. Perhaps you forgot to call execute() on the execution environment.

Actual exception for the job taken from UI:

java.io.IOException: Cannot run program "kill": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
    at java.lang.Runtime.exec(Runtime.java:620)
    at java.lang.Runtime.exec(Runtime.java:485)
    at org.apache.flink.python.api.streaming.data.PythonStreamer.destroyProcess(PythonStreamer.java:223)
    at org.apache.flink.python.api.streaming.data.PythonStreamer.close(PythonStreamer.java:198)
    at org.apache.flink.python.api.functions.PythonMapPartition.close(PythonMapPartition.java:69)
    at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:43)
    at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:507)
    at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    ... 10 more

Any idea what could be wrong?

Thanks!

VojtechBartos commented 5 years ago

(Update)

This actually works when i install flink locally with following commands

git clone https://github.com/apache/flink
cd flink
mvn clean install -DskipTests
/build-target/bin/pyflink.sh <path to the example>

Anyway, still would be nice to run it inside the docker ;)

eightrivers commented 5 years ago

@VojtechBartos The same issue we meet now... and we noticed that the java runtime using in the docker is OpenJDK 1.8.0_181. We tried serveral times and found that the exception "Cannot run program "kill": error=2, No such file or directory" was thrown out if run the code " Runtime.getRuntime().exec(new String[] { "kill", "-9", "1234" }); ", even there is no flink in the environment.

So maybe it is reasonable to say that this issue is not resulted from flink but from openjdk.

docker-flink / docker-flink

Python batch API example issues #55