search

jaehyeon-kim / flink-demos

Apache Flink (Pyflink) and Related Projects
https://jaehyeon.me/tags/apache-flink/
23 stars 10 forks source link

Issue related to env.set_parallelism on local environment. #18

Closed waynelxb closed 1 month ago

waynelxb commented 2 months ago

When running the samples on local WINDOWS machine, we may run into issue like the following:

Traceback (most recent call last): File "D:\Projects\GitHub\flink-demos-master\building-pyflink-apps\src\s11_transformation.py", line 15, in env.execute() File "d:\Projects\GitHub\flink-demos-master\building-pyflink-apps.venv\lib\site-packages\pyflink\datastream\stream_execution_environment.py", line 764, in execute return JobExecutionResult(self._j_stream_execution_environment.execute(j_stream_graph)) File "d:\Projects\GitHub\flink-demos-master\building-pyflink-apps.venv\lib\site-packages\py4j\java_gateway.py", line 1322, in call return_value = get_return_value( File "d:\Projects\GitHub\flink-demos-master\building-pyflink-apps.venv\lib\site-packages\pyflink\util\exceptions.py", line 146, in deco return f(*a, kw) File "d:\Projects\GitHub\flink-demos-master\building-pyflink-apps.venv\lib\site-packages\py4j\protocol.py", line 326, in get_return_value raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o0.execute.** : org.apache.flink.runtime.client.JobExecutionException: Job execution failed.

I find this issue could be related to env.set_parallelism. If add or set "env.set_parallelism = 1", such issue will be gone.

Thanks

jaehyeon-kim commented 2 months ago

Flink supports distributed computing and it wouldn't be useful if we have to set parallelism to 1. Can you try on non-Windows? You may try WSL2 or docker.

waynelxb commented 2 months ago

Flink supports distributed computing and it wouldn't be useful if we have to set parallelism to 1. Can you try on non-Windows? You may try WSL2 or docker.

Sure! I will give it a try and let you know soon. Also, yes, I found this issue on windows local machine.

jaehyeon-kim commented 1 month ago

Is there an update? Please close if it gets fixed.

waynelxb commented 1 month ago

Thank you so much!!!!!