search

awslabs / python-deequ

Python API for Deequ
Apache License 2.0
702 stars 132 forks source link

Why the callback server always listens to 25334 by default #19

Open ricdong opened 3 years ago

ricdong commented 3 years ago

Why the callback server always listens to 25334 by default, which will cause the pyspark applications to not run on the same server at the same time. I tried to modify the listening port before starting the callback server like below:

scala_utils.py self.gateway.start_callback_server(callback_server_parameters = CallbackServerParameters(port = 0))

this can work on the different port for each pyspark application, but the Checks Failure with the constraint_message:

Can't execute the assertion: Error while obtaining a new communication channel

Does anyone have any suggestions on this issue ?

yk-st commented 3 years ago

plz. same here

cghyzel commented 3 years ago

Could you modify the listening port to a valid port #? Say 25444

dash-hugo-vanthournhout commented 1 year ago

We have the same issue here, only when run locally, it is working on aws EMR.

How can we modify the listening port while using pydeequ?

pydeequ -> 1.0.1 spark -> 3.1.3 java -> 11.0.16

Ashokgoa commented 9 months ago

@ricdong did you implement any workaround?