tf serving with different port

[marziehoghbaie]

Hi, I'm trying to change the port number for docker when serving a model. in most toturials the port number is 8501, but i want to use it on 8502. But when I change it enter the event loop but for inference I get the following error: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) I run this on a gpu server and the following is the output of docker ps: 85aaaa50ab1f tensorflow/serving "/usr/bin/tf_serving…" 59 seconds ago Up 56 seconds 8500-8501/tcp, 0.0.0.0:8502->8502/tcp zen_wiles

[gowthamkpr]

@marziehoghbaie You cannot change the port number for docker as it has already been configured in TF serving as shown here. Please use port 8500 for gRPC and 8501 for REST API.

[marziehoghbaie]

TNX, so it's not possible to run a tf serving docker for different models simultaneously? when I do so, each model enters the loop, but for inference I get this error: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

[wdirons]

What is the docker run command your using? As mentioned above port 8500 is used for gRPC and 8501 is used for the REST API inside of the container. You can make this port on the host whatever number you want.

For example:

docker run -t --rm -p 8502:8501 \
    -v "$TESTDATA/saved_model_half_plus_two_cpu:/models/half_plus_two" \
    -e MODEL_NAME=half_plus_two \
    tensorflow/serving &

Will bind port 8502 on the host, to port 8501 of the docker container. That being the rest api port.

[marziehoghbaie]

thank you so much, problem's solved, I was wrong at port mapping.

[roarjn]

Hi, I am seeing the same error using the tensorflow/serving:latest-gpu image. I do not see this error when using the tensorflow/serving:latest image. I am pointing to the right 8501 port.

[marziehoghbaie]

Hi, are you following the correct notation? ""Will bind port 8502 on the host, to port 8501 of the docker container. That being the rest api port."" your_port_No:port_No_on_docker(which is 8501)

[NikhilShaw]

I am trying to run multiple containers at once and want them to access via different ports using REST API. TF serving only uses 8501 for the REST API. Any solution @marziehoghbaie so that I can use different ports for REST API call ?

[wdirons]

@NikhilShaw , You need to bind different ports on the host to the 8501 port of each container.

Something like: Host Port Container Container Port 8502 container1 8501 8503 container2 8501 8504 container3 8501

See my example above for doing this for a single container: https://github.com/tensorflow/serving/issues/1627#issuecomment-631654789