GET /v4/default/registry/instances ... Connection was closed

[sxcooler]

There's a lot of this log in my production:

2019-06-05 11:50:21,689 |-ERROR[registry-vert.x-eventloop-thread-0] LINE: 95 TraceId:- ServiceName:- in org.apache.servicecomb.serviceregistry.client.http.RestUtils : GET /v4/default/registry/instances?rev=865440c1754ef7dfe66be741a48e22ee6e8a5ecc&appId=risk&global=true&serviceName=risk%3Alai2-risk&env=production&version=0.0.0%2B fail, endpoint is service-center:30100, message: Connection was closed

And finally it turns out like this:

2019-06-04 14:09:25.302 WARN 1 --- [io-8888-exec-32] o.a.s.bizkeeper.BizkeeperHandler : bizkeeper execution error, info=cause:InvocationException,message:InvocationException: code=490;msg=CommonExceptionData [message=Cse Internal Bad Request];cause:CseException,message:No available address found. microserviceName=lai2-user, version=0.0.0.0+, discoveryGroupName=0.0.0.0+/empty/empty/instancesAll/

Then my gateway got "Forwarding error" and my in-app invocation got errors (because the registry of instances may not be there) I tried this:

servicecomb:
  ...
  request:
    timeout: 30000

But some of the services goes well and some other didn't work! Could anyone plz dig into it to find the real reason?

[sxcooler]

Here is my pom:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Licensed to the Apache Software Foundation (ASF) under one or more
  ~ contributor license agreements.  See the NOTICE file distributed with
  ~ this work for additional information regarding copyright ownership.
  ~ The ASF licenses this file to You under the Apache License, Version 2.0
  ~ (the "License"); you may not use this file except in compliance with
  ~ the License.  You may obtain a copy of the License at
  ~
  ~     http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.baolongxing.loan</groupId>
        <artifactId>lai2-parent</artifactId>
        <version>1.0-SNAPSHOT</version>
        <relativePath/>
    </parent>

    <artifactId>lai2-gateway</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.servicecomb</groupId>
            <artifactId>spring-boot-starter-servicecomb</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-zuul</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-ribbon</artifactId>
            <exclusions>
                <exclusion>
                    <artifactId>jsr311-api</artifactId>
                    <groupId>javax.ws.rs</groupId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-hystrix</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.servicecomb</groupId>
            <artifactId>spring-boot-starter-discovery</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.servicecomb</groupId>
            <artifactId>transport-rest-vertx</artifactId>
        </dependency>
        <dependency>
            <groupId>com.baolongxing.loan.common</groupId>
            <artifactId>loan-common</artifactId>
            <version>1.0-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>

    <!--for package and deploy-->
    <build>
        <finalName>lai2-gateway</finalName>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>2.6</version>
                <configuration>
                    <archive>
                        <manifestEntries>
                            <Class-Path>.</Class-Path>
                        </manifestEntries>
                    </archive>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

And this is my microservice.yaml in production:

#More details can be found :
# 1.http://servicecomb.apache.org/users/service-definition/
# 2.http://servicecomb.apache.org/users/service-configurations/
# 3.http://servicecomb.apache.org/users/communicate-protocol/

#Indicates an application name
APPLICATION_ID: zhaohui
service_description:
  #Indicates a microservice name
  #The microservice name should be unique within an application.
  #The name can contain digits, uppercase and lowercase letters, hyphens(-), underscores(_), and periods(.); and can neither start nor end with punctuations.
  #The naming rule is as follows: ^[a-zA-Z0-9]+$|^[a-zA-Z0-9][a-zA-Z0-9_-.]*[a-zA-Z0-9]$.
  name: lai2-gateway
  #Indicates a service version
  version: 1.0.0
  environment: production
servicecomb:
  service:
    #Specifies the service center IP address.
    registry:
      address: http://service-center:30100
  #Specifies the rest transport listening IP address.
#  rest:
#    address: 0.0.0.0:8080
#  #Specifies the highway transport listening IP address.
#  highway:
#    address: 0.0.0.0:7070
  request:
    timeout: 30000

[yhs0092]

Hello, the error log shows that your microservice instance is trying to send request to the address service-center:30100, which seems not a common address of ServiceComb-Service-Center. Do you run a service-center instance and can you get access to it by the hostname service-center?

[sxcooler]

Hello, the error log shows that your microservice instance is trying to send request to the address service-center:30100, which seems not a common address of ServiceComb-Service-Center. Do you run a service-center instance and can you get access to it by the hostname service-center?

Yes I can, this is the local domain in our k8s environment.

[sxcooler]

And I can read the error text literally.

[sxcooler]

The question is this error occasionally happened on some service while other services are doing good.

[xiaoxiong581]

I also see this issue, one pod register micro-services to servicecenter always wait timeout, then occur exception Connection was closed by url :Post /v4/default/registry/microservices, but when service in other pods work fine

version: 1.2.0.B006

[sxcooler]

I don't really know the reason of this issue, but here are some suggestions: First, set timeout like above. then, make sure you've set publishAddress as k8s ingress address, as default servicecomb will use pods address, like "foo-bar-99788d85d-8nqx5", this will cause some error when your pods scaling. Now I've got 2-3 "GET /v4/default/registry/instances ... Connection was closed" per day, I think it's acceptable.

servicecomb:
  ...
  service:
    publishAddress: foo-bar
  request:
    timeout: 30000

And in your gateway's config (I'm using zuul):

zuul:
  ...
  host:
    maxTotalConnections: 1000
    maxPerRouteConnections: 500
ribbon:
  eureka:
    enabled: false
  eager-load:
  ConnectTimeout: 5000
  ReadTimeout: 15000
  SocketTimeout: 15000
  MaxTotalHttpConnections: 1000
  MaxConnectionsPerHost: 500
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds: 60000

Closing this issue for now.