[Configuration Lazy Loading]: Failed to access external service

[ydh926]

Bug description After enabling configuration lazy loading, external services cannot be accessed normally

Affected sub-moudle (please put an X in all that apply)

[x] Configuration Lazy Loading [ ] Http Plugin Management [ ] Adaptive Ratelimit [ ] Slime Boot

Steps to reproduce the bug

1. Set port 80 as a lazy loading port， using slime-boot as follow：

   apiVersion: v1
   items:
   - apiVersion: config.netease.com/v1alpha1
   kind: SlimeBoot
   metadata:
   name: lazyload
   namespace: mesh-operator
   spec:
   component:
     globalSidecar:
       enable: true
       namespace:
       - default
       resources:
         limits:
           cpu: 500m
           memory: 200Mi
         requests:
           cpu: 500m
           memory: 200Mi
     pilot:
       enable: true
       image:
         repository: docker.io/bcxq/pilot
         tag: preview-1.3.7-v0.0.1
       resources:
         limits:
           cpu: 500m
           memory: 200Mi
         requests:
           cpu: 500m
           memory: 200Mi
     reportServer:
       enable: true
       image:
         repository: docker.io/bcxq/mixer
         tag: preview-1.3.7-v0.0.1
       resources:
         limits:
           cpu: 500m
           memory: 200Mi
         requests:
           cpu: 500m
           memory: 200Mi
   image:
     pullPolicy: Always
     repository: docker.io/bcxq/slime
     tag: preview-v0.0.1
   module:
   - fence:
       enable: true
       wormholePort:
       - "80"
     name: lazyload

2. Access external services in the service container with lazy loading enabled. For example, execute apt-get update command, it will return 503, as follows:

   root@productpage-v1-64794f5db4-2pgl4:/opt/microservices# apt-get update
   Err:1 http://security.debian.org/debian-security buster/updates InRelease
   503  Service Unavailable [IP: 151.101.110.132 80]
   Err:2 http://deb.debian.org/debian buster InRelease
   503  Service Unavailable [IP: 151.101.110.132 80]
   Err:3 http://deb.debian.org/debian buster-updates InRelease
   503  Service Unavailable [IP: 151.101.110.132 80]
   Reading package lists... Done
   W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  503  Service Unavailable [IP: 151.101.110.132 80]
   W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  503  Service Unavailable [IP: 151.101.110.132 80]
   W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  503  Service Unavailable [IP: 151.101.110.132 80]
   W: Some index files failed to download. They have been ignored, or old ones used instead.

[ydh926]

Before solving this problem, we should try to avoid setting the port commonly used by external services as wormhole Port.

[vjoke]

Is there any plan to fix this bug?

[ydh926]

@caitong93 , Did you have any ideas?

[caitong93]

The cause of this problem is that global sidecar send traffic to external services to the Passthrough Cluster, but it cannot get the original destination and return 503. To solve that we need to pass original destination to global sidecar. A simple workaround is configure request_headers_to_add in RouteConfiguration as following and set use_http_header in OriginalDstLbConfig to true , but that only works for Http. For non-http traffic we need to investigate other possible techniques(maybe we can try to passthrough traffic to external service before it goes to the global sidecar).

request_headers_to_add:
- header: 
     key: x-envoy-original-dst-host
     value: "%DOWNSTREAM_LOCAL_ADDRESS%" 

[vjoke]

I feel that It's unexpected behavior to forward external service traffic to global sidecar, which will introduce another hop and bring more latency. The external service traffic should be routed to PassthroughCluster in the sidecar itself.

[caitong93]

I feel that It's unexpected behavior to forward external service traffic to global sidecar, which will introduce another hop and bring more latency. The external service traffic should be routed to PassthroughCluster in the sidecar itself.

Agree with you. I feel we can use some matching rules like cidr to identify external traffic and route that to PassthroughCluster.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[YonkaFang]

we've provided an approach and will'be released at the v0.2.1 version.