search

containous / traefik-extra-service-fabric

Traefik extra: Service Fabric Provider
Apache License 2.0
12 stars 14 forks source link

Error retrieving serviceExtensionMap using ServiceFabric. "Version" not properly encoded when added to the URL. #59

Open MonDeveloper opened 3 years ago

MonDeveloper commented 3 years ago

Do you want to request a feature or report a bug?

Bug

What did you do?

Simply started the traefik exe pointing to my SF Cluster with no special configuration except the SF Cluster provider part

What did you expect to see?

It could start, connect to SF Cluster properly and recognize the existing services

What did you see instead?

It crashes when it try to get further information through the SF Client API due to a malformed URL. Service Fabric has no limitation on the Version format for both Applications and Services, it means it is possible to find Version Labels using some offending character that need to be encoded when added to the URL.

This is our case. From the log pasted here under, you can easily find the ERROR is generated because this URL is used: http://cac-sf-02.icc.crifnet.com:19080/ApplicationTypes/Crif.REApp.010.3rdPARTIES__Type/$/GetServiceTypes?api-version=6.0&ApplicationTypeVersion=2.0.0 [20200719.194357-7] and I have verified the same error is returned if I put the same URL into Postman but If I properly encode the final part of the URL (the version parameter value) from 2.0.0 [20200719.194357-7] to 2.0.0%20%5B20200719.194357-7%5D the Postman with the new URL stars working as expected: http://cac-sf-02.icc.crifnet.com:19080/ApplicationTypes/Crif.REApp.010.3rdPARTIES__Type/$/GetServiceTypes?api-version=6.0&ApplicationTypeVersion=2.0.0%20%5B20200719.194357-7%5D

Output of traefik version: (What version of Traefik are you using?)

2020/08/12 12:18:34 Using high precision timer
Version:      v1.7.26
Codename:     maroilles
Go version:   go1.14.6
Built:        2020-07-28_03:45:27PM
OS/Arch:      windows/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

################################################################
# Global configuration
################################################################

debug = true
logLevel = "DEBUG"
defaultEntryPoints = ["http", "https"]

[entryPoints]
  [entryPoints.http]
    address = ":99"
  [entryPoints.traefik]
    address = ":8099"

################################################################
# API definition
################################################################

[api]
  entryPoint = "traefik"
  dashboard = true
  debug = true

################################################################
# Service Fabric provider
################################################################

# Enable Service Fabric configuration backend
[servicefabric]

# Service Fabric Management Endpoint
clustermanagementurl = "http://cac-sf-02.icc.crifnet.com:19080"
apiversion = "6.0"

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

PS C:\Users\738marmonduc.ICC\Downloads\traefik> .\traefik_windows-amd64.exe --configfile=traefik.toml
2020/08/12 12:20:22 Using high precision timer
INFO[2020-08-12T12:20:23+02:00] Using TOML configuration file C:\Users\738marmonduc.ICC\Downloads\traefik\traefik.toml
INFO[2020-08-12T12:20:23+02:00] Traefik version v1.7.26 built on 2020-07-28_03:45:27PM
DEBU[2020-08-12T12:20:23+02:00] Global configuration loaded {"LifeCycle":{"RequestAcceptGraceTimeout":0,"GraceTimeOut":10000000000},"GraceTimeOut":0,"Debug":true,"CheckNewVersion":true,"SendAnonymousUsage":false,"AccessLogsFile":"","AccessLog":null,"TraefikLogsFile":"","TraefikLog":null,"Tracing":null,"LogLevel":"DEBUG","EntryPoints":{"http":{"Address":":99","TLS":null,"Redirect":null,"Auth":null,"WhitelistSourceRange":null,"WhiteList":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}},"traefik":{"Address":":8099","TLS":null,"Redirect":null,"Auth":null,"WhitelistSourceRange":null,"WhiteList":null,"Compress":false,"ProxyProtocol":null,"ForwardedHeaders":{"Insecure":true,"TrustedIPs":null}}},"Cluster":null,"Constraints":[],"ACME":null,"DefaultEntryPoints":["http","https"],"ProvidersThrottleDuration":2000000000,"MaxIdleConnsPerHost":200,"IdleTimeout":0,"InsecureSkipVerify":false,"RootCAs":null,"Retry":null,"HealthCheck":{"Interval":30000000000},"RespondingTimeouts":null,"ForwardingTimeouts":null,"AllowMinWeightZero":false,"KeepTrailingSlash":false,"Web":null,"Docker":null,"File":null,"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":null,"Zookeeper":null,"Boltdb":null,"Kubernetes":null,"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null,"ServiceFabric":{"Watch":false,"Filename":"","Constraints":null,"Trace":false,"TemplateVersion":0,"DebugLogGeneratedTemplate":false,"ClusterManagementURL":"http://cac-sf-02.icc.crifnet.com:19080","APIVersion":"6.0","RefreshSeconds":0,"TLS":null,"AppInsightsClientName":"","AppInsightsKey":"","AppInsightsBatchSize":0,"AppInsightsInterval":0},"Rest":null,"API":{"EntryPoint":"traefik","Dashboard":true,"Debug":true,"CurrentConfigurations":null,"Statistics":null},"Metrics":null,"Ping":null,"HostResolver":null}
INFO[2020-08-12T12:20:23+02:00]
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://docs.traefik.io/v1.7/basics/#collected-data
WARN[2020-08-12T12:20:23+02:00] clientTLS is nil
INFO[2020-08-12T12:20:23+02:00] Preparing server http &{Address::99 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0009c28e0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s
INFO[2020-08-12T12:20:23+02:00] Preparing server traefik &{Address::8099 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] WhiteList:<nil> Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc0009c28c0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s
INFO[2020-08-12T12:20:23+02:00] Starting server on :99
INFO[2020-08-12T12:20:23+02:00] Starting provider configuration.ProviderAggregator {}
INFO[2020-08-12T12:20:23+02:00] Starting server on :8099
INFO[2020-08-12T12:20:23+02:00] Starting provider *servicefabric.Provider {"Watch":false,"Filename":"","Constraints":null,"Trace":false,"TemplateVersion":0,"DebugLogGeneratedTemplate":false,"ClusterManagementURL":"http://cac-sf-02.icc.crifnet.com:19080","APIVersion":"6.0","RefreshSeconds":10000000000,"TLS":null,"AppInsightsClientName":"","AppInsightsKey":"","AppInsightsBatchSize":0,"AppInsightsInterval":0}
INFO[2020-08-12T12:20:33+02:00] Checking service fabric config
ERRO[2020-08-12T12:20:33+02:00] Error retrieving serviceExtensionMap: error requesting service extensions: Service Fabric responded with error code 400 Bad Request to request http://cac-sf-02.icc.crifnet.com:19080/ApplicationTypes/Crif.REApp.010.3rdPARTIES__Type/$/GetServiceTypes?api-version=6.0&ApplicationTypeVersion=2.0.0 [20200719.194357-7] with body &{0xc0008ea040 {0 0} false <nil> 0x733d90 0x733d10}
ERRO[2020-08-12T12:20:33+02:00] error requesting service extensions: Service Fabric responded with error code 400 Bad Request to request http://cac-sf-02.icc.crifnet.com:19080/ApplicationTypes/Crif.REApp.010.3rdPARTIES__Type/$/GetServiceTypes?api-version=6.0&ApplicationTypeVersion=2.0.0 [20200719.194357-7] with body &{0xc0008ea040 {0 0} false <nil> 0x733d90 0x733d10}
ERRO[2020-08-12T12:20:33+02:00] Error retrieving serviceExtensionMap: error requesting service extensions: Service Fabric responded with error code 400 Bad Request to request http://cac-sf-02.icc.crifnet.com:19080/ApplicationTypes/Crif.REApp.110.CFF-PTF-CORE__Type/$/GetServiceTypes?api-version=6.0&ApplicationTypeVersion=2.0.0 [20200719.221114-155] - cff 7.9.1-SNAPSHOT with body &{0xc000056600 {0 0} false <nil> 0x733d90 0x733d10}
ERRO[2020-08-12T12:20:33+02:00] error requesting service extensions: Service Fabric responded with error code 400 Bad Request to request http://cac-sf-02.icc.crifnet.com:19080/ApplicationTypes/Crif.REApp.110.CFF-PTF-CORE__Type/$/GetServiceTypes?api-version=6.0&ApplicationTypeVersion=2.0.0 [20200719.221114-155] - cff 7.9.1-SNAPSHOT with body &{0xc000056600 {0 0} false <nil> 0x733d90 0x733d10}
ERRO[2020-08-12T12:20:33+02:00] Error retrieving serviceExtensionMap: error requesting service extensions: Service Fabric responded with error code 400 Bad Request to request http://cac-sf-02.icc.crifnet.com:19080/ApplicationTypes/Crif.REApp.110.CFF-PTF-CORE__Type/$/GetServiceTypes?api-version=6.0&ApplicationTypeVersion=2.0.0 [20200719.221114-155] - cff 7.9.1-SNAPSHOT with body &{0xc0008ea200 {0 0} false <nil> 0x733d90 0x733d10}
ERRO[2020-08-12T12:20:33+02:00] error requesting service extensions: Service Fabric responded with error code 400 Bad Request to request http://cac-sf-02.icc.crifnet.com:19080/ApplicationTypes/Crif.REApp.110.CFF-PTF-CORE__Type/$/GetServiceTypes?api-version=6.0&ApplicationTypeVersion=2.0.0 [20200719.221114-155] - cff 7.9.1-SNAPSHOT with body &{0xc0008ea200 {0 0} false <nil> 0x733d90 0x733d10}
INFO[2020-08-12T12:20:34+02:00] I have to go...
INFO[2020-08-12T12:20:34+02:00] Stopping server gracefully
DEBU[2020-08-12T12:20:34+02:00] Waiting 10s seconds before killing connections on entrypoint http...
DEBU[2020-08-12T12:20:34+02:00] Waiting 10s seconds before killing connections on entrypoint traefik...
DEBU[2020-08-12T12:20:34+02:00] Entrypoint http closed
DEBU[2020-08-12T12:20:34+02:00] Entrypoint traefik closed
INFO[2020-08-12T12:20:34+02:00] Server stopped
INFO[2020-08-12T12:20:34+02:00] Shutting down
MonDeveloper commented 3 years ago

Moreover, as a worst side effect, the system enters into a never ending loop where it coutinuously invoke those API with no delay at all (thousands of calls per minute) without properly handling the TCP/HTTP socket and after a few minutes the entire machine stucks completely due to a network resourse shortage.

This is an example of an Error taken on the machine trying to use any other software that need to open a network socket

Error: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

Shooting down the Traefik application everything return operative on the machine.

Sezamo commented 3 years ago

Hi there! I've exactly the same issue!

It took me a while to understand the root cause, after trying to stop unresponsive servers, unreachable systems,, bloated logs... a nightmare!

Any solution over there?

Thanks, Maurizio.