Azure / oms-log-analytics-firehose-nozzle

Microsoft Azure Log Analytics Nozzle
Apache License 2.0
11 stars 14 forks source link

Nozzle app is crashing with DNS lookup issues #31

Open bpada00 opened 4 years ago

bpada00 commented 4 years ago

2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR panic: Could not get api /v2/info: Get https://api.system.prod.**.**.com/v2/info: dial tcp: lookup api.system.prod.stratus.albertsons.com on 169.254.0.2:53: read udp 10.255.79.151:42306->169.254.0.2:53: i/o timeout 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR goroutine 1 [running]: 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager.(logger).Fatal(0xc4201204e0, 0x5595c357001b, 0x17, 0x5595c38f8940, 0xc420160850, 0x0, 0x0, 0x0) 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/vendor/code.cloudfoundry.org/lager/logger.go:152 +0x4d4 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/caching.(Caching).Initialize(0xc420172540, 0x5595c31f4101) 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/caching/caching.go:99 +0x39f 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle.(*OmsNozzle).Start(0xc420186000, 0xc420156320, 0xc420184000) 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/omsnozzle/oms_nozzle.go:69 +0x3f 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR main.main() 2020-01-20T15:50:59.71-0800 [APP/PROC/WEB/4] ERR /tmp/gobuildpack.gopath476918737/.go/src/github.com/Azure/oms-log-analytics-firehose-nozzle/main.go:187 +0x1af2

bpada00 commented 4 years ago

We feel that the app is getting too many DNS lookup requests that it cannot handle and hence crashing. Please let us know what you think. For now, we have added the below setting to the manifest file and the app is stable.

https://golang.org/pkg/net/#hdr-Name_Resolution export GODEBUG=netdns=cgo # force cgo resolver