Open JenySadadia opened 7 months ago
I noticed DNS resolution is unreliable for last few days on Azure services in general, it is affecting even deploy scripts. Unfortunately not much we can do yet,we might add more DNS servers in network config
It's happening again, it seems. If these services are meant to be long-lived could we introduce any kind of mechanism to re-launch them before we move to production. Not a good idea at this moment, since some of them are still under development and could exit due to a programming error, and we don't want to keep re-launching them in those cases.
I added 3 more resolver entry on staging host, but not sure it will help anyhow with docker services, will investigate more now
After starting API and Pipeline services, the services worked fine for some time. Then suddenly
monitor
,tarball
, andscheduler-k8s
services stopped. Other pipeline and API services were running OK while this issue was observed.Error logs:
It seems like something is blocking the pipeline services from accessing API. Maybe some Sysadmin related issue? @nuclearcat