search

iris-hep / idap-200gbps-atlas

benchmarking throughput with PHYSLITE
6 stars 1 forks source link

Fix pod startup #83

Closed gordonwatts closed 4 months ago

gordonwatts commented 4 months ago

When a new science pod starts up, it starts up a transformer and a side-car. When the transformer starts up, it tries to contact the side-car.

Unfortunately, the transformer was starting up faster than the side-car. This caused the transformer to crash. k8s would then restart it, and it would work just fine at that point because the side-car was up and running.

This wastes about 10 seconds burning wall clock time and also adds overhead to k8s, and thus slows down the time a new pod is ready to do work.

gordonwatts commented 4 months ago

https://github.com/ssl-hep/ServiceX/commit/f65595a515fadc6a70269f867cf28033bcaac114 - fix is to sleep for 1 second. Artificial, but works for now.