Support connection draining in kubernetes environments

[ferraith]

Hello!

We would like to support autoscaling (HPA) for our Buildbarn frontends. To ensure that scaling in of frontends is working without immediately terminating active requests I'm interested if the bb-storage application does a graceful shutdown after a SIGTERM signal was sent to the pods. I already checked the bb-storage config and found an interesting flag called stop_gracefully: https://github.com/buildbarn/bb-storage/blob/995e5867c1fe5b77149658218b97e3c95a000cc4/pkg/proto/configuration/grpc/grpc.proto#L237

Is that flag meaningful to enable support for connection draining?

Would be great if you could share your thoughts about that topic.

[EdSchouten]

Even though you could use it in principle, do note this comment right above it:

  // In the general case, it is recommended to leave this option unset,
  // as enabling this option may lead to underutilization and slow
  // rolling upgrades. As clients are generally capable of performing
  // retries, it is often sufficient to just terminate immediately.

Just make sure you call Bazel with --remote_retries= set to some meaningful value. If you make sure to do that, then there is absolutely no harm in performing rolling upgrades, regardless of whether stop_gracefully is set.

[ferraith]

@EdSchouten Thx a lot for your reply!