Access to listener_names within before_service_worker_ready

[ollym]

We're trying to setup a monitoring service within before_service_worker_ready however when we try and Pitchfork.listener_names from within this block we get an error. How do you advise we can do this?

[casperisfine]

Yes, the service worker has the listeners closed, so that it doesn't appear as listening: https://github.com/Shopify/pitchfork/blob/ec75abc15c4da4c2d0520bca9038b51d6e7e2015/lib/pitchfork/http_server.rb#L930

You didn't give any detail on what solution you are using for monitoring. Is it Raindrops or something else?

For us it isn't too much of a problem because we receive the listeners as environment, so it's not really duplication.

Right now you can work around this via:

      listener_names = nil

      after_mold_fork do |_, _|
        listener_names = Pitchfork.listener_names
      end

      before_service_worker_ready do |server, service|
        $stderr.puts listener_names
      end

It's a bit hacky, but I just tested it, it works.

I guess Pitchfork could be changed to record Pitchfork.listener_names before closing it so it remain accessible for the service worker, so you wouldn't need such a workaround.

[ollym]

Hi yea we're using Raindrops, for now we simplified it by hard coding the bind address and stopped relying on Pitchfork.listener_names entirely.

How are you monitoring pitchfork? We're using yabeda + prometheus and our collection script now looks a bit like this:

pitchfork.workers.set({}, Pitchfork::Info.workers_count)
pitchfork.live_workers.set({}, live_workers_count = Pitchfork::Info.live_workers_count)

if defined?(Raindrops::Linux.tcp_listener_stats)
  listener_address = "0.0.0.0:#{ENV.fetch('PORT', 3000)}"
  stats = Raindrops::Linux.tcp_listener_stats([listener_address])[listener_address]
  pitchfork.active_workers.set({}, stats.active)
  pitchfork.request_backlog.set({}, stats.queued)

  pitchfork.live_capacity.set({}, live_workers_count - stats.active)
  pitchfork.live_busy_percentage.set({}, live_workers_count.zero? ? 100.0 : (stats.active * 100.0) / live_workers_count)
end

[casperisfine]

More or less the same. Raindrops and some StatsD metrics.

One thing you may want to be aware of, is that currently every call to tcp_listener_stats allocate an IO instance and doesn't eagerly close it, so you you do this a lot in a process that doesn't do much more, it can accumulate a lot of file descriptors until GC finally trigger: https://yhbt.net/raindrops-public/20230926214000.M564322@dcvr/T/#e73072adeb83bb8bc74ac6829d1446e316309edc4

So we use this fork for now: https://github.com/casperisfine/raindrops/tree/file-descriptor-leak

[casperisfine]

As for Pitchfork.listener_names, I think it would make sense to just compute it before closing the listeners so it can continue to return the same value. PR welcome, but otherwise I might do it at some point.

[ollym]

What are your thoughts on inlining listener_stats to pitchfork? I've seen you dropped the dependency on raindrops but maybe just inline the function it was providing?

Pitchfork.listener_stats

And taking care of the leak in the process? This kind of monitoring feels pretty mission critical that everyone will need to do at some point.

[casperisfine]

Yeah, I considered some form of that. Was more thinking of doing a companion gem as it could be used by other servers, but I'd like to stop depending on raindrops at Shopify, just haven't gotten around to do it.