launchflow / buildflow

BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.
https://docs.launchflow.com/buildflow
Apache License 2.0
193 stars 7 forks source link

Metrics need to be injected as a dependency to work. #315

Open boetro opened 1 year ago

boetro commented 1 year ago

Describe the bug If you use ray metrics inside of a processor the metrics do not get properly exported. I think this is cause ray deregisters metrics when they get garbage collected.

The current workaround is to inject the metric into a replica dependency to avoid it being garbage collected.

Probably the best work around is to make buildflow metrics keep a reference to the metric around, this would allow them to be properly exported.

To Reproduce Steps to reproduce the behavior:

  1. define a metric inside your processor or as a global metric
  2. increment the metric inside your process
  3. observe it not being reported

Expected behavior I would expect the metric to be exported