gnosis / prediction-market-agent-tooling

Tools to benchmark, deploy and monitor prediction market agents.
GNU Lesser General Public License v3.0
13 stars 0 forks source link

Refactor adding of Langfuse callbacks in PMAT, PMA, Prophet #346

Closed kongzii closed 2 weeks ago

kongzii commented 4 weeks ago

Check out this comment https://github.com/gnosis/prediction-market-agent-tooling/pull/345/files#r1718271854

How could it be done nicer? Some ideas:

  1. Refactor these lines into a separate function, so it would be like
config = prepare_config_with_langfuse(existing_config, add_langfuse_callback=add_langfuse_callback)

Where existing_config could be just None.

  1. Accept callbacks in function arguments, and always create a second function that will pass the callback, if add_langfuse_callback is True:
def infer_category(..., callbacks=None): ...

def infer_category_observed(...):
  callbacks = [init langfuse here]
  return infer_category(..., callbacks)
  1. Monkey-patch all Langchain invoke methods, automatically adding the langfuse callback. But this would be nicer as an experimental feature in Langfuse itself.

  2. Create a wrapper class around Langchain classes that we can import from PMAT, and it will include the callback automatically.

  3. ???

kongzii commented 4 weeks ago

I asked them about (3) https://github.com/orgs/langfuse/discussions/2949

kongzii commented 2 weeks ago

I think this can be closed, https://github.com/orgs/langfuse/discussions/2949 isn't getting much traction and the current solution of using get_langfuse_config function + observing everything by default, because it can be (1) easily disabled and (2) un-observed version of underlying function can be accessed by __wrapped__ attribute.