More verbose output for tool calls / responses

[dexhorthy]

so far we're havin a lot of fun over here, had a question - I'd imagine there's probably an easy-ish way to do this, but I haven't done a ton w/ prefect so bear with me.

I'm interested to know not just when tool calls finish, but also when they start, and what params they were started with. This helps a lot during development as I'm trying to document the functions/tools I'm building and make sure the LLM can use them well.

I did manage to spin up the prefect web UI and start a server in a another shell, and trigger a deployment from a third shell, but wondering if there's a way to just up the verbosity of the logs a little bit without leaving the CLI / IDE

Here's a subset of the code, minus the tools impls

researcher = cf.Agent(
    name="Researcher",
    instructions="Use a formal tone and clear language",
    tools=[
        list_meetings,
        list_zoom_meeting_recordings,
        transcribe_recording,
        download_zoom_file,
        split_local_file,
        get_file_size,
    ]
)

person = "<redacted>"

@cf.flow
def research() -> str:
    zoomid = cf.Task(
        f"""
        tell me the zoom meeting ID for my call w/ {person}? 
        """,
        agents=[researcher]
    )

    downloadURL = cf.Task(
        """
        What is the download URL for the m4a audio file from that meeting? 
        """,
        context={"meeting_id": zoomid},
        agents=[researcher]
    )

    summary = cf.Task(
        """
        download the file  to recording.m4a
        """,
        context={"download_url": downloadURL},
        agents=[researcher]
    )

    summary = cf.Task(
        """
        transcribe the file and then tell me the highlights of the meeting.

        You can split the file in half if its too big. The size limit for whisper transcription is 25MB.
        """,
        context={"filename": "recording.m4a"},
        agents=[researcher]
    )
    return summary

def main():
    result = research()
    print(result)

if __name__ == "__main__":
    main()

I'd be looking for a way to opt-in to something like showing the params in, and maybe the first 200 chars of the response if it's real big

Tool call: list_zoom_meeting_recordings(meeting_id=123456789)
Tool call: ✅  w/ response `[{"meeting_id":123456789, "downloadable_files":[{"type":"video", "url": ....
---------------
Tool call: download_file(download_url=..., filename="dex meeting from 2024-06-24.m4a")
Tool call: ❌   w/ response `{status: 503, message: "something is broken try back later"}`

just riffing here - let me know what y'all think or if I missed a flag/option

[aaazzam]

Hey @dexhorthy! Oh man you're reading our minds.

We're trying to figure out the right representation for this -- have been playing with lowlevel logs, rich printing, and something as big as a tui a la textual. Don't want to overdo it though.

@jlowin there's a flag for this at the moment now, right? But maybe concurrent tui frames have been wacky or something?

[jlowin]

Yes -- as of a week ago we printed everything and the feedback was that it's too much, which is how we ended up in this more compressed state. I want to make this configurable. We do have an experimental TUI intended to let you basically expand things dynamically as they're running but it's not 100% right now. I'm going to open a tracking issue for making verbosity more configurable. never mind that's obviously what this is...

In the meantime set CONTROLFLOW_TOOLS_VERBOSE=1 (or controlflow.settings.tools_verbose=True at runtime) to print everything

[dexhorthy]

for sure yeah the runtime setting change worked (didn't try the env var yet) - thanks!

I can see what y'all mean about the verbose one being real extra loud 🙂 - the "show me the inputs" was great but the "show me the outputs" could stand to be sliced or otherwise summarized. Or maybe I just need to make leaner tools 😉

[dexhorthy]

Thinking more on this, I guess what I really want is some kind of streaming or event based interface where I can implement what happens on_tool_call / on_tool_result

That would let me customize what I print and also make it easy to stitch these cf flows or tasks into parts of a eg fastapi server and stream progress back to the user