Vertex streaming and buffer killing

[r0man]

Hi @ahyatt,

here's another PR. The Vertex provider wasn't using the streaming endpoint. Now it does, and the tester ran successfully with it.

The other change is that I'm now killing the response buffer when the request is complete. There is however one situation where I don't know how to do this: For sync requests that raise an exception. The error thrown does not have any handle to that buffer. I removed the change to adds the process to the plz-response. That could have gave us access to the process buffer.

[ahyatt]

Thanks for catching this! See, this kind of thing is where debug output from plz comes in handy - otherwise it's hard to see that I'm using the wrong url.

[r0man]

Hi @ahyatt ,

I agree about the debug output. To debug the issue with the Vertex provider using the wrong URL, the first I did, was adding a message form to the llm-plz-request function to print the url. Having this behind some flag in a debug buffer is useful. So, we could give this a try in plz-media-type-request. We can test drive it there. So I could print the following to a buffer:

• The request url, method and headers. The request body under some feature flag? (it could be large).
• The response status and headers. Again, the body under some feature flag? How to print the body in a streaming/non-streaming case? (I would suggest to print the "body" in terms of the objects/concepts of the media type. This could be a "event" for a streaming media type, or a complete document in a media type that is handled synchronously.

I know the llm library added something like this recently. I can take a look at this and try to come up with something. If you have anything specific in mind, please let me know.

[ahyatt]

I'm glad you are looking into this!

The llm debug is showing abstracted information of what is sent to the LLM, but this would be specific information on the connection. I think these would be pretty different.

In general, the debug output should be a separate track of work; right now I think bringing all your work back to the plz library is the next step, and we should separately bring up debug of plz as a separate feature request.