search

simonw / llm-replicate

LLM plugin for models hosted on Replicate
Apache License 2.0
59 stars 6 forks source link

Feature request: log the prediction time #14

Closed jamesscottbrown closed 1 year ago

jamesscottbrown commented 1 year ago

The docs for the replicate HTTP API say that the response includes a metrics object with a predict_time attribute.

It would be useful to log this to the SQLite database (in addition to the duration_ms field that is already logged, and which is apparently measured by the client), since it is the prediction time on the server that is used for the billing calculation.

jamesscottbrown commented 1 year ago

On further thought, it looks like the charges for each API request are obtained by rounding to the nearest second, so the duration_ms is close enough for.