search

neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
2.94k stars 169 forks source link

[V2 Logger] logger middleware #1543

Closed horheynm closed 5 months ago

horheynm commented 6 months ago
Screenshot 2024-01-17 at 2 04 47 PM

Example output 


"[metric.JoinOutput.max][⏱️0.0009205341339111328] max(JoinOutput['generated_tokens]'): 13513", "
[metric.JoinOutput.max][⏱️0.0009205341339111328] max(JoinOutput['generated_logits]'): 20.59090805053711",
[metric.GenerateNewTokenOperator.max][⏱️2.6226043701171875e-05] max(GenerateNewTokenOperator[0]['finish_reason]'): max_new_tokens", 
"[metric.GenerateNewTokenOperator.max][⏱️2.6226043701171875e-05] max(GenerateNewTokenOperator[1]['token_generator]'.token_frequencies): 1.0"