search

ml-explore / mlx-examples

Examples in the MLX framework
MIT License
6.3k stars 898 forks source link

Return logprobs from generate api #1016

Closed farris closed 1 month ago

farris commented 1 month ago

Continuation of https://github.com/ml-explore/mlx-examples/issues/894#issuecomment-2236464052

Could be incorporated into new function altogether but proposing to build into existing generate as the logic is minimal...

Thanks 😄

fblissjr commented 1 month ago

I actually did this myself when it first came up in the PR to introduce logprobs. Comment here: https://github.com/ml-explore/mlx-examples/pull/806#issuecomment-2211931951

Yours makes more sense from a library standpoint though. I would find it way more useful in utils, personally.

I use logprobs and the associated tokens/vocab quite often.

farris commented 1 month ago

I actually did this myself when it first came up in the PR to introduce logprobs. Comment here: #806 (comment)

Yours makes more sense from a library standpoint though. I would find it way more useful in utils, personally.

I use logprobs and the associated tokens/vocab quite often.

Nice, maybe the functionality should be shared between the server implementation which you did and the regular generate interface

@awni , thoughts?