Open 0xc1c4da opened 1 year ago
Thanks for reporting this. There is an issue how we track uniquely sampled token sequences in caching. I will have a look. This also affects local models.
For the omission of n
, it behaves like a standard default keyword argument i.e. n=4
is defined in the implementation of beam_sample
. I agree however, that we should probably validate more to avoid the case of beam(n, 4)
, where both n
and 4
will actually be ignored and not past to the decoder function or validated in any other way.
2 issues:
n
iebeam_sample(n, 1.0)
defaults to returning 4 results.calling the following query with
sampled_thoughts = sample_thoughts(4)
returns:
However calling the query with
sampled_thoughts = [sample_thoughts(1, prompt) for _ in range(4) ]
returns expected unique samples: