Closed josharian closed 2 months ago
In case it is helpful, here's my current implementation using cgo.
func (m *Model) Decode(tokens []uint32) string {
if len(tokens) == 0 {
return ""
}
cres := C.decode(m.proc, (*C.uint32_t)(&tokens[0]), (C.size_t)(len(tokens)))
if cres == nil {
panic("failed to decode")
}
out := C.GoString(cres)
C.free(unsafe.Pointer(cres))
return out
}
char *decode(struct cspm_processor *proc, const uint32_t *input, size_t length) {
std::vector<std::string> pieces;
for (size_t i = 0; i < length; i++) {
pieces.push_back(proc->processor.IdToPiece(input[i]));
}
std::string result;
proc->processor.Decode(pieces, &result);
return strdup(result.c_str());
}
Understood. Decoding wasn't really in scope when I set to work on this; my goal was to implement the encoder. Decoding would definitely require some additional work.
Added a decoder in v0.4.0, and renamed s/Encoder/Processor/ (aligned with the name in the C++/Python implementation)
https://pkg.go.dev/github.com/eliben/go-sentencepiece#Processor.Decode
Please reopen if needed
I see an Encoder, but to use the output of a gemma model I need to Decode too.
It might be better to rename
Encoder
toModel
or the like, so it can do decoding as well.