Open tmm1 opened 1 month ago
uses unsafe + std::mem::transmute to re-use encoder keys as decoder values and in the sorted_token_bytes list
this should be safe because all these objects share a lifetime inside CoreBPE
results in memory savings, and performance improvements in some scenarios
uses unsafe + std::mem::transmute to re-use encoder keys as decoder values and in the sorted_token_bytes list
this should be safe because all these objects share a lifetime inside CoreBPE
results in memory savings, and performance improvements in some scenarios