Apple's cross entropy computation

unslothai / unsloth

Finetune Llama 3.2, Mistral, Phi, Qwen 2.5 & Gemma LLMs 2-5x faster with 80% less memory

https://unsloth.ai

Apache License 2.0

18.37k stars 1.28k forks source link

Closed fzyzcjy closed 4 days ago

fzyzcjy commented 4 days ago

Hi thanks for Unsloth! Today I see a paper https://openreview.net/forum?id=E4Fk3YuG56 (code: https://github.com/apple/ml-cross-entropy), which seems to discuss a way to compute cross entropy. Thus I share this here in case it is useful for this library.

shimmyshimmer commented 4 days ago

On it! There's already a previous issue for this! :) https://github.com/unslothai/unsloth/issues/1298

fzyzcjy commented 4 days ago

Looks great, thank you!