Since I am probably going to use the library now more and send some more patches in case I will find some real/potential problems (like #133 ) I think it may be usefull to add some autotests to the library.
So far it covers the following cases:
CUDA kernel
forward pass.
Basically this test see the model next-token-predictions on some phrase:
check the most probable next token for every current token is the same as some canonical sequence
check if most probable next token scores are similar to some canonical range
so it checks if model output in all states is the same as once-recorded standard attention + disabled "faster" mode
Attention types:
(covered) Standard one
(covered) Flash attention
(not-covered) XFormers
Act/no act order models:
(not covered) act order models
(covered) no-act order models
"Faster" matmul4bit modes
(covered) disable
(covered) old_faster
(covered) faster
finetuning.
Basically, here I just check if I can succesfully overfit LoRA model to remember some phrase much better than the original LLaMA model
Attention types
(covered) Standard
(not covered) Flash attention (since flash-attn can not compute backward pass for such a models using customer-grade GPUs like my 2080Ti - I can not test if the test will work now)
(not covered) XFormers
Act/no act order models:
(not covered) act order models
(covered) no-act order models
"Faster" matmul4bit modes
(covered) disable
(covered) old_faster
(covered) faster
(not covered) Triton kernel. Since I am using Windows system now I can't check if triton kernels will give the same result as CUDA ones
So what drawbacks may be improved in the future patches (but probably it may be useful in the current state):
Triton kernel tests, I will probably do it using WSL a few days later
Since I am probably going to use the library now more and send some more patches in case I will find some real/potential problems (like #133 ) I think it may be usefull to add some autotests to the library.
So far it covers the following cases:
So what drawbacks may be improved in the future patches (but probably it may be useful in the current state):
So, now to run these tests you need to do something like
not sure what is the best way to make pytest run installation before tests (or is it good idea at all)