Tests

[alex4321]

Since I am probably going to use the library now more and send some more patches in case I will find some real/potential problems (like #133 ) I think it may be usefull to add some autotests to the library.

So far it covers the following cases:

• CUDA kernel
  • forward pass. Basically this test see the model next-token-predictions on some phrase:
    • check the most probable next token for every current token is the same as some canonical sequence
    • check if most probable next token scores are similar to some canonical range
    • so it checks if model output in all states is the same as once-recorded standard attention + disabled "faster" mode
    • Attention types:
      • (covered) Standard one
      • (covered) Flash attention
      • (not-covered) XFormers
    • Act/no act order models:
      • (not covered) act order models
      • (covered) no-act order models
    • "Faster" matmul4bit modes
      • (covered) disable
      • (covered) old_faster
      • (covered) faster
  • finetuning. Basically, here I just check if I can succesfully overfit LoRA model to remember some phrase much better than the original LLaMA model
  • Attention types
    • (covered) Standard
    • (not covered) Flash attention (since flash-attn can not compute backward pass for such a models using customer-grade GPUs like my 2080Ti - I can not test if the test will work now)
    • (not covered) XFormers
    • Act/no act order models:
      • (not covered) act order models
      • (covered) no-act order models
    • "Faster" matmul4bit modes
      • (covered) disable
      • (covered) old_faster
      • (covered) faster
• (not covered) Triton kernel. Since I am using Windows system now I can't check if triton kernels will give the same result as CUDA ones

So what drawbacks may be improved in the future patches (but probably it may be useful in the current state):

• Triton kernel tests, I will probably do it using WSL a few days later
• All attention types
• It depends on the fact https://huggingface.co/TheBloke/orca_mini_7B-GPTQ/tree/main exists on huggingface hub
• Act-order model tests

So, now to run these tests you need to do something like

python -m pip install . # To rebuild native stuff and generally reinstall the package
python -m pytest # To run tests

not sure what is the best way to make pytest run installation before tests (or is it good idea at all)

[johnsmith0031]

Thanks for your contribution!