mir-am
commented
4 months ago This is a pretty useful feature for TestSpark, as we discussed at ICSE'24, especially for people who don't want to send their code to cloud-hosted LLMs for privacy reasons. However, besides the challenges mentioned, using LLMs locally (1B+ size) can be computationally expensive depending on developers' machine specs like CPU and GPU vendors. Essentially, it needs different optimizations depending on the hardware. For example, with decent speed, LLMs can be run with TensorRT-LLM on Nvidia GPUs.
Also, aside from HuggingFace, deepinfra provides an OpenAI-compatible API to query some of the latest LLMs like Llama 3 at a relatively cheap price. Disclaimer: I am NOT affiliated with deepinfra and I recently started to use their service for personal projects.
pderakhshanfar
Description
We would like to add the integration between TestSpark and the HuggingFace platform to allow the use of more (smaller) models for test generation. Using models should be possible both using Inference APIs and also using models locally.
Challenges
Many models in this platform are not as powerful as the GPT models, and hence, we also need some prior evaluation based on the existing models to find the most suitable way of using these models in TestSpark. Can we integrate as part of our LLM-based test generation approach? Or they can be used, for instance, to improve the tests generated by our local test generators.