Research LLama AI for rewriting

[gpseal]

David has suggested we investigate using LLanma2 for rewriting our articles: https://ai.meta.com/llama/ https://huggingface.co/blog/llama2 https://huggingface.co/meta-llama/Llama-2-13b-chat-hf

[gpseal]

Looks like we will need a powerful VM to run Llama 2.

Suggested requirements / prerequisites: https://aituts.com/llama/

Suggested specs: https://learn.microsoft.com/en-us/azure/virtual-machines/sizes-gpu

https://www.hardware-corner.net/guides/computer-to-run-llama-ai-model/

Azure price calculator for VMs https://azure.microsoft.com/en-us/pricing/calculator/?&ef_id=_k_Cj0KCQjwz8emBhDrARIsANNJjS6RoyPAMkaxLzqBYPkQMHI1hoEpKef5WL_ZiVrcRIaYFw59kosI0DoaAix9EALw_wcB_k_&OCID=AIDcmmp3m06wb2_SEM__k_Cj0KCQjwz8emBhDrARIsANNJjS6RoyPAMkaxLzqBYPkQMHI1hoEpKef5WL_ZiVrcRIaYFw59kosI0DoaAix9EALw_wcB_k_&gad=1&gclid=Cj0KCQjwz8emBhDrARIsANNJjS6RoyPAMkaxLzqBYPkQMHI1hoEpKef5WL_ZiVrcRIaYFw59kosI0DoaAix9EALw_wcB

Quite Pricey by the look of things

[gpseal]

I have put together requirements and possible VM options that would meet our requirements for LLaMA 2. These have been sent to rob with the assumption that we will only need the VM for 10-20 hours total

LLaMA Model	Model Size	Minimum VRAM Requirement	Recommended GPU Examples	RAM/Swap to Load*
LLaMA-7B	3.5GB	6GB	RTX 3060, GTX 1660, 2060, AMD 5700 XT, RTX 3050	16 GB
LLaMA-13B	6.5GB	10GB	AMD 6900 XT, RTX 2060 12GB, 3060 12GB, 3080, A2000	32 GB
LLaMA-30B	15.8GB	20GB	RTX 3080 20GB, A4500, A5000, 3090, 4090, 6000, Tesla V100, Tesla P40	64 GB
LLaMA-65B	31.2GB	40GB	A100 40GB, 2x3090, 2x4090, A40, RTX A6000, 8000	128 GB

Source: https://aituts.com/llama/

Azure GPU accelerated VMs

Size	vCPU	Memory: GiB	Temp Storage: GiB	GPU	GPU mem: GiB	Price p/hr
NC24	24	224	1440	4	48	$6.199
NC64as T4 v3	64	440	2880	4	64	$5.68
NV24	24	224	1440	4	32	$6.358
NV48s v3	48	448	1280	4	32	$6.358
NV72ads a10 v5	72	880	2880	2	48	$9.454