epic: Structure Manual QA for cortex.cpp

[dan-homebrew]

Goal

• cortex.cpp

Tasklist

• [x] Build up Manual QA checklist for cortex.cpp
  • [ ] Hardware compatibility matrix
  • [ ] Should cover key stories
  • [ ] Should include column that tracks "automated testing" (i.e. is there automated tests to cover)
• [ ] Create releases folder in cortex.cpp, where the QA checklist should be committed together with key learnings, lessons to preserve for younger engineers

[dan-homebrew]

@gabrielle-ong For v1.0's Manual QA, I would like us to focus on answering the following questions:

1. How do I provision all Hardware/OS types to do the testing
2. Does the Installer install the correct files to their respective locations?
3. Does the Installer install the correct version of llama.cpp?
4. Is the user able to pull, run and chat with llama3.1 successfully?
5. Does the Uninstaller uninstall all files, and not leave any dangling references?

I provide more context and links to key issues below:

Provisioning of Hardware/OS

Most of our bugs come from when cortex.cpp breaks on some hardware/OS. Catching these will reduce the number of bugs in Jan.

• We have invested in internal hardware that covers most critical/common hardware and OS types
• We should be able to provision clean VMs to test Jan for QA (speak to @hiento09)
• For each release, we should QA this cortex.cpp on all hardware and OS that we are compatible with

• 1125

Additionally, I would like to document our process provisioning a VM. This may grow to become its own product in the future (e.g. Menlo Cloud), as other teams may also need cross-platform testing.

Testing Installer

Installers are OS-specific, and install a clear set of files for each operating system. Installers will also need to detect hardware and pull the correct version of llama.cpp (I will cover that in the next section).

First, we will need to verify that the Installer for a particular operating system writes the correct files to the correct locations. The following issues/discussions have relevant context:

• https://github.com/janhq/cortex.cpp/discussions/1114
• https://github.com/janhq/cortex.cpp/issues/1030

Testing Hardware Detection and llama.cpp binary

Installers will also detect the user's hardware, and then pull the correct version of llama.cpp. This is based on:

• Operating System (e.g. Windows, Mac, Linux)
• CPU Extension (e.g. AVX-2, noAVX, AVX-512)
• GPU Acceleration (e.g. CUDA11, CUDA12, Vulkan, sycl, etc)

We should be aligned with the versions published by llama.cpp: https://github.com/ggerganov/llama.cpp/releases

We need to verify that an AVX-2 system is correctly identified as an AVX-2 system, and the correct version is pulled.

We have an open issue, discussing how llama.cpp is installed, which may change how we QA this: https://github.com/janhq/cortex.cpp/issues/1217

Pulling and Running Models on llama.cpp

We should verify that Pulling and Running Models writes to the correct Model Folder, and with model.yaml. The following issues/discussions have relevant context:

• https://github.com/janhq/cortex.cpp/issues/1154

Testing the OpenAI-compatible API, and CLI commands

We should verify that key API endpoints work:

• Postman collection testing (e.g. manual testing)
• Later on, the eng will likely implement some automated tests using the OpenAI library (e.g. for /chat/completions)
• The key things to verify are the non-OpenAI endpoints, e.g. model loading, model starting, etc

We should also verify that key CLI commands work:

• e.g. cortex run, etc

Uninstallation

We should verify that Cortex's uninstaller removes all relevant files, and does not leave dangling references.

Updater

We should verify that Cortex's updater works, but this may be challenging for now.

[0xSage]

Manual checklist template:

• https://www.notion.so/jan-ai/QA-Cortex-Sprint-20-1065b91df6b1809bbb3cdb153d5ed14e?pvs=4

Bugs

• https://github.com/janhq/cortex.cpp/issues/1257
• https://github.com/janhq/cortex.cpp/issues/1259
• https://github.com/janhq/cortex.cpp/issues/1266
• https://github.com/janhq/cortex.cpp/issues/1264
• https://github.com/janhq/cortex.cpp/issues/1263
• https://github.com/janhq/cortex.cpp/issues/1269
• https://github.com/janhq/cortex.cpp/issues/1270
• https://github.com/janhq/cortex.cpp/issues/1274
• https://github.com/janhq/cortex.cpp/issues/1273
• https://github.com/janhq/cortex.cpp/issues/1272

Not QA'ed

Issues that are still in progress:

• model.yaml structure
  • Atm, downloaded models duplicate yamls in /model_id folder and in /models, whcih will likely be fixed

Questions

• https://github.com/janhq/cortex.cpp/issues/1262
• https://github.com/janhq/cortex.cpp/issues/1261
• https://github.com/janhq/cortex.cpp/issues/1265
• https://github.com/janhq/cortex.cpp/issues/1268

[0xSage]

Nonurgent Question @vansangpfiev @namchuai :

1. Where do I view the new API?
2. Do we autogenerate the API reference documentation?
3. I'd like to QA for OpenAI compatibility at some point. Should we QA API in this milestone?

[0xSage]

I itemized test cases here: https://github.com/janhq/cortex.cpp/issues/1147

[namchuai]

@0xSage ,

1. Where do I view the new API? When we update the API, we also have to update the json file https://github.com/janhq/cortex.cpp/blob/dev/docs/static/openapi/jan.json and it will be publish in https://cortex.so/api-reference

2. Do we autogenerate the API reference documentation? We using continue.dev to gen the update API. Drogon does not have builtin swagger generation (more info: https://github.com/drogonframework/drogon/pull/923)

3. I'd like to QA for OpenAI compatibility at some point. Should we QA API in this milestone? IMO, yes.

[gabrielle-ong]

Updated QA list in #1535

(will prefer to close this issue and iterate on the QA list with each release)

[dan-homebrew]

Updated QA list in #1535 (will prefer to close this issue and iterate on the QA list with each release)

Yes, please go ahead to close it. We should add a Github Issue template for QA list, and allow us to version control and incrementally improve the Manual QA checklist