AI Features added: AI Search, Chat

[kaixxx]

Finally, my experimental version of QualCoder with AI enhanced features (first discussed here: #875) has matured to a point where I feel somewhat comfortable suggesting a merge into the main branch.

I have added the following features:

• An AI based search function to explore the data, look for a particular phenomenon or concept in the whole corpus and interpret the data interactively with the AI. This is part of the text coding workspace. I have demonstrated this feature in my video tutorial on “Horizontal Coding” (under the hood, it has been greatly improved since then).
• An AI Chat: Here, the AI analyzes data, summarizes the results (with exact sources, linked to the original documents), and allows to discuss them. I have implemented three types of chats:
  • A “code chat” that works on already coded data,
  • a “topic chat” that explores a certain phenomenon in the data, even if it has not been coded yet (this is internally using the ‘AI search’ described above), and
  • a “general chat” that allows to discuss any question not directly related to the data (basically a build in ChatGPT).
• Prompt Editing: All the functions described above can be modified by varying the underlying prompts that instruct the AI what to do (in plain English). QualCoder already comes with a predefined set of prompts. These allow, for instance, to modify the AI search to be either very open, interpretative and inclusive or to be more focused, adhering to predefined coding rules outlined in the code’s memo (as it is common practice in content or thematic analysis).
• Methodological control and transparency: The user can also rewrite these prompts or define their own, targeted at their particular methods and research questions. This is a major advantage over commercial QDA software packages that do not offer the same level of transparency and control since they keep the underlying prompts confidential. Not so with QualCoder, which offers an easy to use interface to view, modify, and share such prompts.
• Different AI models: QualCoder uses a combination of a local AI model, specialized on multilingual sematic search in a large corpus of data (an E5 sentence encoder) and a Large Language Model (LLM) like GPT-4, usually hosted online. QualCoder can now also use other LLMs, in particular the open-source models offered by the German academic research agency Helmholtz Society (project “Blablador”, free academic access available). The available models are defined in QualCoders config.yml

Implementation details:

I have tried implementing the AI features in a way that does not interfere with the normal use of QualCoder. By default, the AI features are disabled. The user is asked at the first startup whether they want to enable it. If so, a wizard will guide through the setup process. This wizard can also be started later via the new “AI” menu.

• Startup: Even with the AI turned off, the startup time will be a little longer, so I’ve added a splash screen. With the AI turned on, some of the heavy AI-related libraries (especially “sentence-encoder”) will be loaded in a background thread to prevent the app from becoming unresponsive. A new status message at the bottom of the app indicates when the AI is ready.
• Vector store: For the semantic search to operate, all text based empirical documents in a project must be converted into semantic embeddings using the local AI model (sentence encoder) mentioned above. These embeddings are then stored in a special database called a “vector store”, based on chroma_db. This happens only once (and when a document is edited, renamed or newly added). Since the analysis of a large document can take up to a minute or longer, this is done in a background thread (status bar: “AI: reading data”). The AI features cannot be used until this process is finished.
• Data Structure: I have not altered the existing data structure of QualCoder projects in any way. The additional AI-related data will be stored in a newly created subfolder in the project called “ai_data”. This contains the above-mentioned vector store, a yaml-file with project specific prompts (if there are any) and an SQLite database storing the AI-chats.
• UI integration: I have created a new tab for the AI chat and a new menu for the AI features. All AI-related settings are incorporated in the normal settings dialog.
• Code integration: The code changes are mainly concentrated in __main__.py, codetext.py (for the AI search), as well as several new files, all having an “ai” prefix (e.g., “ai_search_dialog.py” and the corresponding UI files). In addition I had to ensure that whenever a text source is added or modified, the semantic embeddings in the vector store are updated accordingly. This resulted in small changes in many files, e.g. edit_texfile.py, merge_projects.py, etc.

To Do

• Write a wiki page explaining the new features, maybe even record a video
• Add the new libraries to the installation instructions in the readme. I would suggest that we use the requirements.txt (which I have updated) instead of listing all the necessary libraries for each operating system separately. (If needed, one can add platform specific libraries in requirements.txt by using environment markers, e.g.: `pywin32; sys_platform == 'win32') Would that be ok for you?

Status

I have tested this version for quite a while now and hope that all the major problems are eliminated. However, I did a lot of cleanups in the code over the last couple of weeks, which might have introduced some new errors, who knows. Please take a thorough look at my suggestions and tell me if you like the new features and the way I integrated them into the existing QualCoder. I can make changes if you want. If you decide to accept the PR, I’ll promise to have a close look at the bug tracker over the next couple of weeks and try to fix any problems that might pop up.

Best Kai

closes #875

[ccbogel]

This is quite amazing. I guess you also tested it out with colleagues in your area. I will download and experiment with it soon.

[kaixxx]

Ok, I've already found the first typo that I introduced a few days ago ;) It is corrected now.

Also: I wasn't aware of the possibility to edit text in the coding window. So I made the AI functions compatible with edit mode (adapting UI, updating the vector store after editing), see last commit.

[ccbogel]

OK. I will merge and see. I see there will be some code conflicts to resolve.

These changes are not in the colab branch. Is the colab branch still needed?

I see you added opentelemetry - how is this being used?

The pyinstller splashscreen is still experimental and might cause problems (e.g. not go away on some OS). But OK will test this out.

I will also fix the code_text.py difflib to diff-match-patch module after the merge.

[kaixxx]

I have resolved the conflict, no big deal.

Is the colab branch still needed?

No, it was just for developing the collaboration features and can be deleted.

The splash screen is not from pyinstaller. I've tested that also, but the entries in the spec file are commented out because of some issues. Now, the splash screen is implemented with Qt's QtWidgets.QSplashScreen (see gui() in __main__). What I've tested is that if an error ocurres during startup, the splash screen will go away and the error message will be visible.

Opentelemetry is used by chroma_db. But I've turned telemetry off at the very top of ai_vectorstore.py by setting the according environment variables before importing chroma_db.

[ccbogel]

RE: Add the new libraries to the installation instructions in the readme. I would suggest that we use the requirements.txt (which I have updated) instead of listing all the necessary libraries for each operating system separately. (If needed, one can add platform specific libraries in requirements.txt by using environment markers, e.g.: `pywin32; sys_platform == 'win32') Would that be ok for you? Yes OK with me, I might add to the install instructions, to hopefully cover for all operating systems ..? : pip install -r requirements.txt

[ccbogel]

OK merged. Lets see how it goes.

[ccbogel]

@kaixx I am trying to run via PyCharm and also via the commandline on Windows 11. I am seeing into some issues. The latest is the chromadb version issue. I changed this line: chromadb==0.5.0 to chromadb>0.4.0

[ccbogel]

@kaixxx I see a deprecation warning for pydantic, with a suggestion for improvement:

There is also a deprecation warning for SipPyTypeDict() that I am aware of but do not know how to resolve.