[Subtask] Replace tokenCounter with tiktoken

[debuggerone]

Subtask Overview

This subtask involves replacing the JavaScript tokenCounter function with the tiktoken library in Python for managing tokenization and token counting during the AgentM Python migration.

Tasks

• [x] Install the tiktoken library for Python.
• [x] Implement a Python function to replace the JavaScript tokenCounter using tiktoken.
• [x] Ensure the new function handles token counting for various model types correctly.

Acceptance Criteria

• The tiktoken library is successfully installed and integrated.
• The Python function effectively replaces the JavaScript tokenCounter with correct token handling.
• Token counting is validated for accuracy across different model types.

[Stevenic]

As an FYI... I don't have any components that use tokenCounter() yet other then textSplitter() but there's nothing that uses textSplitter(). There will likely be Micro Agents coming that do but not yet so I'd just say this is a lower priority support wise.

[debuggerone]

I just wanted to have a functionality that prevents running into rate limits - which is pretty painful to debug if you are new to the model usage - and I am pretty sure the limits will hit hard where I am going to use it.

[debuggerone]

After a couple of hours of sleep I think you are right.. let's not over engineer this. Will throw it out now and just keep it simple and then release a system that utilizes it in an abstract layer - with a couple of systems or a plugin system..

After all it's micro agents with tiny prompts

[debuggerone]

removed the database functionality