[Subtask] Migrate tokenizer configuration logic to Python

[debuggerone]

Subtask Overview

This subtask involves migrating the tokenizer configuration logic from JavaScript to Python, ensuring compatibility with Python tokenizer libraries and validating token counts for different models.

Tasks

• [x] Port tokenizer configuration logic from JavaScript to Python.
• [x] Ensure compatibility with Python tokenizer libraries and validate token counts for different models.
• [x] Test tokenization for various inputs to ensure accuracy and performance.

Acceptance Criteria

• Tokenizer configuration logic is fully migrated to Python.
• Compatibility with Python tokenizer libraries is ensured, and token counts are validated for accuracy.
• Tokenization functions are tested and perform correctly across different models and input scenarios.

[Stevenic]

Isn't this basically a duplicate of this https://github.com/Stevenic/agentm-py/issues/6

[debuggerone]

yup