openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.06k stars 749 forks source link

In-code documentation update #260

Open louisbrulenaudet opened 4 months ago

louisbrulenaudet commented 4 months ago

Dear Developers,

I'm pleased to inform you that I have completed the documentation update the load, model and registry files.

The updated documentation provides clear explanations of function parameters, return types, and expected behavior. Additionally, it adheres to consistent formatting and organization, ensuring ease of understanding for both current and future developers.

Please review the updated documentation at your earliest convenience. If you have any feedback or suggestions for further improvements, please don't hesitate to let me know.

Thank you for your attention to this matter.

Best regards, Louis Brulé Naudet