Open butterl opened 5 months ago
Hello butter, thank you for your question. When I wrote the introduction, only ChatGPT 3.5 Turno was available. We now have ChatGPT 4, which has more tokens. As you mentioned, using a text splitter could work for this.
I've also developed a tool called pstruc (Link) that can generate a json/yaml/txt file with all the required project information. With this tool, you can select which files to include or ignore, as well as specify the file content to include in the project structure information.
I believe that combining both tools (pstruc and the langchain text splitter) could be a valuable approach for this project.
Please feel free to collaborate on this project if you're interested!
"The original idea was to also review the content of the files and the software, but due to the current limitations of ChatGPT's tokens, it is not feasible to cover the folder/files/content"
from the intro you say , ' current limitations of ChatGPT's tokens' which means the max token not able to cover the big file and code split may broken the review ? maybe langchain text splitter with code type maybe work ?