Open slavakurilyak opened 1 year ago
Get the same problem trying to process the microsoft/semantic-kernel repo. Managed to get things working by catching the error, but it's a hack as I don't understand what's throwing it.
src/cli/commands/index/processRepository.ts
let summaryLength: number;
try {
summaryLength = encoding.encode(summaryPrompt).length;
} catch (error) {
console.error(
`Error during encoding of summary prompt: ${(error as Error).message}`,
);
// set summaryLength to a default value
summaryLength = 0;
}
let questionLength: number;
try {
questionLength = encoding.encode(questionsPrompt).length;
} catch (error) {
console.error(
`Error during encoding of question prompt: ${(error as Error).message}`,
);
// set questionLength to a default value
questionLength = 0;
}
For langchain
, I resolved the issue by deleting docs/modules/agents/toolkits/examples/openai_openapi.yml
.
For semantic-kernel
, I resolved the issue by deleting dotnet/src/SemanticKernel/Connectors/OpenAI/Tokenizers/Settings/encoder.json
.
This issue is related to <|endoftext|>
which is used when interacting with OpenAI. Since lanchain
and semantic-kernel
contain this special character in their repo, the doc index
command fails.
Here's a possible fix: https://github.com/hwchase17/langchain/issues/923
@samheutmaker can you patch this?
Sorry, have been swamped. I'll take a look at this when I get a second.
When I run
doc index
on the langchain repository, I receive the following error:I believe this is an issue with autodoc, rather than the langchain repository, as I have followed the instructions in the README file and run
doc init
in the langchain repository before runningdoc index
.Here is some information about my environment:
Please let me know if there is any additional information I can provide or steps I can take to resolve this issue.