Fix the startup perf issue for the openai-gpt agent.
The bottleneck turns out to be the call to GptEncoding.GetEncoding(string encodingName) from the SharpToken package. The first call to that method will trigger the initialization of its internal state, which may involve large table creation and etc.
The startup doesn't immediately require the encoding information, so the fix is to call GptEncoding.GetEncoding on a background thread, so the result will be ready when encoding is actually needed. This reduces the deserialization of setting file from about 2500 ms for cold start and 2000 ms for warm start to about 280 ms for cold start and 20 ms for warm start.
I also changed the serialization/deserialization of the setting file to use source generator in System.Text.Json, to further improve the startup time (~67 ms gain comparing to the reflection-based for cold start, and ~8 ms gain for warm start).
PR Summary
Fix the startup perf issue for the
openai-gptagent.The bottleneck turns out to be the call to
GptEncoding.GetEncoding(string encodingName)from theSharpTokenpackage. The first call to that method will trigger the initialization of its internal state, which may involve large table creation and etc.The startup doesn't immediately require the encoding information, so the fix is to call
GptEncoding.GetEncodingon a background thread, so the result will be ready when encoding is actually needed. This reduces the deserialization of setting file from about2500 msfor cold start and2000 msfor warm start to about280 msfor cold start and20 msfor warm start.I also changed the serialization/deserialization of the setting file to use source generator in
System.Text.Json, to further improve the startup time (~67 msgain comparing to the reflection-based for cold start, and~8 msgain for warm start).