Profiler to profile program

[michaelthwan]

e.g. I think the 5x scrap website is quite slow

[peterhcyuen]

For reference

[karabC]

If the bottleneck is the embedding service for Semantic search, should we try to setup config to use external provider, e.g. says cost effective GPT text-embedding-ada-002 for trial?

We can still have the sentence-transformers/all-mpnet-base-v2 as the default choice, but externalize the loading to other embedding service endpoint seems making more sense to me.

[michaelthwan]

@karabC possibly~

[michaelthwan]

• OpenAIEmbeddings one is a lot faster
• OpenAI use "text-embedding-ada-002" by default, perfect.

[michaelthwan]

quite fast now. Let me close this.