search

michaelthwan / searchGPT

Grounded search engine (i.e. with source reference) based on LLM / ChatGPT / OpenAI API. It supports web search, file content search etc.
MIT License
600 stars 62 forks source link

High memory usage of demo server #61

Closed michaelthwan closed 1 year ago

michaelthwan commented 1 year ago

Have no idea why the app use up >512 MB memory. As you can see, I restarted the app at ~12pm and after some calls from users, it grow very fast (and >100% and using swap memory)

image

michaelthwan commented 1 year ago

Maybe due to trafilatura. memory hotfix, rollback from trafilatura to beautifulsoup. web result=3

C:\Users\MW\Anaconda3\envs\searchgpt\lib\site-packages\trafilatura\external.py:56: size=25.4 MiB (+25.4 MiB), count=308894 (+308894), average=86 B C:\Users\MW\Anaconda3\envs\searchgpt\lib\site-packages\justext\core.py:239: size=15.5 MiB (+15.5 MiB), count=154487 (+154487), average=105 B C:\Users\MW\Anaconda3\envs\searchgpt\lib\site-packages\trafilatura\utils.py:296: size=400 KiB (+400 KiB), count=1246 (+1246), average=329 B C:\Users\MW\Anaconda3\envs\searchgpt\lib\site-packages\lxml\html__init__.py:419: size=254 KiB (+254 KiB), count=650 (+650), average=399 B C:\Users\MW\Anaconda3\envs\searchgpt\lib\site-packages\trafilatura\utils.py:284: size=231 KiB (+231 KiB), count=1879 (+1879), average=126 B

michaelthwan commented 1 year ago

64 temp fix

michaelthwan commented 1 year ago

image ok now

64