deedy5 / duckduckgo_search

Search for words, documents, images, videos, news, maps and text translation using the DuckDuckGo.com search engine. Downloading files and images to a local hard drive.
MIT License
949 stars 117 forks source link

Prevent memory overuse (VQD_DICT cache) #17

Closed deedy5 closed 1 year ago

deedy5 commented 1 year ago

VQD_DICT - delete first key if len(VQD_DICT)>32768

from random import choice, randint
import objsize

td = dict()
for i in range(1, 70000):
    #td[f"test word {i}"] = f"3-313604123259606289599999999542149999999-{(313604123259606289595796089749796290645 - i):<09}"
    td[f"word {i}"] = f"3-313604123259606289599999999542149999999-{i:<039}"
    #print(td)
    if i > 32000 and i % 1024 == 0:
        print(f"{i=}")
        print(f"{len(td)=}", int(objsize.get_exclusive_deep_size(td))) #/ 1_048_576), "Mb")
i=32768
len(td)=32768 5570648
i=33792
len(td)=33792 5703768
i=34816
len(td)=34816 5836888
i=35840
len(td)=35840 5970008
i=36864
len(td)=36864 6103128
i=37888
len(td)=37888 6236248
i=38912