search

gorse-io / gorse

Gorse open source recommender system engine
https://gorse.io
Apache License 2.0
8.63k stars 791 forks source link

How to optimize server ? #892

Open ducvu91 opened 2 days ago

ducvu91 commented 2 days ago

I have 2 point relate to server.

  1. task run very slow, especially Find neighbors of items. my server have 8cpu 4 core, 16g RAM.
  2. how to optimize cache of redis ? i just run testing with about 50k items, but 16G ram redis it not enought. while in reality i have over 5 milion items. and over 1000 category.
  3. Simply calculate, 50k items => 16G RAM, 500k item => 160G RAM, 5000k => 1600G RAM ? OMG, terrible. i was change database type for cache, but it very very slow.

Thanks. Would be very grateful if there is any solution to help me improve

unknownYuan commented 2 days ago

ram的使用不会线性增长

Issues-translate-bot commented 2 days ago

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

RAM usage does not increase linearly

zhenghaoz commented 2 days ago

Tell me the version name please. 0.4 or 0.5

ducvu91 commented 2 days ago

i think version is 0.5. i just pull latest again. Now i reduce data categories each item but the task Find neighbors of items still slow. it run over 30 minute but just not yet 50%. My data 42k item 0 user, 0 feedback, per item have multi label and multi categories. label i think 10 from 20 label per item, categories is from 2 to 5 per item

ducvu91 commented 1 day ago

hello, any solution ? Now my data only have 14k item, only 2 categories. it used 7.5G Ram. idk i wrong config or something wrong else but this used too much resource ? My production data : over 5 milion item, each iteam have 5-15 categories (i used categories for multi purpose, not only category), over 30-50 label. over 1 milion users, 24k active users per day, so I think the feedback will be huge. @zhenghaoz should we have an optimize for it or new solution for it. If it still same, it can't be run on prodution, for research and educational purposes only. I look forward to your solution and reply. Thanks againg.