search

massquantity / LibRecommender

Versatile End-to-End Recommender System
https://librecommender.readthedocs.io/
MIT License
352 stars 62 forks source link

Query about deployment setup #443

Open samruddhag1 opened 7 months ago

samruddhag1 commented 7 months ago

Hi I have been working on recommendation systems project. We found this library to be very useful. I have tried using this library on a subset of data, and most of the things have been smooth. This is as such not an issue with codebase but a general query for experienced users of library. I wanted to know if you have any particular advice in case I go ahead with deployment of training/retraining and serving module. Details: ~10M users ~500K items 4-5 sparse item features No user features We are considering DeepFM, Caser or TwoTower as possible candidates. We also found ALS performance competative too. We will be retraining the model frequently (once in a week or two). I would like to know from community if some additional optimisation can be done in general. If any particular system configuration are must or recommend. What GPU configurations are recommended? If I want to use an AWS instance to host this setup, which setup/AMI I should choose?

Thanks in advance.

massquantity commented 7 months ago

Hi, i think nothing special needs to be taken care of. Just follow the instructions in the library documentation.

gms101 commented 7 months ago

hi @massquantity thanks for replying. Just wanted to clarify the question, is librecommender scalable for the given models with such a scale of data? If yes, what system configuration is kind of a minimum for this scale?

massquantity commented 7 months ago

@gms101 What is your overall data size? All the data will be loaded into memory before training, so TB-size data will be a concern.

samruddhag1 commented 6 months ago

Complete data size is ~5GB.

massquantity commented 6 months ago

You may need instances of at least 16GB memory.