Implement mongodb connection pooling

[syl10100]

Why

• Currently, it seems that the perl connections to mongodb are not shared.
• We have one connection for each client i.e if someone opens https://fr.openfoodfacts.org/ and another person opens https://world.openfoodfacts.org/ingredients in other computer then two connections to mongodb will be opened (to be confirmed).
• Connections are resource consuming, Mongodb is spending a lot of time to open and close connections.

Stats

These are the statistic about connections :

• Period : between: 2020 Apr 26 22:30:08 and 2020 Apr 29 12:28:24
• Number of connections opened: 226286
• Overall average connections opened by minute : 60
• Overall average connection duration: 157s

What

One best practice would be to implement connection pooling.

Part of

• 8764

[syl10100]

https://docs.mongodb.com/manual/faq/diagnostics/#faq-memory

[stephanegigandet]

I think we have one connection per Apache/mod_perl process, but each of them can serve multiple requests.

"Number of connections opened: 226286"

-> Do you have stats on the number of MongoDB queries during the same period? Is it 1 connection = 1 query?

[syl10100]

We cannot get this information from mongodb log because all queries are not logged (only the one that last more than 100ms). However maybe we can check the perl log and see how often new connection to mongo is created ?

[syl10100]

The production perl log in verbose mode shows that the number of mongodb connections created is not excessive : in 5 minutes, we got about 50 connections opened and 1700 connections reused.

That does not match the information we have in mongod.log where we can see about one connexion opened/closed every second.

[raphael0202]

Connection pooling is however a good practice that usually increases latency significantly (at least for SQL DB, I've not much experience with MongoDB)

[github-actions[bot]]

This issue has been open 90 days with no activity. Can you give it a little love by linking it to a parent issue, adding relevant labels and projets, creating a mockup if applicable, adding code pointers from https://github.com/openfoodfacts/openfoodfacts-server/blob/main/.github/labeler.yml, giving it a priority, editing the original issue to have a more comprehensive description… Thank you very much for your contribution to 🍊 Open Food Facts