Open fbanados opened 1 month ago
What we can receive from Digital Resource Alliance of Canada is under the persistent option, cf. https://docs.alliancecan.ca/wiki/Cloud_RAS_Allocations
The available resources have historically increased gradually, though slowly. Anyhow, we should have up to 50GB ram available to us.
I've reactivated https://itwewina.altlab.dev/, with the latest dictionary and with the updates I worked on so far.
Restarting VMs as part of upgrading seems to considerably reduce the memory consumption. Better limits should be introduced in uwsgi
.
This is really an organization-wide issue, for now keeping it in morphodict.
I've just checked out the RAM consumption of our machines, and it seems that it goes about as follows:
speech-db
4GBitwewina.app
11.9 GBsssttt.altlab.dev
4.9GBbackend+frontend
refactorings 3.6GBsemantic-explorer
1.2GBkorp
portions 0.2GBThis doesn't count
itwewina.dev
. But when we ramp up a parallelitwewina.dev
version with a full dictionary, we run close to the memory limits of our server, which could eventually lead to unexpected service drops. Theitwewina.dev
service stopped processing requests on its own, likely because of memory constraints, and I've decided to take it down at least until a reboot of the production itwewina (at a time less likely to disrupt others), perhaps even until after CILLDI ends. But most importantly, we need to address the memory issue with urgency. Some suggestions (first one is the most straightforward):In parallel, it seems that the generation of sound recordings for paradigms can impose a considerable load on
speech-db
CPU usage (bursts per-request, which are not an issue currently but would scale to a problem if many people are using them). We should consider separating the service and functionality of providing recordings for dictionary purposes from the validation and recording services provided by speech-db, to avoid the former taking down the latter.