bludesign / FaxServer

Send and Receive Faxes Using The Twilio Programmable Fax API.
MIT License
108 stars 48 forks source link

Performance issues with 3000+ files #14

Open ZetaTwo opened 5 years ago

ZetaTwo commented 5 years ago

Faxes started to fail due to timeouts. I have about 3000 sent faxes at a total of ~700mb

> db.faxFile.stats()
{
    "ns" : "vapor.faxFile-backup1",
    "size" : 745257940,
    "count" : 3050,
    "avgObjSize" : 244346,
    "storageSize" : 728104960,

Removing all old files by running

> db.faxFile.renameCollection("faxFile_backup1")

Got rid of the issue. In my opinion, 3000 files totalling 700mb isn't that much data and it feels like there is some kind of inefficiency in the code. Unfortunately, I haven't had the time to look closer to try to see what the problem could be or how it could be improved but I post this here in the meanwhile as a start for further investigation.

ZetaTwo commented 5 years ago

What's missing is an index on the faxFile collection, for example, in the Mongo console, running: db.faxFile.createIndex({"faxObjectId":1 }, {"unique": true}) completely solves the performance issue.

I don't know much about MongoKitten but reading their docs would suggest that the following code:

try FaxFile.collection.createIndex(named:"faxObjectId", withParameters: .unique)

somewhere would create the relevant index. I know too little about the codebase however to know where to put this.