z3z1ma / tap-mongodb

Tap for mongodb emphasizing simplicity and leveraging Meltano SDK
MIT License
10 stars 4 forks source link

[FR] Use allowDiskUse per default #6

Open ZuSe opened 1 year ago

ZuSe commented 1 year ago

It seems that the tap is not providing the allowDiskUse option for queries per default. This can cause error on large collections on every mongo db version < 6 e.g:

pymongo.errors.OperationFailure: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting., full error: {'ok': 0.0, 'errmsg': 'Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.', 'code': 16820, 'codeName': 'Location16820'}

https://www.mongodb.com/docs/manual/reference/method/cursor.allowDiskUse/

I would recommend to add it to every query per default