Open chancez opened 5 years ago
Here is the full log of noobaa-server:
https://gist.github.com/chancez/66bac74c81d326a4398ac8acf144139b
Tried reproducing the issue: same setup - openshift 4.1 with noobaa installed over it - using the operator - Didn't reproduce. I tried the following:
aws --profile noobaa --endpoint-url https://my-noobaa-s3-elb.example.org:443 --no-verify-ssl s3 rm --recursive s3://metering-test-1/
@chancez If you can help me with another step I missed or any differences in your setup i missed it will be great
Oh, I guess I also used a node pool + a cloud storage at the time. I was using Azure and 3 pods. I forgot about this. Sorry totally forgot about that.
I just got this again using only regular node pools, and no deletions of data happening. The only thing that happened was some nodes were rebooting during this time.
Honestly it mostly seems like a mongodb issue overall.
The logs show mapReduce calls nearby so it might be that one mapReduce calls is causing it. For example I see this issue on OOM in the mapReduce js engine: https://jira.mongodb.org/browse/SERVER-28521
But if this is unrelated to the mapReduce that precedes it, then perhaps it is one of these general segfaults: https://jira.mongodb.org/browse/SERVER-38791 https://jira.mongodb.org/browse/SERVER-38776 (there's more)...
Your right, i do see
2019-10-16T19:49:49.069+0000 I COMMAND [conn6] command nbcore.objectparts command: find { find: "objectparts", filter: { obj: ObjectId('5da7745871c75a0a59a47c4d') }, projection: { _id: 0, chunk: 1 }, returnKey: false, showRecordId: false, $db: "nbcore" } planSummary: COLLSCAN keysExamined:0 docsExamined:679550 cursorExhausted:1 numYields:5309 nreturned:0 reslen:91 locks:{ Global: { acquireCount: { r: 10620 } }, Database: { acquireCount: { r: 5310 } }, Collection: { acquireCount: { r: 5310 } } } protocol:op_msg 299ms
each time before the crash, reliably. 5da7745871c75a0a59a47c4d
is always the ID preceding the crash.
This should be easily fixable by using a newer MongoDB image...
Hey @chancez @kanadaj That's right, we a lagging behind with mongodb upgrades for a long while which is never a good thing, and I am interested to know if you use mongodb in particular and if we should keep maintaining the mongodb backend option (Red Hat changed to use only postgres as noobaa-db). Would be great to hear from you!
@guymguym I'm using the operator myself, and I'm not quite sure if that one support postgres properly as it still defaults to mongo, nor whether the operator supports migrating from mongo to postgres
@kanadaj Yes the operator supports postgress + it migrates from mongo. This is available since release 5.7 (latest is now 5.8).
@dannyzaken @jackyalbo Where did you document the steps required to change an existing installation? I know this requires mostly just changing the noobaa CR spec.dbType to postgres and the operator will take action from there, but I am not sure if there are more spec that needs to change (dbImage/dbResources). Is this all that needs to be done? Is there any upstream wiki/doc to describe the process? Thanks
@guymguym Okay so, migration happens automatically when switching, but I had to manually set spec.dbImage
to centos/postgresql-12-centos7
because even with spec.dbType
set to postgres
on Operator 5.8 (strangely enough it seems to still default to mongo) it was using the image for mongo.
Hi @chancez @kanadaj Do we still need this issue, or can we close it?
Since Postgres is the recommended installation more, probably good to close. That said, I think the operator might still default to mongo so that would be ideal to fix
@kanadaj Can you describe the way you installed.
I am using the cli command install
and the default is postgres.
We previously had an issue where an installation using only CRD defaulted to mongo, but as far as I know, it should be resolved by now. Please let me know.
I installed just with the CRD and the operator from operatorhub.
This issue had no activity for too long - it will now be labeled stale. Update it to prevent it from getting closed.
Environment info
noobaa install
Actual behavior
Expected behavior
Steps to reproduce
aws --profile noobaa --endpoint-url https://my-noobaa-s3-elb.example.org:443 --no-verify-ssl s3 rm --recursive s3://metering-test-1/
More information - Screenshots / Logs / Other output