I've been observing that some CI/CD jobs are not finishing and failing on the MongoDB init script with the following error:
Configuring replica set
Wrote 172.17.0.4:27017,172.17.0.5:27018,172.17.0.6:27019 to /tmp/host_seeds.txt
Waiting for node mongo1 to become primary
Waiting for the first node to be PRIMARY, current: None
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
Waiting for the first node to be PRIMARY, current: SECONDARY
....
The above log keeps going up to the configured timeout of the pipeline.
Running the CI/CD job manually to troubleshoot, I was able to identify two things:
We were supposing to be configuring the "first node" as the PRIMARY (as far as I could understand from the gitlab_rs_init.py script), however we are using the priority i * 10 == 10 to do that, while mongodb replicaset config suggest the usage of higher values to increase the likelihood of a node to become primary (https://www.mongodb.com/docs/manual/reference/replica-configuration/)
A number that indicates the relative likelihood of a replica set member to become the primary.
To increase the likelihood that a member becomes the primary, specify a higher priority value for that member.
To decrease the likelihood that a member becomes the primary, specify a lower priority value for that member.
Sometimes the primary node won't be the first member of the replSetGetStatus command (probably due to the issue 1 above?):
Hi,
I've been observing that some CI/CD jobs are not finishing and failing on the MongoDB init script with the following error:
The above log keeps going up to the configured timeout of the pipeline.
Running the CI/CD job manually to troubleshoot, I was able to identify two things:
i * 10 == 10
to do that, while mongodb replicaset config suggest the usage of higher values to increase the likelihood of a node to become primary (https://www.mongodb.com/docs/manual/reference/replica-configuration/)replSetGetStatus
command (probably due to the issue 1 above?):