mongoengine test in factory_boy tries to connect to mongo

[ahonnecke]

Description

mongoengine test in factory_boy tries to connect to mongo

I was running into some issues mocking out mongo objects with factory boy. In order to narrow down where the issue is, I grabbed the test_mongoengine file from this repository and tried running it. It did not work.

To Reproduce (My app)

Run the test_mongoengine test in this repository

root@3c4b8d62410b:/opt/apps/repro_factory_boy_mongo_failure# python -m pytest -vv tests/test_mongoengine.py 

Model / Factory code

import os
import unittest

import mongoengine

import factory
from factory.mongoengine import MongoEngineFactory

class Address(mongoengine.EmbeddedDocument):
    street = mongoengine.StringField()

class Person(mongoengine.Document):
    name = mongoengine.StringField()
    address = mongoengine.EmbeddedDocumentField(Address)

class AddressFactory(MongoEngineFactory):
    class Meta:
        model = Address

    street = factory.Sequence(lambda n: 'street%d' % n)

class PersonFactory(MongoEngineFactory):
    class Meta:
        model = Person

    name = factory.Sequence(lambda n: 'name%d' % n)
    address = factory.SubFactory(AddressFactory)

The issue

SKIP_MONGODB = bool(os.environ.get('SKIP_MONGOENGINE') == '1')

@unittest.skipIf(SKIP_MONGODB, "mongodb tests disabled.")
class MongoEngineTestCase(unittest.TestCase):

    db_name = os.environ.get('MONGO_DATABASE', 'factory_boy_test')
    db_host = os.environ.get('MONGO_HOST', 'localhost')
    db_port = int(os.environ.get('MONGO_PORT', '27017'))
    server_timeout_ms = int(os.environ.get('MONGO_TIMEOUT', '300'))

    @classmethod
    def setUpClass(cls):
        from pymongo import read_preferences as mongo_rp
        cls.db = mongoengine.connect(
            db=cls.db_name,
            host=cls.db_host,
            port=cls.db_port,
            # PyMongo>=2.1 requires an explicit read_preference.
            read_preference=mongo_rp.ReadPreference.PRIMARY,
            # PyMongo>=2.1 has a 20s timeout, use 100ms instead
            serverselectiontimeoutms=cls.server_timeout_ms,
        )

    @classmethod
    def tearDownClass(cls):
        cls.db.drop_database(cls.db_name)

    def test_build(self):
        std = PersonFactory.build()
        self.assertEqual('name0', std.name)
        self.assertEqual('street0', std.address.street)
        self.assertIsNone(std.id)

    def test_creation(self):
        std1 = PersonFactory.create()
        self.assertEqual('name1', std1.name)
        self.assertEqual('street1', std1.address.street)
        self.assertIsNotNone(std1.id)

root@3c4b8d62410b:/opt/apps/repro_factory_boy_mongo_failure# python -m pytest -vv tests/test_mongoengine.py 
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.8, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/apps/repro_factory_boy_mongo_failure, configfile: setup.cfg
plugins: celery-4.4.5, Faker-4.1.2
collected 2 items                                                                                                                                                                            

tests/test_mongoengine.py::MongoEngineTestCase::test_build PASSED                                                                                                                      [ 50%]
tests/test_mongoengine.py::MongoEngineTestCase::test_creation FAILED                                                                                                                   [100%]
tests/test_mongoengine.py::MongoEngineTestCase::test_creation ERROR                                                                                                                    [100%]

=========================================================================================== ERRORS ===========================================================================================
___________________________________________________________________ ERROR at teardown of MongoEngineTestCase.test_creation ___________________________________________________________________

cls = <class 'tests.test_mongoengine.MongoEngineTestCase'>

    @classmethod
    def tearDownClass(cls):
>       cls.db.drop_database(cls.db_name)

tests/test_mongoengine.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py:1970: in drop_database
    with self._socket_for_writes(session) as sock_info:
/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py:1293: in _socket_for_writes
    server = self._select_server(writable_server_selector, session)
/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py:1278: in _select_server
    server = topology.select_server(server_selector)
/usr/local/lib/python3.7/site-packages/pymongo/topology.py:243: in select_server
    address))
/usr/local/lib/python3.7/site-packages/pymongo/topology.py:200: in select_servers
    selector, server_timeout, address)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Topology <TopologyDescription id: 5f5bf7de02fcbae02f165baa, topology_type: Single, servers: [<ServerDescription ('loc...st', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>>
selector = <function writable_server_selector at 0x7fe378d373b0>, timeout = 0.3, address = None

    def _select_servers_loop(self, selector, timeout, address):
        """select_servers() guts. Hold the lock when calling this."""
        now = _time()
        end_time = now + timeout
        server_descriptions = self._description.apply_selector(
            selector, address, custom_selector=self._settings.server_selector)

        while not server_descriptions:
            # No suitable servers.
            if timeout == 0 or now > end_time:
                raise ServerSelectionTimeoutError(
                    "%s, Timeout: %ss, Topology Description: %r" %
>                   (self._error_message(selector), timeout, self.description))
E               pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 0.3s, Topology Description: <TopologyDescription id: 5f5bf7de02fcbae02f165baa, topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>

/usr/local/lib/python3.7/site-packages/pymongo/topology.py:217: ServerSelectionTimeoutError
========================================================================================== FAILURES ==========================================================================================
_____________________________________________________________________________ MongoEngineTestCase.test_creation ______________________________________________________________________________

self = <tests.test_mongoengine.MongoEngineTestCase testMethod=test_creation>

    def test_creation(self):
>       std1 = PersonFactory.create()

tests/test_mongoengine.py:73: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.7/site-packages/factory/base.py:527: in create
    return cls._generate(enums.CREATE_STRATEGY, kwargs)
/usr/local/lib/python3.7/site-packages/factory/base.py:464: in _generate
    return step.build()
/usr/local/lib/python3.7/site-packages/factory/builder.py:283: in build
    kwargs=kwargs,
/usr/local/lib/python3.7/site-packages/factory/base.py:316: in instantiate
    return self.factory._create(model, *args, **kwargs)
/usr/local/lib/python3.7/site-packages/factory/mongoengine.py:24: in _create
    instance.save()
/usr/local/lib/python3.7/site-packages/mongoengine/document.py:403: in save
    self.ensure_indexes()
/usr/local/lib/python3.7/site-packages/mongoengine/document.py:880: in ensure_indexes
    collection = cls._get_collection()
/usr/local/lib/python3.7/site-packages/mongoengine/document.py:215: in _get_collection
    if cls._meta.get("auto_create_index", True) and db.client.is_primary:
/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py:1031: in is_primary
    return self._server_property('is_writable')
/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py:856: in _server_property
    writable_server_selector)
/usr/local/lib/python3.7/site-packages/pymongo/topology.py:243: in select_server
    address))
/usr/local/lib/python3.7/site-packages/pymongo/topology.py:200: in select_servers
    selector, server_timeout, address)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Topology <TopologyDescription id: 5f5bf7de02fcbae02f165baa, topology_type: Single, servers: [<ServerDescription ('loc...st', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>>
selector = <function writable_server_selector at 0x7fe378d373b0>, timeout = 0.3, address = None

    def _select_servers_loop(self, selector, timeout, address):
        """select_servers() guts. Hold the lock when calling this."""
        now = _time()
        end_time = now + timeout
        server_descriptions = self._description.apply_selector(
            selector, address, custom_selector=self._settings.server_selector)

        while not server_descriptions:
            # No suitable servers.
            if timeout == 0 or now > end_time:
                raise ServerSelectionTimeoutError(
                    "%s, Timeout: %ss, Topology Description: %r" %
>                   (self._error_message(selector), timeout, self.description))
E               pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 0.3s, Topology Description: <TopologyDescription id: 5f5bf7de02fcbae02f165baa, topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>

/usr/local/lib/python3.7/site-packages/pymongo/topology.py:217: ServerSelectionTimeoutError
================================================================================== short test summary info ===================================================================================
FAILED tests/test_mongoengine.py::MongoEngineTestCase::test_creation - pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 0.3s, Topology...
ERROR tests/test_mongoengine.py::MongoEngineTestCase::test_creation - pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 0.3s, Topology ...
============================================================================ 1 failed, 1 passed, 1 error in 1.39s ============================================================================

Notes

root@3c4b8d62410b:/opt/apps/autodsapi# pip freeze | grep mongo
marshmallow-mongoengine==0.9.1
mongoengine==0.20.0
mongomock==3.16.0
pymongo==3.11.0

[rbarrois]

Hi! Thanks for the report; however, I'm not sure I understand what issue you're having?

When testing the integration of factory_boy with MongoDB-based mappers, we need an actually running MongoDB instance. The connection details for that instance can be configured through the MONGO_HOST, MONGO_PORT and MONGO_DATABASE environment variables.

I usually start a local, temporary instance through docker: docker run --rm --publish 27017:27017 mongo:latest.

Would adding this information to the documentation help solve the problem you've encountered?

[ahonnecke]

Ah, I suspect so! THank you! I'll probably add a mongo instance as a dependency of my pytest service in my docker-compose, would you like me to PR the documentation for that back up?