Mongo connector stops pushing data

[Thunkar]

I'm running mongo-connector inside a docker container with elasticsearch 2 and after a random amount of time it displays the following exception:

After this, no new data is indexed in Elasticsearch. Sometimes deleting the oplog.timestamp file solves the problem (it is always empty, though), sometimes not. My docker-compose file is the following:

version: "2"
services:
    elasticsearch:
        image: elasticsearch:latest
        container_name: elasticsearch
        ports:
            - "9200:9200"
        volumes:
            - ./elasticsearch/config:/usr/share/elasticsearch/config
            - ./elasticsearch/data:/usr/share/elasticsearch/data 
    mongo-connector:
        image: mongo-connector:latest
        build:
            context: ./mongo-connector
            dockerfile: mongo-connector.docker
        container_name: mongo-connector
        command: mongo-connector -c /etc/mongo-connector/config.json 
        restart: always
        volumes:
            - ./mongo-connector/config.json:/etc/mongo-connector/config.json
            - ./mongo-connector/logs:/var/log/mongo-connector
        links:
            - elasticsearch
        depends_on:
            - elasticsearch
    logstash:
        image: logstash:latest
        container_name: logstash
        command: logstash -f /etc/logstash/conf.d/logstash.conf
        volumes:
            - ./logstash/config:/etc/logstash/conf.d
            - /var/log/nginx:/var/log/nginx
        ports:
            - "5000:5000"
        links:
            - elasticsearch
        depends_on:
            - elasticsearch
    kibana:
        image: kibana-w-plugins:latest
        build:
            context: ./kibana
            dockerfile: kibana-w-plugins.docker
        container_name: kibana
        ports:
            - "5601:5601"
        volumes:
            - ./kibana/kibana.yml:/opt/kibana/config/kibana.yml
        links:
            - elasticsearch
        depends_on:
            - elasticsearch

And my mongo-connector config is this:

{
    "mainAddress": "***",
    "oplogFile": "/var/log/mongo-connector/oplog.timestamp",
    "noDump": false,
    "batchSize": 100,
    "verbosity": 1,
    "continueOnError": false,

    "logging": {
        "type": "file",
        "filename": "/var/log/mongo-connector/mongo-connector.log"
    },

    "authentication": {
        "adminUsername": "***",
        "password": "***"
    },

    "docManagers": [
        {
            "docManager": "elastic2_doc_manager",
            "targetURL": "http://elasticsearch:9200"
        }
    ]
}

As you can see, the oplog file resides on the host and it is mounted into the container, which in turn can use it and write to it as the folder has 775 permissions. Any help would be appreciated.

[behackett]

That traceback means that the pymongo.MongoClient instance the cursor manager was referencing was reclaimed. I've never actually seen this happen before, but it would only happen when the MongoClient instance is being destroyed.

Were you shutting down mongo-connector?

[Thunkar]

No, it just happens randomly. I'm testing it now leaving the oplog.timestamp file inside the container to check for permission issues...will report back if I gather new info.

[Thunkar]

Just happened again after some time without issues, exact same error and behavior.

[Thunkar]

After using python 2 as base image instead of the latest one (based on python 3), the issue has dissapeared. Root cause still unknown though...