Getting `OSError: [Errno 29] Invalid seek` when daemon starts

[gjchung]

Hi,

I'm trying to get this listener up and running in Docker but I'm getting the following error when the daemon starts up:

[2018-10-17 16:43:32 +0000] [9] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/opt/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/opt/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/opt/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/opt/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/opt/local/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/app/joker/poller/app.py", line 54, in <module>
    daemon.start()
  File "/opt/local/lib/python3.7/site-packages/sqs_listener/daemon.py", line 98, in start
    self.daemonize()
  File "/opt/local/lib/python3.7/site-packages/sqs_listener/daemon.py", line 60, in daemonize
    so = open(self.stdout, 'a+')
OSError: [Errno 29] Invalid seek

Here's my code:

from flask import Flask
from sqs_listener.daemon import Daemon
from sqs_listener import SqsListener
import sys
import logging
from os import getenv
sys.path.insert(0, '..')

app = Flask(__name__)
app.config.from_object('settings.Config.%sConfig' % getenv('FLASK_ENV').title())

class VoteMessagesListener(SqsListener):
    # TODO: send message to endpoint for processing.
    def handle_message(self, body, attributes, messages_attributes):
        return True

class VoteMessagesDaemon(Daemon):
    def run(self):
        print("Initializing listener")
        listener = VoteMessagesListener(
            '{url}{queue_name}'.format(
                url = app.config['SQS_QUEUE_URL'],
                queue_name = app.config['SQS_POLLS_VOTES_QUEUE']
            ),
            error_queue='{url}{queue_name}'.format(
                url = app.config['SQS_QUEUE_URL'],
                queue_name = app.config['SQS_POLLS_ERROR_QUEUE']
            ),
            region_name=app.config['AWS_DEFAULT_REGION']
        )
        listener.listen()

logger = logging.getLogger('sqs_listener')
logger.setLevel(logging.INFO)

sh = logging.StreamHandler(sys.stdout)
sh.setLevel(logging.INFO)

formatstr = '[%(asctime)s - %(name)s - %(levelname)s]  %(message)s'
formatter = logging.Formatter(formatstr)

sh.setFormatter(formatter)
logger.addHandler(sh)

# Doing this so I can see the daemon running.
daemon = VoteMessagesDaemon('/var/run/sqs_daemon.pid')
daemon.start()

Any thoughts on what is the cause of the problem?

Thank you for your help! Joycelyn

[e271828-]

@gjchung May be related to: https://github.com/moby/moby/issues/31243

in particular, this line: sh = logging.StreamHandler(sys.stdout)

Let me know if changing that fixes it. You could also try the patch in https://github.com/jegesh/python-sqs-listener/pull/17

[gjchung]

I've removed the logging lines for now and passed in a different logs for stdout and stderr to write to.

daemon = VoteMessagesDaemon('/var/run/sqs_daemon.pid', stdout='/var/log/daemon/access.log', stderr='/var/log/daemon/error.log')

But now it looks like it runs and few times and dies.

[2018-10-17 18:05:51 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2018-10-17 18:05:51 +0000] [1] [INFO] Listening at: http://0.0.0.0:5001 (1)
[2018-10-17 18:05:51 +0000] [1] [INFO] Using worker: sync
[2018-10-17 18:05:51 +0000] [7] [INFO] Booting worker with pid: 7
[2018-10-17 18:05:51 +0000] [7] [INFO] Worker exiting (pid: 7)
[2018-10-17 18:05:51 +0000] [8] [INFO] Worker exiting (pid: 7)
__name__ = joker.poller.app
******************** Trying to start daemon *********************
daemon = <joker.poller.app.VoteMessagesDaemon object at 0x7f946e112b38>
Starting vote listener daemon
__name__ = joker.poller.app
******************** Trying to start daemon *********************
daemon = <joker.poller.app.VoteMessagesDaemon object at 0x7f946e112b38>
Starting vote listener daemon
__name__ = joker.poller.app
******************** Trying to start daemon *********************
daemon = <joker.poller.app.VoteMessagesDaemon object at 0x7f946e112b38>
Starting vote listener daemon
[2018-10-17 18:05:52 +0000] [10] [INFO] Booting worker with pid: 10
[2018-10-17 18:05:53 +0000] [10] [INFO] Worker exiting (pid: 10)
[2018-10-17 18:05:53 +0000] [11] [INFO] Worker exiting (pid: 10)
__name__ = joker.poller.app
******************** Trying to start daemon *********************
daemon = <joker.poller.app.VoteMessagesDaemon object at 0x7f946e102c18>
Starting vote listener daemon
__name__ = joker.poller.app
******************** Trying to start daemon *********************
daemon = <joker.poller.app.VoteMessagesDaemon object at 0x7f946e102c18>
Starting vote listener daemon
__name__ = joker.poller.app
******************** Trying to start daemon *********************
daemon = <joker.poller.app.VoteMessagesDaemon object at 0x7f946e102c18>
Starting vote listener daemon
[2018-10-17 18:05:53 +0000] [1] [INFO] Shutting down: Master
[2018-10-17 18:05:53 +0000] [1] [INFO] Reason: Worker failed to boot.
exited with code 3

[e271828-]

It's dying immediately and restarting, so either the same issue is occurring or you're hitting something else. What do the logs say?

[gjchung]

Nothing is being written to the log files

[e271828-]

Suggest you check your docker user's permissions and see if you have access to write to those locations. Seems likely it's a docker misconfiguration issue, so you may want to ask your local docker expert.