Error when uploading competition bundle

[thomasyu888]

The error message is: Sorry, failed to upload file.

Unsure about what logs to gather, but:

django            |   from django.utils import simplejson
django            | 
django            | 2019-12-13 17:46:31,142 ERROR Internal Server Error: /accounts/chahub/login/
django            | Traceback (most recent call last):
django            |   File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 112, in get_response
django            |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
django            |   File "/usr/local/lib/python2.7/site-packages/allauth/socialaccount/providers/oauth2/views.py", line 41, in view
django            |     return self.dispatch(request, *args, **kwargs)
django            |   File "/usr/local/lib/python2.7/site-packages/allauth/socialaccount/providers/oauth2/views.py", line 59, in dispatch
django            |     app = self.adapter.get_provider().get_app(self.request)
django            |   File "/usr/local/lib/python2.7/site-packages/allauth/socialaccount/providers/base.py", line 15, in get_app
django            |     return SocialApp.objects.get_current(self.id)
django            |   File "/usr/local/lib/python2.7/site-packages/allauth/socialaccount/models.py", line 23, in get_current
django            |     provider=provider)
django            |   File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 151, in get
django            |     return self.get_queryset().get(*args, **kwargs)
django            |   File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 310, in get
django            |     self.model._meta.object_name)
django            | DoesNotExist: SocialApp matching query does not exist.
django            | /usr/local/lib/python2.7/site-packages/django_extensions/db/fields/__init__.py:425: DeprecationWarning: Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.
django            |   warnings.warn("Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.", DeprecationWarning)

I followed the instructions to create the 2 s3 buckets.

[thomasyu888]

@Didayolo , I follow the example s3 private bucket, but after updating the bucket policy, the root user is denied access to the bucket... The instruction on this page: https://github.com/codalab/codalab-competitions/wiki/Storage#aws-s3 say:

AWS_S3_HOST=s3-us-west-1.amazonaws.com

Should it be:

AWS_S3_HOST=s3.us-west-1.amazonaws.com

Even after making this switch, I looked at the network and it seems to give me a 403 - forbidden, which means I must have not set up my user correctly. Are all the docs up to date?

[thomasyu888]

The 403 occurs upon POST.

[Didayolo]

Hi,

The AWS_S4_HOST may vary depending on what you chose when creating the bucket on AWS.

The doc should be up to date. It is hard to know where the error comes from. In the logs, there were other error messages?

[thomasyu888]

I don't get logs when the upload fails.

This is a popup that happens after i try to upload the competition bundle. Currently the SSL certificate isn't set, but I just wanted to try it on http first. Not sure if the full logs will help

WARNING: The SSL_CERTIFICATE variable is not set. Defaulting to a blank string.
WARNING: The SSL_CERTIFICATE_KEY variable is not set. Defaulting to a blank string.
Attaching to nginx, django, worker_site, worker_compute, flower, postgres, memcached, rabbit
worker_compute    | /usr/local/lib/python2.7/site-packages/celery/platforms.py:812: RuntimeWarning: You are running the worker with superuser privileges, which is
worker_compute    | absolutely not recommended!
worker_compute    | 
worker_site       | /usr/local/lib/python2.7/site-packages/django_extensions/db/fields/__init__.py:425: DeprecationWarning: Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.
worker_site       |   warnings.warn("Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.", DeprecationWarning)
worker_compute    | Please specify a different user using the -u option.
worker_site       | 
postgres          | LOG:  database system was shut down at 2019-12-18 17:59:03 UTC
worker_compute    | 
nginx             | + echo 'GENERATING NGINX CONF TEMPLATE...'
worker_site       | /usr/local/lib/python2.7/site-packages/celery/platforms.py:812: RuntimeWarning: You are running the worker with superuser privileges, which is
django            | Waiting for database connection...
flower            | [I 191218 17:59:28 command:133] Visit me at http://localhost:5555
postgres          | LOG:  MultiXact member wraparound protections are now enabled
worker_compute    | User information: uid=0 euid=0 gid=0 egid=0
nginx             | + export 'DOLLAR=$'
worker_site       | absolutely not recommended!
django            | WEB IS RUNNING
flower            | [I 191218 17:59:28 command:138] Broker: amqp://guest:**@rabbit:5672//
postgres          | LOG:  database system is ready to accept connections
worker_compute    | 
nginx             | + DOLLAR='$'
worker_site       | 
rabbit            | + echo 'GENERATING RABBITMQ CONF TEMPLATE...'
django            | DNS fwd/rev mismatch: postgres != postgres.codalabcompetitions_default
flower            | [I 191218 17:59:28 command:141] Registered tasks: 
postgres          | LOG:  autovacuum launcher started
worker_compute    |   uid=uid, euid=euid, gid=gid, egid=egid,
nginx             | + [[ -n '' ]]
worker_site       | Please specify a different user using the -u option.
rabbit            | + export 'DOLLAR=$'
django            | postgres [172.28.0.3] 5432 (postgresql) open
flower            |     ['celery.backend_cleanup',
postgres          | LOG:  incomplete startup packet
worker_compute    | [2019-12-18 17:59:26,244: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbit:5672//: [Errno 111] Connection refused.
nginx             | + echo 'No SSL certificate env var (SSL_CERTIFICATE) found'
worker_site       | 
rabbit            | + DOLLAR='$'
django            | npm WARN npm npm does not support Node.js v10.15.2
flower            |      'celery.chain',
worker_compute    | Trying again in 2.00 seconds...
nginx             | + envsubst
worker_site       | User information: uid=0 euid=0 gid=0 egid=0
rabbit            | + [[ -n '' ]]
django            | npm WARN npm You should probably upgrade to a newer version of node as we
flower            |      'celery.chord',
worker_compute    | 
nginx             | GENERATING NGINX CONF TEMPLATE...
worker_site       | 
rabbit            | + echo 'No SSL certificate env var (SSL_CERTIFICATE) found'
django            | npm WARN npm can't make any promises that npm will work with this version.
flower            |      'celery.chord_unlock',
worker_compute    | [2019-12-18 17:59:28,252: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@rabbit:5672//: [Errno 111] Connection refused.
nginx             | No SSL certificate env var (SSL_CERTIFICATE) found
worker_site       |   uid=uid, euid=euid, gid=gid, egid=egid,
rabbit            | GENERATING RABBITMQ CONF TEMPLATE...
django            | npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
flower            |      'celery.chunks',
worker_compute    | Trying again in 4.00 seconds...
nginx             | + echo nginx/default.conf:
worker_site       | 
rabbit            | No SSL certificate env var (SSL_CERTIFICATE) found
django            | npm WARN npm You can find the latest version at https://nodejs.org/
flower            |      'celery.group',
flower            |      'celery.map',
nginx             | + echo
worker_compute    | 
rabbit            | + envsubst
django            | npm ERR! As of npm@5, the npm cache self-heals from corruption issues and data extracted from the cache is guaranteed to be valid. If you want to make sure everything is consistent, use 'npm cache verify' instead.
flower            |      'celery.starmap']
worker_site       | [2019-12-18 17:59:40,429: INFO/MainProcess] Connected to amqp://guest:**@rabbit:5672//
nginx             | + cat /etc/nginx/conf.d/default.conf
worker_compute    | [2019-12-18 17:59:32,264: INFO/MainProcess] Connected to amqp://guest:**@rabbit:5672//
rabbit            | /etc/rabbitmq/rabbitmq.config:
django            | npm ERR! 
flower            | [E 191218 17:59:28 events:123] Failed to capture events: '[Errno 111] Connection refused', trying again in 2 seconds.
worker_site       | [2019-12-18 17:59:40,438: INFO/MainProcess] mingle: searching for neighbors
nginx             | nginx/default.conf:
worker_compute    | [2019-12-18 17:59:32,272: INFO/MainProcess] mingle: searching for neighbors
rabbit            | + echo /etc/rabbitmq/rabbitmq.config:
django            | npm ERR! If you're sure you want to delete the entire cache, rerun this command with --force.
flower            | [E 191218 17:59:28 gen:661] Multiple exceptions in yield list
worker_site       | [2019-12-18 17:59:41,018: INFO/Beat] beat: Starting...
nginx             | #
worker_compute    | [2019-12-18 17:59:33,303: INFO/MainProcess] mingle: all alone
rabbit            | + echo
flower            |     Traceback (most recent call last):
worker_site       | [2019-12-18 17:59:41,049: INFO/Beat] Scheduler: Sending due task phase_migrations (apps.web.tasks.do_phase_migrations)
django            | 
nginx             | # NOTE!
worker_compute    | [2019-12-18 17:59:33,319: WARNING/MainProcess] celery@compute-worker.e7f8a855e0be ready.
rabbit            | + cat /etc/rabbitmq/rabbitmq.config
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
worker_site       | [2019-12-18 17:59:41,459: INFO/MainProcess] mingle: sync with 1 nodes
django            | npm ERR! A complete log of this run can be found in:
nginx             | # To literally get a dollar sign to print, you have to use $ otherwise it is interpreted as an env var!
rabbit            | [
worker_compute    | [2019-12-18 17:59:33,611: INFO/MainProcess] Events of group {task} enabled by remote.
flower            |         result = self.fn(*self.args, **self.kwargs)
worker_site       | [2019-12-18 17:59:41,459: INFO/MainProcess] mingle: sync complete
django            | npm ERR!     /root/.npm/_logs/2019-12-18T17_59_28_271Z-debug.log
nginx             | #
rabbit            |   {rabbit, [
worker_compute    | [2019-12-18 17:59:40,453: INFO/MainProcess] sync with celery@site-worker
worker_site       | /usr/local/lib/python2.7/site-packages/celery/fixups/django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
django            | npm WARN npm npm does not support Node.js v10.15.2
nginx             | # $ = $
rabbit            |      {tcp_listeners, [{"::", 5672}]},
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 109, in conf
worker_site       |   warnings.warn('Using settings.DEBUG leads to a memory leak, never '
django            | npm WARN npm You should probably upgrade to a newer version of node as we
nginx             | #
rabbit            |      {loopback_users, []}
flower            |         return self._request('dump_conf', with_defaults=with_defaults)
worker_site       | 
nginx             | 
django            | npm WARN npm can't make any promises that npm will work with this version.
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
rabbit            |   ]},
worker_site       | [2019-12-18 17:59:41,471: WARNING/MainProcess] /usr/local/lib/python2.7/site-packages/celery/fixups/django.py:265: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
nginx             | access_log /var/log/nginx/access.log main;
django            | npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
flower            |         timeout=self.timeout, reply=True,
worker_site       |   warnings.warn('Using settings.DEBUG leads to a memory leak, never '
rabbit            |   {rabbitmq_management, [
nginx             | error_log /var/log/nginx/error.log debug;
worker_site       | 
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
django            | npm WARN npm You can find the latest version at https://nodejs.org/
rabbit            |        {listener, [{port, 15672}]}
nginx             | 
worker_site       | [2019-12-18 17:59:41,472: WARNING/MainProcess] celery@site-worker ready.
flower            |         limit, callback, channel=channel,
django            | npm WARN codalab@1.0.0 No description
nginx             | server {
worker_site       | [2019-12-18 17:59:41,473: INFO/MainProcess] Received task: apps.web.tasks.do_phase_migrations[ca0ddba9-74d4-4bbb-9442-346a6cda05b5]
rabbit            |   ]}
django            | npm WARN codalab@1.0.0 No repository field.
nginx             |     listen 80;
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
worker_site       |  
rabbit            | ].
django            | 
nginx             |     #server_name _;
flower            |         chan = channel or self.connection.default_channel
worker_site       |  -------------- celery@site-worker v3.1.26.post2 (Cipater)
rabbit            | + rabbitmq-server
django            | up to date in 0.524s
nginx             |     charset utf-8;
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
worker_site       | ---- **** ----- 
rabbit            | 
django            | npm WARN npm npm does not support Node.js v10.15.2
nginx             |     client_max_body_size 10480m;
flower            |         self.connection
worker_site       | [2019-12-18 17:59:41,483: INFO/Worker-4] Checking 0 competitions for phase migrations.
rabbit            |               RabbitMQ 3.6.16. Copyright (C) 2007-2018 Pivotal Software, Inc.
django            | npm WARN npm You should probably upgrade to a newer version of node as we
nginx             |     client_body_buffer_size 32m;
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
worker_site       | --- * ***  * -- Linux-4.15.0-1056-aws-x86_64-with-debian-10.2
django            | npm WARN npm can't make any promises that npm will work with this version.
rabbit            |   ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
nginx             |     sendfile on;
flower            |         self._connection = self._establish_connection()
worker_site       | -- * - **** --- 
django            | npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
rabbit            |   ##  ##
nginx             | 
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
worker_site       | - ** ---------- [config]
django            | npm WARN npm You can find the latest version at https://nodejs.org/
nginx             |     location ~ /static/(.*) {
rabbit            |   ##########  Logs: /var/log/rabbitmq/output.log
flower            |         conn = self.transport.establish_connection()
worker_site       | - ** ---------- .> app:         codalab:0x7fc2487ba610
django            | /usr/local/bin/lessc -> /usr/local/lib/node_modules/less/bin/lessc
nginx             |         autoindex off;
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
rabbit            |   ######  ##        /var/log/rabbitmq/output_sasl.log
worker_site       | - ** ---------- .> transport:   amqp://guest:**@rabbit:5672//
django            | + less@3.10.3
nginx             |         root /app/codalab/;
rabbit            |   ##########
flower            |         conn = self.Connection(**opts)
django            | added 60 packages from 123 contributors in 2.348s
worker_site       | - ** ---------- .> results:     memcached://memcached:11211/
nginx             |         # Get the file from our collected static or our dev static
rabbit            |               Starting broker...
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
django            | npm WARN npm npm does not support Node.js v10.15.2
worker_site       | - *** --- * --- .> concurrency: 4 (prefork)
nginx             |         try_files /static/$1 /apps/web/static/$1 =404;
rabbit            |  completed with 6 plugins.
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
django            | npm WARN npm You should probably upgrade to a newer version of node as we
worker_site       | -- ******* ---- 
nginx             |     }
nginx             | 
django            | npm WARN npm can't make any promises that npm will work with this version.
worker_site       | --- ***** ----- [queues]
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
nginx             |     location /favicon.ico {
django            | npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
worker_site       |  -------------- .> site-worker      exchange=site-worker(direct) key=site-worker
flower            |         return create_transport(host, connect_timeout, ssl)
nginx             |         alias /app/codalab/static/img/favicon.ico;
django            | npm WARN npm You can find the latest version at https://nodejs.org/
worker_site       |                 .> submission-updates exchange=submission-updates(direct) key=submission-updates
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
nginx             |     }
django            | 
nginx             | 
nginx             |     error_page 503 /error/503.html;
flower            |         return TCPTransport(host, connect_timeout)
worker_site       | 
django            | > codalab@1.0.0 build-css /app/codalab
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
worker_site       | [tasks]
nginx             |     error_page 500 501 502 /error/50x.html;
django            | > lessc apps/web/static/less/imports.less > apps/web/static/css/imports.css
flower            |         raise socket.error(last_err)
worker_site       |   . apps.newsletter.tasks.retry_mailing_list
nginx             |     location ^~ /error/ {
flower            |     OSError: [Errno 111] Connection refused
worker_site       |   . apps.web.tasks.create_competition
django            | 
flower            |     
worker_site       |   . apps.web.tasks.do_chahub_retries
nginx             |         internal;
django            | Copying '/app/codalab/apps/web/static/css/imports.css'
flower            |     During handling of the above exception, another exception occurred:
worker_site       |   . apps.web.tasks.do_phase_migrations
nginx             |         root /app/codalab/static/;
django            | 
flower            |     
worker_site       |   . apps.web.tasks.evaluate_submission
nginx             |     }
django            | 1 static file copied to '/app/codalab/static', 1975 unmodified.
flower            |     Traceback (most recent call last):
worker_site       |   . apps.web.tasks.make_modified_bundle
nginx             | 
django            | /usr/local/lib/python2.7/site-packages/django_extensions/db/fields/__init__.py:425: DeprecationWarning: Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 656, in callback
worker_site       |   . apps.web.tasks.re_run_all_submissions_in_phase
django            |   warnings.warn("Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.", DeprecationWarning)
nginx             |     location / {
flower            |         result_list.append(f.result())
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 398, in result
django            | 
nginx             |         if (-f /app/codalab/maintenance_on) {
flower            |         return self.__get_result()
worker_site       |   . apps.web.tasks.send_chahub_general_stats
django            | Syncing...
nginx             |             return 503;
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
worker_site       |   . apps.web.tasks.send_chahub_updates
django            | Creating tables ...
flower            |         raise self._exception
worker_site       |   . apps.web.tasks.send_mass_email
nginx             |         }
django            | Installing custom SQL ...
worker_site       |   . apps.web.tasks.update_submission
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
nginx             | 
django            | Installing indexes ...
worker_site       | 
flower            |         result = self.fn(*self.args, **self.kwargs)
django            | Installed 0 object(s) from 0 fixture(s)
nginx             |         proxy_pass http://django:8000;
worker_site       | 2019-12-18 17:59:41,483 INFO Checking 0 competitions for phase migrations.
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 103, in active_queues
django            | Migrating...
nginx             |         proxy_set_header Host $host:$server_port;
worker_site       | [2019-12-18 17:59:41,485: INFO/MainProcess] Task apps.web.tasks.do_phase_migrations[ca0ddba9-74d4-4bbb-9442-346a6cda05b5] succeeded in 0.0115493759513s: None
flower            |         return self._request('active_queues')
django            | Running migrations for captcha:
nginx             |         proxy_set_header X-Real-IP $remote_addr;
worker_site       | [2019-12-18 17:59:43,588: INFO/MainProcess] Events of group {task} enabled by remote.
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
django            | - Nothing to migrate.
nginx             |         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
django            |  - Loading initial data for captcha.
nginx             |         proxy_read_timeout 1200;
flower            |         timeout=self.timeout, reply=True,
django            | Installed 0 object(s) from 0 fixture(s)
nginx             |         proxy_intercept_errors on;
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
django            | Running migrations for authenz:
flower            |         limit, callback, channel=channel,
django            | - Nothing to migrate.
nginx             |         proxy_redirect off;
django            |  - Loading initial data for authenz.
nginx             |         proxy_buffering off;
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
django            | Installed 0 object(s) from 0 fixture(s)
nginx             |     }
flower            |         chan = channel or self.connection.default_channel
django            | Running migrations for jobs:
nginx             | }
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
django            | - Nothing to migrate.
nginx             | + nginx -g 'daemon off;'
flower            |         self.connection
django            |  - Loading initial data for jobs.
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
django            | Installed 0 object(s) from 0 fixture(s)
flower            |         self._connection = self._establish_connection()
django            | Running migrations for api:
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
django            | - Nothing to migrate.
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
django            |  - Loading initial data for api.
flower            |         conn = self.Connection(**opts)
django            | Installed 0 object(s) from 0 fixture(s)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
django            | Running migrations for web:
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
django            | - Nothing to migrate.
flower            |         return create_transport(host, connect_timeout, ssl)
django            |  - Loading initial data for web.
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
django            | Installed 0 object(s) from 0 fixture(s)
django            | Running migrations for health:
flower            |         return TCPTransport(host, connect_timeout)
django            | - Nothing to migrate.
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
django            |  - Loading initial data for health.
django            | Installed 0 object(s) from 0 fixture(s)
flower            |         raise socket.error(last_err)
django            | Running migrations for forums:
flower            |     OSError: [Errno 111] Connection refused
django            | - Nothing to migrate.
flower            | [E 191218 17:59:28 concurrent:319] Future <tornado.concurrent.Future object at 0x7fb5b009c358> exception was never retrieved: Traceback (most recent call last):
django            |  - Loading initial data for forums.
django            | Installed 0 object(s) from 0 fixture(s)
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
django            | Running migrations for coopetitions:
flower            |         result = self.fn(*self.args, **self.kwargs)
django            | - Nothing to migrate.
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 109, in conf
django            |  - Loading initial data for coopetitions.
flower            |         return self._request('dump_conf', with_defaults=with_defaults)
django            | Installed 0 object(s) from 0 fixture(s)
django            | Running migrations for queues:
django            | - Nothing to migrate.
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
django            |  - Loading initial data for queues.
django            | Installed 0 object(s) from 0 fixture(s)
django            | Running migrations for teams:
flower            |         timeout=self.timeout, reply=True,
django            | - Nothing to migrate.
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
django            |  - Loading initial data for teams.
django            | Installed 0 object(s) from 0 fixture(s)
flower            |         limit, callback, channel=channel,
django            | Running migrations for customizer:
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
django            | - Nothing to migrate.
django            |  - Loading initial data for customizer.
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
django            | Installed 0 object(s) from 0 fixture(s)
django            | Running migrations for newsletter:
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
django            | - Nothing to migrate.
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
django            |  - Loading initial data for newsletter.
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
django            | Installed 0 object(s) from 0 fixture(s)
django            | Running migrations for socialaccount:
flower            |         conn = self.Connection(**opts)
django            | - Nothing to migrate.
django            |  - Loading initial data for socialaccount.
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
django            | Installed 0 object(s) from 0 fixture(s)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
django            | Running migrations for oauth2_provider:
flower            |         return create_transport(host, connect_timeout, ssl)
django            | - Nothing to migrate.
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
django            |  - Loading initial data for oauth2_provider.
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
django            | Installed 0 object(s) from 0 fixture(s)
flower            |     OSError: [Errno 111] Connection refused
django            | Running migrations for django_extensions:
flower            |     
flower            |     During handling of the above exception, another exception occurred:
django            | - Nothing to migrate.
django            |  - Loading initial data for django_extensions.
flower            |     
flower            |     Traceback (most recent call last):
django            | Installed 0 object(s) from 0 fixture(s)
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 876, in run
django            | Running migrations for guardian:
flower            |         yielded = self.gen.throw(*exc_info)
django            | - Nothing to migrate.
flower            |       File "/usr/local/lib/python3.5/site-packages/flower/api/control.py", line 39, in update_workers
flower            |         results = yield futures
django            |  - Loading initial data for guardian.
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 870, in run
django            | Installed 0 object(s) from 0 fixture(s)
flower            |         value = future.result()
django            | 
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 215, in result
django            | Synced:
django            |  > django.contrib.auth
flower            |         raise_exc_info(self._exc_info)
django            |  > django.contrib.contenttypes
flower            |       File "<string>", line 3, in raise_exc_info
django            |  > django.contrib.sessions
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 656, in callback
django            |  > django.contrib.sites
flower            |         result_list.append(f.result())
django            |  > django.contrib.messages
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 398, in result
django            |  > django.contrib.staticfiles
flower            |         return self.__get_result()
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
django            |  > django.contrib.admin
flower            |         raise self._exception
django            |  > django.contrib.humanize
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
django            |  > analytical
flower            |         result = self.fn(*self.args, **self.kwargs)
django            |  > rest_framework
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 90, in stats
django            |  > mptt
django            |  > django_config_gen
flower            |         return self._request('stats')
django            |  > compressor
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
django            |  > django_js_reverse
django            |  > bootstrapform
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
django            |  > s3direct
flower            |         limit, callback, channel=channel,
django            |  > south
django            |  > apps.analytics
django            |  > apps.common
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
django            |  > apps.chahub
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
django            |  > allauth
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
django            |  > allauth.account
flower            |         self._connection = self._establish_connection()
django            |  > tinymce
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
django            |  > haystack
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
django            |  > pin_passcode
flower            |         conn = self.Connection(**opts)
django            |  > debug_toolbar
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
django            | 
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
django            | Migrated:
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
django            |  - captcha
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
django            |  - apps.authenz
flower            |         return TCPTransport(host, connect_timeout)
django            |  - apps.jobs
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
django            |  - apps.api
django            |  - apps.web
flower            |         raise socket.error(last_err)
django            |  - apps.health
flower            |     OSError: [Errno 111] Connection refused
django            |  - apps.forums
flower            | [E 191218 17:59:28 gen:661] Multiple exceptions in yield list
flower            |     Traceback (most recent call last):
django            |  - apps.coopetitions
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
django            |  - apps.queues
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 109, in conf
django            |  - apps.teams
flower            |         return self._request('dump_conf', with_defaults=with_defaults)
django            |  - apps.customizer
django            |  - apps.newsletter
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
django            |  - allauth.socialaccount
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
django            |  - oauth2_provider
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
django            |  - django_extensions
django            |  - guardian
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
django            | /usr/local/lib/python2.7/site-packages/django_extensions/db/fields/__init__.py:425: DeprecationWarning: Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.
flower            |         self.connection
django            |   warnings.warn("Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.", DeprecationWarning)
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
django            | 
flower            |         self._connection = self._establish_connection()
django            | Initial data has been detected in the database: skipping all inserts. Running data migration...
django            | Data migration complete.
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
django            | /usr/local/lib/python2.7/site-packages/django_extensions/db/fields/__init__.py:425: DeprecationWarning: Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.
flower            |         conn = self.transport.establish_connection()
django            |   warnings.warn("Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.", DeprecationWarning)
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
django            | 
django            | /usr/local/lib/python2.7/site-packages/analytical/templatetags/chartbeat.py:13: DeprecationWarning: django.utils.simplejson is deprecated; use json instead.
flower            |         conn = self.Connection(**opts)
django            |   from django.utils import simplejson
django            | 
django            | /usr/local/lib/python2.7/site-packages/django_extensions/db/fields/__init__.py:425: DeprecationWarning: Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.
django            |   warnings.warn("Django 1.8 features a native UUIDField, this UUIDField will be removed after Django 1.7 becomes unsupported.", DeprecationWarning)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
django            | 
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            |     
flower            |     During handling of the above exception, another exception occurred:
flower            |     
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 656, in callback
flower            |         result_list.append(f.result())
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 398, in result
flower            |         return self.__get_result()
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
flower            |         raise self._exception
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 96, in registered
flower            |         return self._request('dump_tasks', taskinfoitems=taskinfoitems)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            | [E 191218 17:59:28 gen:661] Multiple exceptions in yield list
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 109, in conf
flower            |         return self._request('dump_conf', with_defaults=with_defaults)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            |     
flower            |     During handling of the above exception, another exception occurred:
flower            |     
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 656, in callback
flower            |         result_list.append(f.result())
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 398, in result
flower            |         return self.__get_result()
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
flower            |         raise self._exception
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 84, in scheduled
flower            |         return self._request('dump_schedule', safe=safe)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            | [E 191218 17:59:28 gen:661] Multiple exceptions in yield list
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 109, in conf
flower            |         return self._request('dump_conf', with_defaults=with_defaults)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            |     
flower            |     During handling of the above exception, another exception occurred:
flower            |     
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 656, in callback
flower            |         result_list.append(f.result())
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 398, in result
flower            |         return self.__get_result()
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
flower            |         raise self._exception
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 81, in active
flower            |         return self._request('dump_active', safe=safe)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            | [E 191218 17:59:28 gen:661] Multiple exceptions in yield list
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 109, in conf
flower            |         return self._request('dump_conf', with_defaults=with_defaults)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            |     
flower            |     During handling of the above exception, another exception occurred:
flower            |     
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 656, in callback
flower            |         result_list.append(f.result())
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 398, in result
flower            |         return self.__get_result()
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
flower            |         raise self._exception
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 87, in reserved
flower            |         return self._request('dump_reserved', safe=safe)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            | [E 191218 17:59:28 gen:661] Multiple exceptions in yield list
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 109, in conf
flower            |         return self._request('dump_conf', with_defaults=with_defaults)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            |     
flower            |     During handling of the above exception, another exception occurred:
flower            |     
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 656, in callback
flower            |         result_list.append(f.result())
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 398, in result
flower            |         return self.__get_result()
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
flower            |         raise self._exception
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 93, in revoked
flower            |         return self._request('dump_revoked')
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            | [E 191218 17:59:28 gen:661] Multiple exceptions in yield list
flower            |     Traceback (most recent call last):
flower            |       File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 656, in callback
flower            |         result_list.append(f.result())
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 398, in result
flower            |         return self.__get_result()
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
flower            |         raise self._exception
flower            |       File "/usr/local/lib/python3.5/concurrent/futures/thread.py", line 55, in run
flower            |         result = self.fn(*self.args, **self.kwargs)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 109, in conf
flower            |         return self._request('dump_conf', with_defaults=with_defaults)
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 71, in _request
flower            |         timeout=self.timeout, reply=True,
flower            |       File "/usr/local/lib/python3.5/site-packages/celery/app/control.py", line 316, in broadcast
flower            |         limit, callback, channel=channel,
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/pidbox.py", line 283, in _broadcast
flower            |         chan = channel or self.connection.default_channel
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 766, in default_channel
flower            |         self.connection
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 751, in connection
flower            |         self._connection = self._establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/connection.py", line 706, in _establish_connection
flower            |         conn = self.transport.establish_connection()
flower            |       File "/usr/local/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
flower            |         conn = self.Connection(**opts)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 165, in __init__
flower            |         self.transport = self.Transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/connection.py", line 186, in Transport
flower            |         return create_transport(host, connect_timeout, ssl)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 299, in create_transport
flower            |         return TCPTransport(host, connect_timeout)
flower            |       File "/usr/local/lib/python3.5/site-packages/amqp/transport.py", line 95, in __init__
flower            |         raise socket.error(last_err)
flower            |     OSError: [Errno 111] Connection refused
flower            | [I 191218 17:59:30 mixins:231] Connected to amqp://guest:**@rabbit:5672//

[Didayolo]

Are you trying to setup Codalab on a distant server or locally on your computer?

Actually I have the same issue with a Codalab instance I try to setup. If I manage to solve it I'll let you know, maybe this is related.

[thomasyu888]

I am trying to set up codalab on an ec2, not locally.

I see that the s3 wiki page hasn't been updated since 2017. Its possible that s3 bucket policies and configurations have changed since then which could be causing issues

[Didayolo]

We manage to solve our problem by configuring SSL on both Codalab instance and the storage (enabling https protocol).

We followed the wiki for enabling SSL. You'll need a certificate, you can get one easily and for free at Let's Encrypt. For the storage, we used minio so it may be different but I guess amazon ec2 enables SSL by default.

Regarding the bucket policies, I should check my previous ec2 configuration but I think the wiki is up-to-date, or even if there are some differences they are not crucial.