Bootstrapping fails in docker-compose configuration for submission-ui service

[mhl10]

Describe the bug

Tables in the submission database are not getting populated in docker-compose configuration. Trying to create a new submission without the needed table leads to transaction errors.

To Reproduce Steps to reproduce the behavior:

1. Start with a fresh deployment: docker-compose rm -v
2. Follow docker-compose pull/build/up steps as described in README.md
3. Expect to see output from https://github.com/arXiv/arxiv-submission-ui/blob/2cc8b8b202f6cea35a46336fb771b06a3ab53c20/docker-compose.yml#L300 but do not
4. Once containers are running, check database:

$ mysql --port 3307 -h 127.0.0.1 -u foouser -pfoopass
mysql> use submission
show tables;
Empty set (0.00 sec)

Expected behavior

• Required tables should automatically get populated in the submission table by the submission-ui service in docker-compose.yml.
• Ideally, submission-ui should still be available at port 8000, but the flask command is commented out for this service

Additional context

• Running pipenv run python bootstrap.py manually populates the database successfully. (It might be helpful to have a stdout handler for logging output in this scenario)

[erickpeirson]

A couple of things that will help narrow in on what's going on:

• When you say "once containers are running," are you looking at docker-compose ps and seeing that submission-ui is "healthy"?
• Can you try not only docker-compose rm -v, but also looking for and deleting any orphan containers directly via docker ps and docker rm?
• When you say "Expect to see output from...but do not", do you meant that you see no output at all from submission-ui? Or you don't see output from the bootstrap routine?

[mhl10]

I'll confirm on your first two follow-up questions shortly. For the expected output, I'm referring to the output from the bootstrap routine.

[erickpeirson]

Given that you are connecting to MariaDB and not seeing any tables, I suspect that submission UI hadn't completely started yet

[mhl10]

I can confirm there are no orphan containers. Per your suspicions, submission UI comes up 'unhealthy' after 10 minutes uptime. Other arxiv-prefixed services are showing up 'healthy' (with the exception of arxiv/plaintext which isn't showing health status).

[erickpeirson]

Per your suspicions, submission UI comes up 'unhealthy' after 10 minutes uptime.

Ok, interesting. Can you pull out lines from submission-ui in the logs? May want to set LOGLEVEL=20 or so.

[mhl10]

OK, seems like I may need to be sure I have a working compiler service running!

arxiv-submission-ui       | arxiv.vault.middleware: Certificate verification for compiler is disabled; this should not be disabled in production.
arxiv-submission-ui       | arxiv.vault.middleware: Certificate verification for Kinesis is disabled; this should not be disabled in production.
arxiv-submission-ui       | application 27/Aug/2019:12:19:56 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "initialize Kinesis stream"
arxiv-submission-ui       | application 27/Aug/2019:12:19:56 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "initialize Kinesis stream"
arxiv-submission-ui       | arxiv.vault.middleware: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
arxiv-submission-ui       | arxiv.vault.middleware: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
arxiv-submission-ui       | application 27/Aug/2019:12:20:57 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "stream does not exist; creating"
arxiv-submission-ui       | application 27/Aug/2019:12:20:57 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "stream does not exist; creating"
arxiv-submission-ui       | arxiv.vault.middleware: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
arxiv-submission-ui       | application 27/Aug/2019:12:20:57 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "wait for stream to be available"
arxiv-submission-ui       | application 27/Aug/2019:12:20:57 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "wait for stream to be available"
arxiv-submission-ui       | arxiv.vault.middleware: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
arxiv-submission-ui       | arxiv.vault.middleware: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "initialize Kinesis stream"
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "initialize Kinesis stream"
arxiv-submission-ui       | arxiv.vault.middleware: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "storage service is already available"
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission.services.stream.stream - None - [arxiv:null] - INFO: "storage service is already available"
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission - None - [arxiv:null] - INFO: "await StreamPublisher"
arxiv-submission-ui       | arxiv.vault.middleware: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission - None - [arxiv:null] - INFO: "service StreamPublisher is available!"
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission - None - [arxiv:null] - INFO: "await arxiv.submission.services.classic"
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission - None - [arxiv:null] - INFO: "service arxiv.submission.services.classic is available!"
arxiv-submission-ui       | application 27/Aug/2019:12:21:04 +0000 - arxiv.submission - None - [arxiv:null] - INFO: "All upstream services are available; ready to start"
arxiv-submission-ui       | application 27/Aug/2019:12:21:09 +0000 - submit.factory - None - [arxiv:null] - INFO: "await Filemanager"
arxiv-submission-ui       | application 27/Aug/2019:12:21:09 +0000 - submit.factory - None - [arxiv:null] - INFO: "service Filemanager is available!"
arxiv-submission-ui       | application 27/Aug/2019:12:21:09 +0000 - submit.factory - None - [arxiv:null] - INFO: "await Compiler"
arxiv-submission-ui       | application 27/Aug/2019:12:27:16 +0000 - arxiv.submission.services.compiler.compiler - None - [arxiv:null] - ERROR: "Encountered error calling compiler: Could not connect: HTTPConnectionPool(host='compiler-api', port=8000): Max retries exceeded with url: /status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5548025908>: Failed to establish a new connection: [Errno -2] Name or service not known',))"
arxiv-submission-ui       | application 27/Aug/2019:12:27:16 +0000 - arxiv.submission.services.compiler.compiler - None - [arxiv:null] - ERROR: "Encountered error calling compiler: Could not connect: HTTPConnectionPool(host='compiler-api', port=8000): Max retries exceeded with url: /status (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5548025908>: Failed to establish a new connection: [Errno -2] Name or service not known',))"
arxiv-submission-ui       | application 27/Aug/2019:12:27:16 +0000 - submit.factory - None - [arxiv:null] - INFO: "service Compiler is not available; try again"