Closed scalefree closed 6 years ago
Bump. Chris, there's potentially money on the table if you can help get this working for me. Reach out?
Hey @scalefree - sorry this slipped through the cracks, and no worries re: the money. I'm not sure exactly what's going on here - in some cases it appears the output from the Docker commands is not lining up with what the scripts were built for.
First, what version of Docker are you running?
Let's start with debugging the first issue you've listed - running initialize-data-stores.sh
. What is the output of running the following commands from the ws-docker-community
repo directory:
docker-compose down
docker-compose rm
docker-compose up -d
docker ps | grep wsbackend-tasknode
Lot of done's & the grep comes up empty.
How about just docker ps
then?
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9dab4842aa24 redis:latest "docker-entrypoint..." 14 minutes ago Up 14 minutes 6379/tcp wsdockercommunity_redis_1 e3a74bdee8d9 rabbitmq:3-management "docker-entrypoint..." 14 minutes ago Up 14 minutes 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp wsdockercommunity_rabbitmq_1 5dd9fc0a5c2b wsfrontend:latest "nginx -g 'daemon ..." 14 minutes ago Up 14 minutes 443/tcp, 0.0.0.0:8001->80/tcp wsdockercommunity_frontend_1
Forgot the version, here it is: Docker version 17.09.0-ce, build afdb6d4 Running on macOS Sierra
Hey @scalefree - it appears that Docker has changed their automatic naming convention for their containers in docker-compose
.
I've pushed a change to the scripts that should address this (https://github.com/lavalamp-/ws-docker-community/commit/a8500c86183b974031347094e31c10d7b9edd319). Please pull from master
and see if that fixed everything for you.
Pulled latest build. Running initialize-data-stores.sh
I get this:
Recreating wsdockercommunity_tasknode_1 ... done Traceback (most recent call last): File "manage.py", line 23, in
execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 363, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 307, in execute settings.INSTALLED_APPS File "/usr/local/lib/python2.7/site-packages/django/conf/init.py", line 56, in getattr self._setup(name) File "/usr/local/lib/python2.7/site-packages/django/conf/init.py", line 41, in _setup self._wrapped = Settings(settings_module) File "/usr/local/lib/python2.7/site-packages/django/conf/init.py", line 110, in init mod = importlib.import_module(self.SETTINGS_MODULE) File "/usr/local/lib/python2.7/importlib/init.py", line 37, in import_module import(name) File "/ws-backend/wsbackend/settings.py", line 162, in [[CORS_ORIGINS]] NameError: name 'CORS_ORIGINS' is not defined Traceback (most recent call last): File "manage.py", line 23, in execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 363, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 307, in execute settings.INSTALLED_APPS File "/usr/local/lib/python2.7/site-packages/django/conf/init.py", line 56, in getattr self._setup(name) File "/usr/local/lib/python2.7/site-packages/django/conf/init.py", line 41, in _setup self._wrapped = Settings(settings_module) File "/usr/local/lib/python2.7/site-packages/django/conf/init.py", line 110, in init mod = importlib.import_module(self.SETTINGS_MODULE) File "/usr/local/lib/python2.7/importlib/init.py", line 37, in import_module import(name) File "/ws-backend/wsbackend/settings.py", line 162, in [[CORS_ORIGINS]] NameError: name 'CORS_ORIGINS' is not defined
Hey @scalefree - it appears that the contents of your settings.py
file has not been fully updated.
Update the configs/settings.py
file to reflect the following:
CORS_ORIGIN_WHITELIST = (
'localhost:8001',
'127.0.0.1:8001'
)
From there you will want to delete all of the existing dependent images and rebuild them:
docker rmi wsbackend-tasknode
docker rmi wsbackend-api
docker rmi wsbackend-base
./scripts/build-all-images.sh
Once the images are rebuilt try running the above again.
Wiped & rebuilt from scratch. Only error I got was for ALLOWED_HOSTS. Verified postgres, elasticsearch & kibana all running. I noticed latest tasknode.cfg
has es_host = elasticsearch
, mine's on localhost. Aside from that, all green but still no joy.
In the most recent commit to master
I've dockerized Elasticsearch. If you've already set up Elasticsearch locally and want to use it outside of Docker you're still fine but you can also change that tasknode.cfg
value to es_host = elasticsearch
and even more of the deployment is abstracted away into Docker.
In reference to the ALLOWED_HOSTS
issue, adjust your configs/settings.py
file to reflect the following:
ALLOWED_HOSTS = [
'127.0.0.1'
]
Once you've done that, wipe and rebuild again with the following:
docker rmi wsbackend-tasknode
docker rmi wsbackend-api
docker rmi wsbackend-base
./scripts/build-all-images.sh
Hopefully that should finish it!
So here's a new one. Running run-api-tests.sh
:
Traceback (most recent call last): File "manage.py", line 23, in
execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 363, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 337, in execute django.setup() File "/usr/local/lib/python2.7/site-packages/django/init.py", line 27, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate app_config.import_models() File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models self.models_module = import_module(models_module_name) File "/usr/local/lib/python2.7/importlib/init.py", line 37, in import_module import(name) File "/ws-backend/rest/models/init.py", line 12, in from .dns import ( File "/ws-backend/rest/models/dns.py", line 6, in from lib import JsonSerializableMixin File "/ws-backend/lib/init.py", line 102, in from .pubsub import File "/ws-backend/lib/pubsub/init.py", line 4, in from .connector import ( File "/ws-backend/lib/pubsub/connector.py", line 25, in File "/ws-backend/lib/pubsub/init.py", line 4, inclass BasePubSubConnector(object): File "/ws-backend/lib/pubsub/connector.py", line 37, in BasePubSubConnector def publish_message(self, topic=config.pubsub_publish_topic, message=None): File "/ws-backend/lib/config.py", line 1043, in pubsub_publish_topic return self.get_string("PubSub", "pubsub_publish_topic") File "/ws-backend/lib/config.py", line 104, in __get_string return str(self.get_value(section, name)) File "/ws-backend/lib/config.py", line 89, in get_value return self.config.get(section, name) File "/usr/local/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'PubSub' Traceback (most recent call last): File "manage.py", line 23, in execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/ init.py", line 363, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 337, in execute django.setup() File "/usr/local/lib/python2.7/site-packages/django/init.py", line 27, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate app_config.import_models() File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models self.models_module = import_module(models_module_name) File "/usr/local/lib/python2.7/importlib/init.py", line 37, in import_module import(name) File "/ws-backend/rest/models/init.py", line 12, infrom .dns import ( File "/ws-backend/rest/models/dns.py", line 6, in init__.py", line 102, infrom lib import JsonSerializableMixin File "/ws-backend/lib/ from .pubsub import from .connector import ( File "/ws-backend/lib/pubsub/connector.py", line 25, in class BasePubSubConnector(object): File "/ws-backend/lib/pubsub/connector.py", line 37, in BasePubSubConnector def publish_message(self, topic=config.pubsub_publish_topic, message=None): File "/ws-backend/lib/config.py", line 1043, in pubsub_publish_topic return self.get_string("PubSub", "pubsub_publish_topic") File "/ws-backend/lib/config.py", line 104, in __get_string return str(self.get_value(section, name)) File "/ws-backend/lib/config.py", line 89, in __get_value return self.config.get(section, name) File "/usr/local/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'PubSub'
The tasknode.cfg.example
config file has been updated since you last pulled the ws-docker-community
repository. Please take a look at your tasknode.cfg
file and ensure that it has all of the configuration values found in tasknode.cfg.example
, rebuild, and try again.
Wiped, fresh download & install again just in case, got the latest changes into tasknode.cfg
& settings.py
(used the .example files & copied values into them then renamed). Still get the PubSub error, this time from initialize-data-stores.sh
.
Didn't seem to affect the build but I did notice these while running build-all-images.sh
:
W: Size of file /var/lib/apt/lists/security.debian.org_dists_jessie_updates_main_binary-amd64_Packages.gz is not what the server reported 587960 609758 [...] debconf: delaying package configuration, since apt-utils is not installed
That error is specifically stating that the PubSub
section within tasknode.cfg
is not present. To confirm, you can shell into a Docker container of the image:
docker run -it wsbackend-api:latest /bin/bash
Once in the container at the bash prompt you can look at the contents of the tasknode.cfg
file as follows:
cat tasknode/tasknode.cfg
I have a feeling you will find that the PubSub
section is not present. As of right now the configs are (unfortunately) baked into the images instead of loaded in as a volume as I ran into an annoying Docker bug when I was putting this repo together. Ergo, if PubSub
is not present in the config file, you will need to re-bake the image:
docker rmi wsbackend-tasknode
docker rmi wsbackend-api
docker rmi wsbackend-base
./scripts/build-all-images.sh
Give that a try and if it's still complaining about PubSub
then I can take a deeper dive.
Time for a deeper dive. I've done two clean builds (docker rmi
, freshly downloaded source, copy/paste data into config files then rename) & both times got hit with PubSub
. Sorry.
And when you go into the container and cat tasknode/tasknode.cfg
do you see a section entitled [PubSub]
?
Better yet - try running this and paste the output here:
docker run -it wsbackend-api:latest grep -i pubsub /ws-backend/tasknode/tasknode.cfg
Did not see any such section, no.
Came up empty.
As I stated previously then, either the contents of your configs/tasknode.cfg
file does not include the PubSub
section or that file has not been baked into your images. Perhaps you are getting this mixed up with configs/tasknode.cfg.example
? Either that or your images aren't being deleted with the docker rmi
invocations.
Try building the backend base image without using a cache using the following command:
docker build --no-cache -f dockerfiles/backend-base.dockerfile -t wsbackend-base:latest .
Once the image is built run this and see if the PubSub
section is now found in your image:
docker run -it wsbackend-base:latest grep -i pubsub /ws-backend/tasknode/tasknode.cfg
Please confirm that the contents of your configs/tasknode.cfg
file are correct as well.
No-cache build came up empty too. Each build is as clean as I can make it. I'm doing docker rmi -f
on all 3, deleting the whole ws-docker-community
file structure & getting a fresh one from GitHub. I use the latest tasknode.cfg
& settings.py
& copy/paste data into them leaving the rest alone.
And what about this:
grep -i pubsub configs/tasknode.cfg
In tasknode.cfg
I replace AWS, DB & SMTP. In settings.py
I replace Django SECRET_KEY, ALLOWED_HOSTS, DATABASES, CORS_ORIGIN_WHITELIST & EMAIL. All else stays intact.
Checking GitHub I don't see any mention of pubsub
in ws-docker-community/configs/tasknode.cfg.example
of Master branch.
Everything that needs to be replaced in the configuration files is surrounded by square brackets (with the exception of dns_dnsdb_api_key
if you're not using dnsdb and gcp_project_name
if you're not using Google Cloud Platform). If you've replaced all the necessary fields, then the following commands should return empty (minus the exemptions mentioned previously):
grep "\[\[" configs/tasknode.cfg
grep "\[\[" configs/settings.py
Aha - the tasknode.cfg.example
file in this repo hasn't been updated to reflect the state of ws-backend-community
. Use this file instead:
https://github.com/lavalamp-/ws-backend-community/blob/master/tasknode/tasknode.cfg.example
I'd push a change but I've got some other staged commits and don't want to break anything in master right now.
Taking a break, will pick it back up tonight or tomorrow. Thanks for all your help.
Of course - I'm checking to make sure that things are working with Dockerizing all of Postgres as well and will hopefully have it pushed up soon.
Once we get this working I'm gonna be back on it with the enterprise edition, building that from scratch; cause Docker can't hold a couple hundred thousand hosts. Probably after Christmas break.
I have postrgres running on 5432, verified listening on localhost using nmap -sV
. But this is the error I get.
Traceback (most recent call last): File "manage.py", line 23, in
execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 363, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 355, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(*args, *cmd_options) File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute output = self.handle(args, options) File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/makemigrations.py", line 110, in handle loader.check_consistent_history(connection) File "/usr/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 282, in check_consistent_history applied = recorder.applied_migrations() File "/usr/local/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 65, in applied_migrations self.ensure_schema() File "/usr/local/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 52, in ensure_schema if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()): File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor return self._cursor() File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor self.ensure_connection() File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection self.connect() File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in exit six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection self.connect() File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect self.connection = self.get_new_connection(conn_params) File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection connection = Database.connect(conn_params) File "/usr/local/lib/python2.7/site-packages/psycopg2/init.py", line 130, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) django.db.utils.OperationalError: could not connect to server: Connection refused Is the server running on host "localhost" (127.0.0.1) and accepting TCP/IP connections on port 5432? could not connect to server: Cannot assign requested address Is the server running on host "localhost" (::1) and accepting TCP/IP connections on port 5432?
First of all thank you @lavalamp- for the contribution. I would like to test your project in the docker deployment but I having the same that @scalefree.
@scalefree are you getting the 8001 port open? I thing webSight is listening in that port at least in my case.
Starting wsdockercommunity_frontend_1 ... done Traceback (most recent call last): File "manage.py", line 23, in
execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 363, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 337, in execute django.setup() File "/usr/local/lib/python2.7/site-packages/django/init.py", line 27, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate app_config.import_models() File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models self.models_module = import_module(models_module_name) File "/usr/local/lib/python2.7/importlib/init.py", line 37, in import_module import(name) File "/ws-backend/rest/models/init.py", line 12, in from .dns import ( File "/ws-backend/rest/models/dns.py", line 6, in from lib import JsonSerializableMixin File "/ws-backend/lib/init.py", line 102, in from .pubsub import * File "/ws-backend/lib/pubsub/init.py", line 4, in from .connector import ( File "/ws-backend/lib/pubsub/connector.py", line 25, in class BasePubSubConnector(object): File "/ws-backend/lib/pubsub/connector.py", line 37, in BasePubSubConnector def publish_message(self, topic=config.pubsub_publish_topic, message=None): File "/ws-backend/lib/config.py", line 1043, in pubsub_publish_topic return self.get_string("PubSub", "pubsub_publish_topic") File "/ws-backend/lib/config.py", line 104, in __get_string return str(self.get_value(section, name)) File "/ws-backend/lib/config.py", line 89, in __get_value return self.config.get(section, name) File "/usr/local/lib/python2.7/ConfigParser.py", line 607, in get raise NoSectionError(section) ConfigParser.NoSectionError: No section: 'PubSub'
In my case during the first user sign up, I am not getting any response from the POST to the API (port 8000) path /users/. So I am not getting any responde from the backend.
Any suggestion on how I can debug what is happening?
@zeoculls check above for solution on Pubsub. Need to get tasknode.cfg.example
that's not been pushed to Master yet.
Hey all - I'll do what I can to get the most recent pushes out to master tonight. I'm in the process of Dockerizing all of PostgreSQL so that it's even easier to set up without dependencies and have to figure out this one bug...
@scalefree - so you have your tasknode.cfg
and settings.py
files configured to point to localhost for your database?
Yes on both. localhost:5432. Verified listening using nmap.
Chris, what's your expected timeframe on dockerizing postgres? Is your time best spent on that than my bug? If it's just a couple days I can wait.
@scalefree no worries - this one is fairly easy.
So "localhost" in the context of the Docker containers is not the same as "localhost" on your local system. Every container is effectively its own VM with its own IP address. As such when your Docker container is attempting to connect to "localhost" it's attempting to connect to itself. Instead, do the following:
This should address your issue.
Yes! We're moving forward. Postgres is populating. On to the next hurdle, elastic.
Traceback (most recent call last): File "manage.py", line 23, in
execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 363, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/site-packages/django/core/management/init.py", line 355, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(*args, *cmd_options) File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute output = self.handle(args, **options) File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/shell.py", line 95, in handle exec(options['command']) File " ", line 1, in File "/ws-backend/lib/bootstrap.py", line 32, in bootstrap_data_stores bootstrap_elasticsearch() File "/ws-backend/lib/bootstrap.py", line 52, in bootstrap_elasticsearch update_model_mappings() File "/ws-backend/wselasticsearch/bootstrap.py", line 50, in update_model_mappings bootstrap_index_model_mappings(index=config.es_default_index, delete_first=delete_first) File "/ws-backend/wselasticsearch/bootstrap.py", line 31, in bootstrap_index_model_mappings indices = es_helper.get_indices() File "/ws-backend/wselasticsearch/helper.py", line 242, in get_indices return self.connection.indices.get_alias().keys() File "/ws-backend/wselasticsearch/helper.py", line 423, in connection self._connection = self.get_es_connection() File "/ws-backend/wselasticsearch/helper.py", line 386, in get_es_connection return self.get_standard_es_connection() File "/ws-backend/wselasticsearch/helper.py", line 400, in __get_standard_es_connection if config.es_use_http_auth: File "/ws-backend/lib/config.py", line 630, in es_use_http_auth return self.get_bool("Elasticsearch", "es_use_http_auth") File "/ws-backend/lib/config.py", line 58, in get_bool return True if self.__get_value(section, name) == "True" else False File "/ws-backend/lib/config.py", line 89, in get_value return self.config.get(section, name) File "/usr/local/lib/python2.7/ConfigParser.py", line 618, in get raise NoOptionError(option, section) ConfigParser.NoOptionError: No option 'es_use_http_auth' in section: 'Elasticsearch' Username:
Hey @scalefree - this is the same problem as last time. The value can be found in tasknode.cfg.example
here:
https://github.com/lavalamp-/ws-backend-community/blob/master/tasknode/tasknode.cfg.example#L69
Please update and try again.
This has a bunch of Celery fields marked [[REPLACE]]. Is OK?
Also has [[REPLACE]] for all ES fields. Are you sure this is right version to use?
@scalefree - place the es_use_http_auth
value into your config, rebuild, and see if that fixes the error. No reason to replace everything in your config file.
The value for es_use_http_auth
is True
. Tried it, got the password prompt again. Tried setting it to false, also got the prompt.
@scalefree - what password prompt?
Excuse me, username prompt.
ConfigParser.NoOptionError: No option 'es_username' in section: 'Elasticsearch' Username:
That's with es_use_http_auth = True
, same as in the file you pointed me to. Tried setting it to False
just to see, slightly different error but again the prompt.
Whenever you see an error being triggered by a lack of a configuration value, go ahead and add that value to the configuration file. In this case, it doesn't matter what the value of the configuration key es_username
is as it won't be used since es_use_http_auth
is false.
https://github.com/lavalamp-/ws-backend-community/blob/master/tasknode/tasknode.cfg.example#L70
I just pushed to master:
https://github.com/lavalamp-/ws-docker-community/commit/d032709c5416d259060723b0b5363b7f446928db
Please pull from latest and re-do your config file again. This time around the only things you'll need to be populating are aws_key_id
and aws_secret_key
in tasknode.cfg
. The settings.py
file you have now should remain unchanged.
Let me know if that works for you.
So I'm finally at the build & test steps. I believe I have everything except SMTP properly configured & I'm working on getting email creds. Build output looked clean but came up with a few errors in the test scripts & when I try the URLs I get nothing.
From initialize-data-stores.sh I get this:
From run-tasknode-tests.sh I get this:
From run-api-tests.sh:
From run-tasknode-deploy-checker.sh: