search

openstates / issues

Having trouble? Looking to contribute? Issues live here!
15 stars 2 forks source link

Running openstates.org/docker/init-db.sh fails #425

Closed AliAbstract closed 2 years ago

AliAbstract commented 3 years ago

Issue Description: When running the init-db.sh script (as described here, the following warnings/errors occur.

First, when attempting to create a Node.js instance, the following deprecation warning is given.

2021-05-17 16:31:00 (22.3 MB/s) - ‘nodesource.sh’ saved [13836/13836]

================================================================================
================================================================================

                              DEPRECATION WARNING

  Node.js 10.x is no longer actively supported!

  You will not receive security or critical stability updates for this version.

  You should migrate to a supported version of Node.js as soon as possible.
  Use the installation script that corresponds to the version of Node.js you
  wish to install. e.g.

   * https://deb.nodesource.com/setup_12.x — Node.js 12 LTS "Erbium"
   * https://deb.nodesource.com/setup_14.x — Node.js 14 LTS "Fermium" (recommended)
   * https://deb.nodesource.com/setup_15.x — Node.js 15 "Fifteen"
   * https://deb.nodesource.com/setup_16.x — Node.js 16 "Gallium"

  Please see https://github.com/nodejs/Release for details about which
  version may be appropriate for you.

  The NodeSource Node.js distributions repository contains
  information both about supported versions of Node.js and supported Linux
  distributions. To learn more about usage, see the repository:
    https://github.com/nodesource/distributions

Second, during + docker-compose run --rm -e PYTHONPATH=docker/ --entrypoint 'poetry run ./manage.py shell -c "import testdata"' django, the following error can be found:

+ docker-compose run --rm -e PYTHONPATH=docker/ --entrypoint 'poetry run ./manage.py shell -c "import testdata"' django
Creating openstatesorg_django_run ... done
Traceback (most recent call last):
  File "./manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/openstates/.cache/pypoetry/virtualenvs/openstates.org-MATOk_fk-py3.7/lib/python3.7/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/openstates/.cache/pypoetry/virtualenvs/openstates.org-MATOk_fk-py3.7/lib/python3.7/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/openstates/.cache/pypoetry/virtualenvs/openstates.org-MATOk_fk-py3.7/lib/python3.7/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/openstates/.cache/pypoetry/virtualenvs/openstates.org-MATOk_fk-py3.7/lib/python3.7/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/openstates/.cache/pypoetry/virtualenvs/openstates.org-MATOk_fk-py3.7/lib/python3.7/site-packages/django/core/management/commands/shell.py", line 87, in handle
    exec(options['command'], globals())
  File "<string>", line 1, in <module>
  File "/code/docker/testdata.py", line 16, in <module>
    fb.sites.add(Site.objects.get())
  File "/home/openstates/.cache/pypoetry/virtualenvs/openstates.org-MATOk_fk-py3.7/lib/python3.7/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/openstates/.cache/pypoetry/virtualenvs/openstates.org-MATOk_fk-py3.7/lib/python3.7/site-packages/django/db/models/query.py", line 442, in get
    num if not limit or num < limit else 'more than %s' % (limit - 1),
django.contrib.sites.models.MultipleObjectsReturned: get() returned more than one Site -- it returned 2!
ERROR: 1

You can see the cause of the error below; there are two sites (example.com and openstates.org). If you remove example.com, 

openstates.org % docker-compose run --rm -e PYTHONPATH=docker/ --entrypoint 'poetry run ./manage.py shell' django
Creating openstatesorg_django_run ... done
Python 3.7.10 (default, May 12 2021, 16:05:48)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.22.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from django.contrib.sites.models import Site

In [2]: Site.objects.all()
Out[2]: <QuerySet [<Site: example.com>, <Site: openstates.org>]>

In [3]: Site.objects.first()
Out[3]: <Site: example.com>

In [4]: Site.objects.first().delete()
Out[4]: (2, {'socialaccount.SocialApp_sites': 1, 'sites.Site': 1})

In [5]: Site.objects.all()
Out[5]: <QuerySet [<Site: openstates.org>]>

I have tried running ./docker/init-db.sh multiple times, and originally I would get an error that there were multiple user accounts with the same information (I assume this is because I ran the command multiple times).

+ docker-compose run --rm -e PYTHONPATH=docker/ --entrypoint 'poetry run ./manage.py shell -c "import testdata"' django
Creating openstatesorg_django_run ... done
Traceback (most recent call last):
  File "/home/openstates/.cache/pypoetry/virtualenvs/openstates.org-MATOk_fk-py3.7/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL:  Key (username)=(local) already exists.

[...]

ERROR: 1

Fixing this error is easy enough (see below), but I continue to receive the sites error.

In [1]: from django.contrib.auth.models import User

In [2]: User.objects.all()
Out[2]: <QuerySet [<User: local>]>

In [3]: User.objects.first().delete()
Out[3]: (3, {'account.EmailAddress': 1, 'profiles.Profile': 1, 'auth.User': 1})

In [4]: User.objects.all()
Out[4]: <QuerySet []>

I've included the full output of a run after I have deleted both the extra example.com site as well as the local user to have as clean of a run as possible. It does end up working, but it might be worth examining why "example.com" is returned from the original run from scratch. It also shows other warnings/info messages that might be worth investigating as well.

openstates.org % ./docker/init-db.sh
+ unset DATABASE_URL
+ docker-compose down
Stopping openstatesorg_db_1 ... done
Removing openstatesorg_db_1 ... done
Removing network openstates-network
+ docker-compose up -d db
Docker Compose is now in the Docker CLI, try `docker compose up`

Creating network "openstates-network" with the default driver
Creating openstatesorg_db_1 ... done
+ sleep 3
+ docker-compose run --rm --entrypoint 'poetry run ./manage.py migrate' django
Creating openstatesorg_django_run ... done
System check identified some issues:

WARNINGS:
account.EmailAddress: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AccountConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
account.EmailConfirmation: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AccountConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
bulk.DataExport: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
bundles.Bundle: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
bundles.BundleBill: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
dashboards.DataQualityReport: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
data.SearchableBill: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
people_admin.DeltaSet: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
people_admin.PersonDelta.data_changes: (fields.W904) django.contrib.postgres.fields.JSONField is deprecated. Support for it (except in historical migrations) will be removed in Django 4.0.
    HINT: Use django.db.models.JSONField instead.
people_admin.PersonDelta: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
people_admin.UnmatchedName: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
profiles.Profile: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the ProfilesConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
profiles.Subscription: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the ProfilesConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
profiles.UsageReport: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the ProfilesConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
reports.ImportObjects: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
reports.RunPlan: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
reports.ScrapeObjects: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
reports.ScrapeReport: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
reports.SessionDataQualityReport: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
socialaccount.SocialAccount: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the SocialAccountConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
socialaccount.SocialApp: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the SocialAccountConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
socialaccount.SocialToken: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
    HINT: Configure the DEFAULT_AUTO_FIELD setting or the SocialAccountConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
Operations to perform:
  Apply all migrations: account, admin, auth, bulk, bundles, contenttypes, dashboards, data, people_admin, profiles, reports, sessions, sites, socialaccount, v1
Running migrations:
  No migrations to apply.
+ DATABASE_URL=postgis://openstates:openstates@db/openstatesorg
+ docker-compose run --rm --entrypoint 'poetry run os-initdb' django
Creating openstatesorg_django_run ... done
loading AK
loading AL
loading AR
loading AZ
loading CA
loading CO
loading CT
loading DC
loading DE
loading FL
loading GA
loading HI
loading IA
loading ID
loading IL
loading IN
loading KS
loading KY
loading LA
loading MA
loading MD
loading ME
loading MI
loading MN
loading MO
loading MS
loading MT
loading NC
loading ND
loading NE
loading NH
loading NJ
loading NM
loading NV
loading NY
loading OH
loading OK
loading OR
loading PA
loading PR
loading RI
loading SC
loading SD
loading TN
loading TX
loading UT
loading VA
loading VT
loading WA
loading WI
loading WV
loading WY
loading US
+ docker-compose run --rm -e PYTHONPATH=docker/ --entrypoint 'poetry run ./manage.py shell -c "import testdata"' django
Creating openstatesorg_django_run ... done
+ docker-compose run --rm --entrypoint 'poetry run ./manage.py shell -c "from django.contrib.sites.models import Site;Site.objects.get_or_create(domain=\"openstates.org\", name=\"openstates.org\")"' django
Creating openstatesorg_django_run ... done
openstates.org %

URL where this issue may be seen (API or OpenStates.org): When attempting to create an instance of OpenStates.org for local development following these steps: https://docs.openstates.org/en/latest/contributing/running-the-site.html

jamesturk commented 3 years ago

Thanks for this report, I'll take a closer look at what might be going on here. Unfortunately it probably will take me a little while since I'll need to find a time when it is safe to start with a fresh DB to see if I can reproduce. If you make any progress in the meantime let us know :)

jamesturk commented 2 years ago

This was fixed up a while back from what I can tell.