setup_db fails at models.2691_ES_upgrade migration

[azaroth42]

Describe the bug

Migration fails at line 21 of its code, when it gets a 400 response from ES 6.7.1

  ...
  Applying models.4658_adds_nodevalue_type_widget_config... OK
  Applying models.2691_ES_upgrade...Traceback (most recent call last):
  File "./manage.py", line 27, in <module>
    execute_from_command_line(sys.argv)
    ...
  File "/Users/rsanderson/Development/getty/arches/linked_art/arches/arches/app/models/migrations/2691_ES_upgrade.py", line 21, in forwards_func
    if(se.es.indices.exists(index="%s_strings" % prefix)):
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
   ...
      File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 162, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, u'')

This leaves me rather dead in the water!

[azaroth42]

Full stack trace:

  Applying models.2691_ES_upgrade...Traceback (most recent call last):
  File "./manage.py", line 27, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/Users/rsanderson/Development/getty/arches/linked_art/arches/arches/management/commands/setup_db.py", line 50, in handle
    self.setup_db()
  File "/Users/rsanderson/Development/getty/arches/linked_art/arches/arches/management/commands/setup_db.py", line 185, in setup_db
    management.call_command('migrate')
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/Users/rsanderson/Development/getty/arches/linked_art/arches/arches/app/models/migrations/2691_ES_upgrade.py", line 21, in forwards_func
    if(se.es.indices.exists(index="%s_strings" % prefix)):
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 268, in exists
    return self.transport.perform_request("HEAD", _make_path(index), params=params)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/elasticsearch/transport.py", line 353, in perform_request
    timeout=timeout,
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 236, in perform_request
    self._raise_error(response.status, raw_data)
  File "/Users/rsanderson/Development/getty/arches/linked_art/ENV/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 162, in _raise_error
    status_code, error_message, additional_info
elasticsearch.exceptions.RequestError: RequestError(400, u'')

[mradamcox]

Can you confirm that the same error happens when you just run python manage.py migrate?

[azaroth42]

Yup, exact same error

[apeters]

@azaroth42 what were the settings (and settings values) that you were changing that caused the issue?

[azaroth42]

Thanks @apeters :)

Settings_local was:

MODE = 'DEV'
USE_LIVERELOAD=True

ARCHES_NAMESPACE_FOR_DATA_EXPORT = 'http://localhost:8000/'
ONTOLOGY_PATH = 'ontologies/linked_art'
ONTOLOGY_BASE = 'cidoc.xml'
ONTOLOGY_BASE_NAME = 'Linked Art'
ONTOLOGY_EXT = ['linkedart.xml', 'linkedart_crm_enhancements.xml']

ONTOLOGY_NAMESPACES = {
        "http://purl.org/dc/terms/": "dcterms",
        "http://purl.org/dc/elements/1.1/": "dc",
        "http://schema.org/": "schema",
        "http://www.w3.org/2004/02/skos/core#": "skos",
        "http://www.w3.org/2000/01/rdf-schema#": "rdfs",
        "http://xmlns.com/foaf/0.1/": "foaf",
        "http://www.w3.org/2001/XMLSchema#": "xsd",
        "https://linked.art/ns/terms/": 'la',
    'http://www.w3.org/1999/02/22-rdf-syntax-ns#': 'rdf',
    'http://www.cidoc-crm.org/cidoc-crm/': '',
    'http://www.ics.forth.gr/isl/CRMgeo/': 'geo',
    'http://www.ics.forth.gr/isl/CRMsci/': 'sci'
}

DATABASES = {
    'default': {
        'ENGINE': 'django.contrib.gis.db.backends.postgis', # Add 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'arches',          # Or path to database file if using sqlite3.
        'USER': 'postgres',                  # Not used with sqlite3.
        'PASSWORD': 'postgis',               # Not used with sqlite3.
        'HOST': 'localhost',                 # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '5432',                      # Set to empty string for default. Not used with sqlite3.
        'POSTGIS_TEMPLATE': 'template_postgis_20',
    }
}

ELASTICSEARCH_HTTP_PORT = 9200
ELASTICSEARCH_HOSTS = [
    {'host': 'localhost', 'port': ELASTICSEARCH_HTTP_PORT}
]
ELASTICSEARCH_CONNECTION_OPTIONS = {'timeout': 30}
# a prefix to append to all elasticsearch indexes, note: must be lower case
ELASTICSEARCH_PREFIX = ''

SESSION_COOKIE_NAME = "Arches_Linked_Art_Session"

I moved the ontologies, reset the port for elastic back to default, used the default db name and it ran through okay ... so it seems to be something in here that's the cause.

[apeters]

I think maybe this is the culprit ELASTICSEARCH_PREFIX = '' it should be set to a non-blank value

[azaroth42]

Ahha :) Thanks @apeters!