Upgrade mislukt - Githubissues

goegol commented 7 years ago

Tijdens upgraden per ongeluk in verkeerde venster wat proberen te kopieren (control + c). Onder linux is dat afbreken...

Hij was bezig met database upgrade

Niet alle tabellen zijn goed geupgrade, staat ergens de sql code voor de upgrade? zodat ik handmatig kan rechtrekken?

Opnieuw deploy draaien werkt niet.

De foutmelding die ik krijg is:

Server Error

Exception:

OperationalError: (1054, "Unknown column 'dsmr_frontend_frontendsettings.phase_delivered_l1_color' in 'field list'")
Traceback:

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/views/generic/base.py", line 68, in view
    return self.dispatch(request, *args, **kwargs)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/views/generic/base.py", line 88, in dispatch
    return handler(request, *args, **kwargs)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/views/generic/base.py", line 155, in get
    context = self.get_context_data(**kwargs)

  File "/home/dsmr/dsmr-reader/dsmr_frontend/views/dashboard.py", line 29, in get_context_data
    frontend_settings = FrontendSettings.get_solo()

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/solo/models.py", line 43, in get_solo
    obj, created = cls.objects.get_or_create(pk=1)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/models/query.py", line 473, in get_or_create
    return self.get(**lookup), False

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/models/query.py", line 379, in get
    num = len(clone)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/models/query.py", line 238, in __len__
    self._fetch_all()

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._result_cache = list(self.iterator())

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/models/query.py", line 54, in __iter__
    results = compiler.execute_sql()

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
    cursor.execute(sql, params)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/mysql/base.py", line 110, in execute
    return self.cursor.execute(query, args)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/connections.py", line 42, in defaulterrorhandler
    raise errorvalue

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)

  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/connections.py", line 270, in query
    _mysql.connection.query(self, query)

goegol commented 7 years ago

Tijdens upgrade loopt tie hier tegen aan:

  Applying dsmr_consumption.0005_phase_currently_delivered...Traceback (most recent call last):
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/mysql/base.py", line 110, in execute
    return self.cursor.execute(query, args)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/connections.py", line 42, in defaulterrorhandler
    raise errorvalue
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/connections.py", line 270, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (1060, "Duplicate column name 'phase_currently_delivered_l1'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/migrations/operations/fields.py", line 84, in database_forwards
    field,
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/mysql/schema.py", line 43, in add_field
    super(DatabaseSchemaEditor, self).add_field(model, field)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/base/schema.py", line 409, in add_field
    self.execute(sql, params)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/base/schema.py", line 112, in execute
    cursor.execute(sql, params)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/django/db/backends/mysql/base.py", line 110, in execute
    return self.cursor.execute(query, args)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/connections.py", line 42, in defaulterrorhandler
    raise errorvalue
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages/MySQLdb/connections.py", line 270, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1060, "Duplicate column name 'phase_currently_delivered_l1'")

dennissiemensma commented 7 years ago

Can you try:

sudo su - dsmr
./manage.py showmigrations

It should show a list of migrations and which were completed. Can you paste it?

goegol commented 7 years ago

Thnx for fast responds.

Here the output:

admin
 [X] 0001_initial
 [X] 0002_logentry_remove_auto_add
auth
 [X] 0001_initial
 [X] 0002_alter_permission_name_max_length
 [X] 0003_alter_user_email_max_length
 [X] 0004_alter_user_username_opts
 [X] 0005_alter_user_last_login_null
 [X] 0006_require_contenttypes_0002
 [X] 0007_alter_validators_add_error_messages
 [X] 0008_alter_user_username_max_length
contenttypes
 [X] 0001_initial
 [X] 0002_remove_content_type_name
dsmr_api
 [X] 0001_api_settings
 [X] 0002_generate_random_auth_key
dsmr_backup
 [X] 0001_initial
dsmr_consumption
 [X] 0001_squashed_0004_recalculate_gas_consumption
 [X] 0002_verbose_text
 [X] 0003_electricity_consumption_indexes
 [X] 0004_merge_electricity_tariffs
 [ ] 0005_phase_currently_delivered
dsmr_datalogger
 [X] 0001_squashed_0005_optional_gas_readings
 [X] 0002_meta_names
 [ ] 0003_telegram_checksum
 [ ] 0004_phase_currently_delivered
dsmr_frontend
 [X] 0001_initial
 [X] 0002_recent_history_weeks
 [X] 0003_drop_history_page
 [X] 0004_chart_colors
 [X] 0005_notifications
 [X] 0006_notifications_initial
 [X] 0007_merge_electricity_tariffs
 [X] 0008_merge_electricity_tariffs_notification
 [X] 0009_docs_no_reverse_match
 [ ] 0010_drop_reverse_dashboard_graphs_setting
 [ ] 0011_phase_currently_delivered
dsmr_mindergas
 [X] 0001_mindergas
 [X] 0002_mindergas_notification
dsmr_notification
 [X] 0001_initial
 [X] 0002_notificationsetting_next_notification
 [X] 0003_notification_notification
dsmr_stats
 [X] 0001_squashed_0016_drop_stats_settings
 [X] 0002_regenerate_missing_gas_data
 [X] 0003_hour_statistics_gas_default
 [X] 0004_hour_statistics_gas_default_retroactive
 [X] 0005_statistics_exportverbose_names
 [X] 0006_min_max_temperature_statistics
 [X] 0007_min_max_temperature_statistics_retroactive
 [X] 0008_meta_names
dsmr_weather
 [X] 0001_weather_models
 [X] 0002_meta
 [ ] 0003_next_sync_setting
 [ ] 0004_next_sync_setting_retroactive
sessions
 [X] 0001_initial

dennissiemensma commented 7 years ago

Can you try to run the previous version for now?

sudo su - dsmr
git checkout tags/v1.4.1
./post-deploy.sh (not deploy.sh)

dennissiemensma commented 7 years ago

By the way, another option is to restore the database backup you've created before upgrading (I hope you did?). Then you can easily retry it.

You would only lose the readings between the backup and the restore, which seems reasonable to me.

goegol commented 7 years ago

Yes working again with the post-deploy.

I have backup from tonight (backup automatic to dropbox) Max lose of one day is reasonable for me! Thnx for fast support

dennissiemensma commented 7 years ago

Okay, that's something. You can now either:

delete the database and restore a backup and retry the upgrade.
try to fix it manually.

If you'd like the latter, please execute this (I'm seeing you're using MySQL):

sudo mysql --defaults-file=/etc/mysql/debian.cnf -e 'show columns from dsmr_consumption_electricityconsumption;' -D dsmrreader

What does the output list?

goegol commented 7 years ago

Restore or fix manually is not necessary

After the post-deploy command, the upgrade works fine. I'm now running version 1.5

Output list:

+------------------------------+--------------+------+-----+---------+----------------+
| Field                        | Type         | Null | Key | Default | Extra          |
+------------------------------+--------------+------+-----+---------+----------------+
| id                           | int(11)      | NO   | PRI | NULL    | auto_increment |
| read_at                      | datetime     | NO   | UNI | NULL    |                |
| delivered_1                  | decimal(9,3) | NO   |     | NULL    |                |
| returned_1                   | decimal(9,3) | NO   |     | NULL    |                |
| delivered_2                  | decimal(9,3) | NO   |     | NULL    |                |
| returned_2                   | decimal(9,3) | NO   |     | NULL    |                |
| currently_delivered          | decimal(9,3) | NO   | MUL | NULL    |                |
| currently_returned           | decimal(9,3) | NO   | MUL | NULL    |                |
| phase_currently_delivered_l1 | decimal(9,3) | YES  | MUL | NULL    |                |
| phase_currently_delivered_l2 | decimal(9,3) | YES  | MUL | NULL    |                |
| phase_currently_delivered_l3 | decimal(9,3) | YES  | MUL | NULL    |                |
+------------------------------+--------------+------+-----+---------+----------------+

dennissiemensma commented 7 years ago

Okay, well that is both unexpected and great to hear.

The only reason to use post-deploy instead of deploy is that deploy also updates the code to the latest version (which is not something you'd want when checking out an old version). It should not have fixed anything.

But did you still execute the git checkout tags/v1.4.1 statement before? Because you might get in trouble later when both the code and migrations are not in sync.
Does the ./manage.py showmigrations now show [x] for each and every migration listed?

goegol commented 7 years ago

yes and yes:

admin [X] 0001_initial [X] 0002_logentry_remove_auto_add auth [X] 0001_initial [X] 0002_alter_permission_name_max_length [X] 0003_alter_user_email_max_length [X] 0004_alter_user_username_opts [X] 0005_alter_user_last_login_null [X] 0006_require_contenttypes_0002 [X] 0007_alter_validators_add_error_messages [X] 0008_alter_user_username_max_length contenttypes [X] 0001_initial [X] 0002_remove_content_type_name dsmr_api [X] 0001_api_settings [X] 0002_generate_random_auth_key dsmr_backup [X] 0001_initial dsmr_consumption [X] 0001_squashed_0004_recalculate_gas_consumption [X] 0002_verbose_text [X] 0003_electricity_consumption_indexes [X] 0004_merge_electricity_tariffs [X] 0005_phase_currently_delivered dsmr_datalogger [X] 0001_squashed_0005_optional_gas_readings [X] 0002_meta_names [X] 0003_telegram_checksum [X] 0004_phase_currently_delivered dsmr_frontend [X] 0001_initial [X] 0002_recent_history_weeks [X] 0003_drop_history_page [X] 0004_chart_colors [X] 0005_notifications [X] 0006_notifications_initial [X] 0007_merge_electricity_tariffs [X] 0008_merge_electricity_tariffs_notification [X] 0009_docs_no_reverse_match [X] 0010_drop_reverse_dashboard_graphs_setting [X] 0011_phase_currently_delivered dsmr_mindergas [X] 0001_mindergas [X] 0002_mindergas_notification dsmr_notification [X] 0001_initial [X] 0002_notificationsetting_next_notification [X] 0003_notification_notification dsmr_stats [X] 0001_squashed_0016_drop_stats_settings [X] 0002_regenerate_missing_gas_data [X] 0003_hour_statistics_gas_default [X] 0004_hour_statistics_gas_default_retroactive [X] 0005_statistics_exportverbose_names [X] 0006_min_max_temperature_statistics [X] 0007_min_max_temperature_statistics_retroactive [X] 0008_meta_names dsmr_weather [X] 0001_weather_models [X] 0002_meta [X] 0003_next_sync_setting [X] 0004_next_sync_setting_retroactive sessions [X] 0001_initial

dennissiemensma commented 7 years ago

That's good, then you should be able to execute the 'normal' ./deploy.sh now. Can you try it? It should not error on anything and end with the v1.5 version on the end.

If that goes well, you're upgraded and ready for any future releases as well.

goegol commented 7 years ago

yes, ./deploy.sh works fine, end with version 1.5 and nothing strange in the logs:

 --- You are currently running version:
1.5

 Please make sure you've read EACH release note BEFORE deploying, since some
 changes might not always be backwards compatible or depend on your database!

     >>>   https://github.com/dennissiemensma/dsmr-reader/releases   <<<

 ~~~ Still want to deploy? (type Y + ENTER to continue or press CTRL + C to abort)
y

 --- Pulling remote repository for new commits...

 --- Merging/updating checkout.
Already up-to-date.

 >>> Running post-deployment script. <<<

 --- Checking & synchronizing base requirements for changes.
Requirement already satisfied (use --upgrade to upgrade): crcmod==1.7 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): django==1.10.4 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): django-colorfield==0.1.11 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): django-debug-toolbar==1.6 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): django-solo==1.1.2 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): dropbox==7.1.1 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): gunicorn==19.6 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): pyserial==3.2.1 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil==2.6.0 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): raven==5.32.0 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 10))
Requirement already satisfied (use --upgrade to upgrade): pytz==2016.10 in /home/dsmr/.virtualenvs/dsmrreader/lib/python3.4/site-packages (from -r dsmrreader/provisioning/requirements/base.txt (line 11))
Cleaning up...

 --- Checking & synchronizing database changes/migrations.
System check identified some issues:

WARNINGS:
?: (mysql.W002) MySQL Strict Mode is not set for database connection 'default'
        HINT: MySQL's Strict Mode fixes many data integrity problems in MySQL, such as data truncation upon insertion, by escalating warnings into errors. It is strongly recommended you activate it. See: https://docs.djangoproject.com/en/1.10/ref/databases/#mysql-sql-mode
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, dsmr_api, dsmr_backup, dsmr_consumption, dsmr_datalogger, dsmr_frontend, dsmr_mindergas, dsmr_notification, dsmr_stats, dsmr_weather, sessions
Running migrations:
  No migrations to apply.

 --- Checking & synchronizing static file changes.

0 static files copied to '/var/www/dsmrreader/static', 331 unmodified.

 --- Reloading app code...

 * Reloading Gunicorn (webinterface)...
 [x] Done!

 * Reloading backend process...
 [x] Done!

 * Reloading datalogger process...
 [x] Done!

 --- Clearing cache...
Clearing cache: default
Done

 --- Deployed version:
1.5

 >>> Deployment complete. <<<

dennissiemensma commented 7 years ago

Great! Then this is solved. Thanks for sharing then and have fun with this release. I appreciate you taking the time for both testing and providing this feedback. Have a good one.

dsmrreader / dsmr-reader

Upgrade mislukt #201