search

learningequality / kolibri

Kolibri Learning Platform: the offline app for universal education
https://learningequality.org/kolibri/
MIT License
772 stars 646 forks source link

Database disk image is malformed #3825

Closed mrpau-richard closed 5 years ago

mrpau-richard commented 6 years ago

Observed behavior

Upgrading kolibri from 0.7.2 to 0.10.0b3 will have this Database disk image is malformed error when running the kolibri start command.

Errors and logs

442 static files copied to '/home/user/.kolibri/static', 3 unmodified.
Operations to perform:
  Apply all migrations: admin, auth, content, contenttypes, device, exams, kolibriauth, lessons, logger, morango, sessions
Running migrations:
  Applying auth.0008_alter_user_username_max_length... OK
  Applying content.0007_auto_20180212_1155... OK
  Applying content.0008_contentnode_coach_content...Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.DatabaseError: database disk image is malformed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/kolibri", line 9, in <module>
    load_entry_point('kolibri==0.10.0b3', 'console_scripts', 'kolibri')()
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 646, in main
    start(port, daemon=daemon)
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 261, in start
    update()
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 239, in update
    _migrate_databases()
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 176, in _migrate_databases
    call_command("migrate", interactive=False, database=database)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/operations/fields.py", line 88, in database_forwards
    field,
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/schema.py", line 238, in add_field
    self._remake_table(model, create_field=field)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/schema.py", line 206, in _remake_table
    self.quote_name(model._meta.db_table),
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/base/schema.py", line 136, in execute
    cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.DatabaseError: database disk image is malformed

Steps to reproduce

Context

indirectlylit commented 6 years ago

cc @benjaoming looks like we have one in the wild! Would you mind checking into this with @mrpau-richard and seeing if your dbrestore command helps?

benjaoming commented 6 years ago

@indirectlylit indeed - and it seems that I might have to go back to 0.7.2 to reproduce malformed databases.

Thanks @mrpau-richard for reproducing this :tada:

I have two favors to ask from you @mrpau-richard :)

  1. The technique for restoring the database as described here did not seem to work on Ubuntu 18.04, but I would appreciate if you can confirm this for your problematic db?

  2. What are your simplest steps to reproduce a malformed database? I mean, not by copying the user data directory that you so kindly uploaded - but can you repeat the minimal set of steps to end up with this malformed db in ~/.kolibri?

benjaoming commented 6 years ago

Refs: #3228

mrpau-richard commented 6 years ago

The technique for restoring the database as described here did not seem to work on Ubuntu 18.04, but I would appreciate if you can confirm this for your problematic db?

@benjaoming after I ran this command sqlite3 /home/<your_username>/.kolibri/db.sqlite3 .dump | sqlite3 fixed.db, The error database malformed still exist.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlite3.DatabaseError: database disk image is malformed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/kolibri", line 9, in <module>
    load_entry_point('kolibri==0.10.0b3', 'console_scripts', 'kolibri')()
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 646, in main
    start(port, daemon=daemon)
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 261, in start
    update()
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 245, in update
    update_channel_metadata()
  File "/usr/lib/python3/dist-packages/kolibri/content/utils/annotation.py", line 43, in update_channel_metadata
    import_channel_from_local_db(channel_id)
  File "/usr/lib/python3/dist-packages/kolibri/content/utils/channel_import.py", line 464, in import_channel_from_local_db
    import_manager.import_channel_data()
  File "/usr/lib/python3/dist-packages/kolibri/content/utils/channel_import.py", line 311, in import_channel_data
    raise e
  File "/usr/lib/python3/dist-packages/kolibri/content/utils/channel_import.py", line 304, in import_channel_data
    unflushed_rows = self.table_import(model, row_mapper, table_mapper, unflushed_rows)
  File "/usr/lib/python3/dist-packages/kolibri/content/utils/channel_import.py", line 221, in table_import
    for record in table_mapper(SourceRecord):
  File "/usr/lib/python3/dist-packages/kolibri/content/utils/channel_import.py", line 187, in base_table_mapper
    return self.source.session.query(SourceRecord).all()
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/orm/query.py", line 2773, in all
    return list(self)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/orm/query.py", line 2925, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/orm/query.py", line 2948, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/lib/python3/dist-packages/kolibri/dist/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed [SQL: 'SELECT content_contentnode.id AS content_contentnode_id, content_contentnode.title AS content_contentnode_title, content_contentnode.content_id AS content_contentnode_content_id, content_contentnode.channel_id AS content_contentnode_channel_id, content_contentnode.description AS content_contentnode_description, content_contentnode.sort_order AS content_contentnode_sort_order, content_contentnode.license_owner AS content_contentnode_license_owner, content_contentnode.author AS content_contentnode_author, content_contentnode.kind AS content_contentnode_kind, content_contentnode.available AS content_contentnode_available, content_contentnode.stemmed_metaphone AS content_contentnode_stemmed_metaphone, content_contentnode.lft AS content_contentnode_lft, content_contentnode.rght AS content_contentnode_rght, content_contentnode.tree_id AS content_contentnode_tree_id, content_contentnode.level AS content_contentnode_level, content_contentnode.lang_id AS content_contentnode_lang_id, content_contentnode.license_description AS content_contentnode_license_description, content_contentnode.license_name AS content_contentnode_license_name, content_contentnode.coach_content AS content_contentnode_coach_content, content_contentnode.parent_id AS content_contentnode_parent_id \nFROM content_contentnode'] (Background on this error at: http://sqlalche.me/e/4xp6)

What are your simplest steps to reproduce a malformed database? I mean, not by copying the user data directory that you so kindly uploaded - but can you repeat the minimal set of steps to end up with this malformed db in ~/.kolibri?

I can't reproduce the issue @benjaoming if I'm not using the 0.7.2 sample data. When I install the 0.7.2 then manually create some data there then upgrade to 0.10.x the malformed database can't reproduce.

benjaoming commented 6 years ago

@mrpau-richard thanks for elaborating - okay so fixing malformed db's in 18.04 is definitely not working.

Where did you get the 0.7.2 sample data from?

mrpau-richard commented 6 years ago

I created the 0.7.2 sample data at the Windows 7 @benjaoming.

benjaoming commented 6 years ago

Also interesting: It seems to be breaking during migrations:

  Applying auth.0008_alter_user_username_max_length... OK
  Applying content.0007_auto_20180212_1155... OK
  Applying content.0008_contentnode_coach_content...Traceback (most recent call last):
benjaoming commented 6 years ago

A note from @mrpau-richard testing on Windows: It seems that the malformed database works in 0.7.2. So while it's interesting to investigate if a malformed DB in 0.10.0 can actually run in 0.7.2.

lyw07 commented 6 years ago

I reproduced this error under the circumstances:

  1. the download speed is 0.5 Mbit/s
  2. import KA channel (not the content)

The error I got is:

27%|███████████████████████████████████████████▋                                                                                                                   | 44540408/162202624 [14:59<38:02, 51545.79it/s]
ERROR    An error occured during channel import: (sqlite3.DatabaseError) database disk image is malformed [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/4xp6)
ERROR:kolibri.content.management.commands.importchannel:An error occured during channel import: (sqlite3.DatabaseError) database disk image is malformed [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/4xp6)
WARNING  Job 96691133b74b4a048d7fb89af6179cbb raised an exception: Traceback (most recent call last):
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlite3.DatabaseError: database disk image is malformed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/iceqube/worker/backends/inmem.py", line 75, in handle_finished_future
    result = future.result()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/iceqube/worker/backends/inmem.py", line 149, in wrap
    raise e
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/iceqube/worker/backends/inmem.py", line 145, in wrap
    return f(*args, **kwargs)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/iceqube/common/classes.py", line 119, in y
    return func(*args, **kwargs)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/tasks/management/commands/base.py", line 99, in handle
    return self.handle_async(*args, **options)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 150, in handle_async
    self.download_channel(options["channel_id"], options["baseurl"])
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 78, in download_channel
    self._transfer(DOWNLOAD_METHOD, channel_id, baseurl)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 145, in _transfer
    raise e
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 114, in _transfer
    import_channel_by_id(channel_id, self.is_cancelled)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 24, in import_channel_by_id
    channel_import.import_channel_from_local_db(channel_id, cancel_check=cancel_check)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/channel_import.py", line 653, in import_channel_from_local_db
    import_manager = initialize_import_manager(channel_id, cancel_check=cancel_check)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/channel_import.py", line 627, in initialize_import_manager
    channel_metadata = read_channel_metadata_from_db_file(get_content_database_file_path(channel_id))
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/channels.py", line 56, in read_channel_metadata_from_db_file
    source = Bridge(sqlite_file_path=channeldbpath)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/sqlalchemybridge.py", line 207, in __init__
    db_matches_schema(self.Base, self.session)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/check_schema_db.py", line 22, in db_matches_schema
    tables = iengine.get_table_names()
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/reflection.py", line 201, in get_table_names
    self.bind, schema, info_cache=self.info_cache)
  File "<string>", line 2, in get_table_names
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/reflection.py", line 54, in cache
    ret = fn(self, con, *args, **kw)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/dialects/sqlite/base.py", line 1160, in get_table_names
    rs = connection.execute(s)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 2075, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 942, in execute
    return self._execute_text(object, multiparams, params)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1104, in _execute_text
    statement, parameters
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/4xp6)

WARNING:iceqube.worker.backends.base:Job 96691133b74b4a048d7fb89af6179cbb raised an exception: Traceback (most recent call last):
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlite3.DatabaseError: database disk image is malformed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/iceqube/worker/backends/inmem.py", line 75, in handle_finished_future
    result = future.result()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/iceqube/worker/backends/inmem.py", line 149, in wrap
    raise e
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/iceqube/worker/backends/inmem.py", line 145, in wrap
    return f(*args, **kwargs)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/iceqube/common/classes.py", line 119, in y
    return func(*args, **kwargs)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/tasks/management/commands/base.py", line 99, in handle
    return self.handle_async(*args, **options)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 150, in handle_async
    self.download_channel(options["channel_id"], options["baseurl"])
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 78, in download_channel
    self._transfer(DOWNLOAD_METHOD, channel_id, baseurl)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 145, in _transfer
    raise e
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 114, in _transfer
    import_channel_by_id(channel_id, self.is_cancelled)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/management/commands/importchannel.py", line 24, in import_channel_by_id
    channel_import.import_channel_from_local_db(channel_id, cancel_check=cancel_check)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/channel_import.py", line 653, in import_channel_from_local_db
    import_manager = initialize_import_manager(channel_id, cancel_check=cancel_check)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/channel_import.py", line 627, in initialize_import_manager
    channel_metadata = read_channel_metadata_from_db_file(get_content_database_file_path(channel_id))
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/channels.py", line 56, in read_channel_metadata_from_db_file
    source = Bridge(sqlite_file_path=channeldbpath)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/sqlalchemybridge.py", line 207, in __init__
    db_matches_schema(self.Base, self.session)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/content/utils/check_schema_db.py", line 22, in db_matches_schema
    tables = iengine.get_table_names()
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/reflection.py", line 201, in get_table_names
    self.bind, schema, info_cache=self.info_cache)
  File "<string>", line 2, in get_table_names
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/reflection.py", line 54, in cache
    ret = fn(self, con, *args, **kw)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/dialects/sqlite/base.py", line 1160, in get_table_names
    rs = connection.execute(s)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 2075, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 942, in execute
    return self._execute_text(object, multiparams, params)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1104, in _execute_text
    statement, parameters
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/lingyiwang/Documents/LearningEquality/kolibri/kolibri/dist/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/4xp6)
benjaoming commented 6 years ago

Here is another type of malformed database. Happening on a Raspberry Pi w/ Raspian and Kolibri from PPA:

  1. Upgraded Kolibri 0.10.1 on top of 0.10.1-beta1, which was working fine.
  2. Observation 1: There are not db schema differences between these two.
  3. Observation 2: It was a very heavy db (126 MB), carrying all of Khan Academy.
  4. Then got the malformed db. It seems to have happened while stopping the old instance!
INFO 2018-08-03 15:03:16,730 options Option HTTP_PORT in section [Deployment] being overridden by environment variable KOLIBRI_LISTEN_PORT
INFO 2018-08-03 15:03:19,796 apps Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO 2018-08-03 15:03:21,587 cli Kolibri server has successfully been stopped.
ERROR 2018-08-03 16:14:07,731 cli 
Your database is corrupted. This is a known issue that is usually fixed by running this command: 

    sqlite3 /mnt/data/.kolibri/db.sqlite3 .dump | sqlite3 fixed.db 
    cp fixed.db /mnt/data/.kolibri/db.sqlite3

Notice that you need the 'sqlite3' command available on your system prior to running this.

ERROR 2018-08-06 16:09:36,350 cli 
Your database is corrupted. This is a known issue that is usually fixed by running this command: 

    sqlite3 /mnt/data/.kolibri/db.sqlite3 .dump | sqlite3 fixed.db 
    cp fixed.db /mnt/data/.kolibri/db.sqlite3

Notice that you need the 'sqlite3' command available on your system prior to running this.