Open heshaShawky opened 10 months ago
You should be awaiting methods on the migrator, but that is unlikely to cause the issue.
Are you using a database backed by the sqflite
package? The E/SQLiteLog
indicates that. That package is using the sqlite3 version from the operating system, which can be outdated. Older sqlite3 versions don't support upserts.
Is there a particular reason you're not using a NativeDatabase
with sqlite3_flutter_libs
as described in the setup guide? That setup includes its own sqlite3
library with your application, which is typically much newer.
You should be awaiting methods on the migrator, but that is unlikely to cause the issue.
Are you using a database backed by the
sqflite
package? TheE/SQLiteLog
indicates that. That package is using the sqlite3 version from the operating system, which can be outdated. Older sqlite3 versions don't support upserts.Is there a particular reason you're not using a
NativeDatabase
withsqlite3_flutter_libs
as described in the setup guide? That setup includes its ownsqlite3
library with your application, which is typically much newer.
I am using the same as the guide
dependencies:
drift: ^2.10.0
drift_sqflite: ^2.0.1
sqlite3_flutter_libs: ^0.5.17
Also, I have solved the issue of backup by doing this,
but now on some user devices, he is getting an error creating a record ( mostly a migration thing ) but it is only on his device so far, so I don't know why there is a difference between devices like that and what I did wrong to cause this :(
// await _database.batch(
// (batch) {
// batch.insertAllOnConflictUpdate(_database.transactions, transactions);
// batch.insertAllOnConflictUpdate(_database.categories, categories);
// batch.insertAllOnConflictUpdate(_database.wallets, wallets);
// },
// );
await _database.batch(
(batch) {
batch.insertAll(_database.transactions, transactions,
mode: InsertMode.insertOrReplace);
batch.insertAll(_database.categories, categories,
mode: InsertMode.insertOrReplace);
batch.insertAll(_database.wallets, wallets,
mode: InsertMode.insertOrReplace);
},
);
he is getting an error creating a record ( mostly a migration thing ) but it is only on his device so far
Do you have access to the stack trace?
he is getting an error creating a record ( mostly a migration thing ) but it is only on his device so far
Do you have access to the stack trace?
Yes from Firebase Crashlytics ( it says that the field doesn't exist, even when he uninstalls the app and reinstalls it again, so it should there's no migration there but still get an error that causes his screen to go grey blank )
Fatal Exception: io.flutter.plugins.firebase.crashlytics.FlutterError: DatabaseException(no such column: transaction_in_details (code 1): , while compiling: UPDATE "transactions" SET "value" = ?, "transaction_in_details" = ?, "description" = ?, "date" = ?, "repeated_period" = ?, "category_id" = ?, "wallet_id" = ?, "attachments" = ?, "priority" = ?, "contact" = ?, "location" = ?, "synced" = ? WHERE "id" = ?;
#################################################################
Error Code : 1 (SQLITE_ERROR)
Caused By : SQL(query) error or missing database.
(no such column: transaction_in_details (code 1): , while compiling: UPDATE "transactions" SET "value" = ?, "transaction_in_details" = ?, "description" = ?, "date" = ?, "repeated_period" = ?, "category_id" = ?, "wallet_id" = ?, "attachments" = ?, "priority" = ?, "contact" = ?, "location" = ?, "synced" = ? WHERE "id" = ?;)
#################################################################) sql 'UPDATE "transactions" SET "value" = ?, "transaction_in_details" = ?, "description" = ?, "date" = ?, "repeated_period" = ?, "category_id" = ?, "wallet_id" = ?, "attachments" = ?, "priority" = ?, "contact" = ?, "location" = ?, "synced" = ? WHERE "id" = ?;' args [60.0, null, فروج, 1652615700, null, 2, 7410de8c-8189-49c4-b75f-9c121db28f49, [], 0, null, {"lat":null,"lng":null,"address":null}, 1, 04257e41-628b-41d6-b690-35643c0fd378]
at .wrapDatabaseException(exception_impl.dart:11)
at SqfliteDatabaseMixin.txnApplyBatch.<fn>(database_mixin.dart:699)
at BasicLock.synchronized(basic_lock.dart:33)
at SqfliteDatabaseMixin.txnSynchronized(database_mixin.dart:490)
at _SqfliteDelegate.runBatched(drift_sqflite.dart:77)
at Batch._commit(batch.dart:199)
at DatabaseRepository.transactionsBackedup(database_repository.dart:110)
at SettingsService.backup(settings_service.dart:202)
at BackupCubit.backup(backup_cubit.dart:31)
I hope there's an update with this, why on some phones does the new column not exist?
Sorry, I still don't fully understand what's happening here. Given that this happens on some phones only, I think it's due to an older sqlite3 version. Since you already have a dependency on sqlite3_flutter_libs
, you could try using a NativeDatabase
to use more recent sqlite3 versions. That won't fix the problem for existing users where the database is missing the column though.
How are you calling the _migrationV6
functions, are there try/catch blocks in the onUpgrade
callback?
Here's my whole database file
The weird thing is, it was all fine before this last migration!
Also I don't know how to do what you saying about using more recent SQLite 3, how to achieve that
@singleton
@DriftDatabase(tables: [
Users,
Settings,
Wallets,
Transactions,
Categories,
], daos: [
UsersDao,
SettingsDao,
WalletsDao,
TransactionsDao,
CategoryDao,
], include: {
'categories.drift'
})
class AppDatabase extends _$AppDatabase {
static const Uuid uuid = Uuid();
AppDatabase.connect(DatabaseConnection connection)
: super.connect(connection);
AppDatabase()
: super(SqfliteQueryExecutor.inDatabaseFolder(
path: 'db.sqlite',
logStatements: true,
));
@override
int get schemaVersion => 6;
@override
MigrationStrategy get migration =>
MigrationStrategy(beforeOpen: _beforeOpen, onUpgrade: _onUpgrade);
Future<void> _beforeOpen(OpeningDetails details) async {
if (details.wasCreated) {
// create new settings row on database first created
await into(settings).insert(
const Setting(
isFirstLaunch: true,
id: 1,
),
);
} else if (details.hadUpgrade && details.versionBefore == 3) {
// create a backup after users updating
getIt<ISettingsService>().backup();
}
}
Future<void> _onUpgrade(Migrator m, int from, int to) async {
switch (from) {
case 1:
await _migrationV2(m);
break;
case 2:
await _migrationV3(m);
break;
case 3:
await _migrationV4(m);
break;
case 4:
await _migrationV5(m);
break;
case 5:
await _migrationV6(m);
break;
default:
}
}
Future<void> _migrationV2(Migrator m) async {
await Future.wait([
m.addColumn(wallets, wallets.endDate),
m.addColumn(wallets, wallets.doesItHasExpireDate),
]);
}
Future<void> _migrationV3(Migrator m) async {
await Future.wait(
[
m.addColumn(settings, settings.isProtected),
m.alterTable(
TableMigration(
users,
columnTransformer: {
users.email: const Constant(null),
users.token: const Constant(null),
users.profilePicture: const Constant(null),
},
newColumns: [users.email, users.token, users.profilePicture],
),
),
m.alterTable(
TableMigration(
wallets,
columnTransformer: {
wallets.id: wallets.id.cast<String>(),
wallets.countryAndCurrency: const Constant(null)
},
newColumns: [wallets.countryAndCurrency],
),
),
m.alterTable(
TableMigration(
categories,
columnTransformer: {
categories.id: categories.id.cast<String>(),
categories.parent: categories.parent.cast<String>(),
categories.cOrder: const Constant(0),
},
newColumns: [categories.cOrder],
),
),
m.alterTable(
TableMigration(
transactions,
columnTransformer: {
transactions.id: transactions.id.cast<String>(),
transactions.walletId: transactions.walletId.cast<String>(),
transactions.categoryId: transactions.categoryId.cast<String>(),
transactions.priority: const Constant(null),
transactions.contact: const Constant(null),
transactions.location: const Constant(null),
},
newColumns: [
transactions.priority,
transactions.contact,
transactions.location
],
),
)
],
);
}
Future<void> _migrationV4(Migrator m) async {
// users migration
m.addColumn(users, users.isActive);
// transactions migration
m.alterTable(
TableMigration(transactions, columnTransformer: {
transactions.synced: const Constant(false),
}, newColumns: [
transactions.synced
]),
);
// wallets migration
m.alterTable(
TableMigration(wallets, columnTransformer: {
wallets.moneySource: const CustomExpression('moeny_resource'),
wallets.isActive: const Constant(false),
}, newColumns: [
wallets.isActive
]),
);
// updating categories
getIt<ICategoryService>().updateDefaultCategories(
(await getIt<ISettingsService>().getSavedSettings())?.language ?? 'en');
final List<Transaction> oldTransactions = await (select(transactions)
..where((tbl) => tbl.contact.isNotNull()))
.get();
for (var i = 0; i < oldTransactions.length; i++) {
final Transaction transaction = oldTransactions[i];
final Contact? contact =
await getIt<ContactPickerService>().find(transaction.contact);
update(transactions).replace(
transaction.copyWith(
contact: Value(contact),
),
);
}
// updating users database
customUpdate(
'UPDATE wallets SET user_id = ( SELECT id FROM users LIMIT 1 )');
}
Future<void> _migrationV5(Migrator m) async {
await m.alterTable(
TableMigration(transactions, columnTransformer: {
transactions.transactionInDetails: Constant(
json.encode([]),
),
}, newColumns: [
transactions.transactionInDetails,
]),
);
}
Future<void> _migrationV6(Migrator m) async {
await m.addColumn(transactions, transactions.transactionInDetails);
}
}
Your onUpgrade
callback is only doing a single migration step :scream: If a user has a version of your app with schema version 2 installed and now updates to the latest version, _onUpgrade(m, 2, 6)
is called. The switch then calls _migrationV3
and calls it a day, missing the migrations from 4 to 5 and 5 to 6.
Your migration should look like this:
Future<void> _onUpgrade(Migrator m, int from, int to) async {
for (var current = from; current < to; current++) {
switch (current) {
case 1:
await _migrationV2(m);
break;
case 2:
await _migrationV3(m);
break;
case 3:
await _migrationV4(m);
break;
case 4:
await _migrationV5(m);
break;
case 5:
await _migrationV6(m);
break;
default:
throw StateError('Unhandled migration from version $from');
}
current++;
}
}
This is likely the source of the problem - old users will never get the latest version of your database. Unfortunately this is pretty hard to fix afterwards - the database having the latest schema version doesn't mean much if the schema could contain anything. If losing local data is an acceptable solution, you could upgrade the schema version to 7 and write a migration script that resets the database:
Future<void> _onUpgrade(Migrator m, int from, int to) async {
if (from < 7) {
// Older migrations were broken, reset the database in v7 to fix this.
final reversedEntities = allSchemaEntities.toList().reversed;
for (final entity in reversedEntities) {
await m.drop(entity);
}
// Re-create them now
await m.createAll();
return;
}
for (var current = from; current < to; current++) {
switch (current) {
// This is where you'd put migrations from v7 onwards if you need
// a v8 version later.
default:
throw StateError('Unhandled migration from version $from');
}
current++;
}
}
Also I don't know how to do what you saying about using more recent SQLite 3, how to achieve that
Replace
AppDatabase()
: super(SqfliteQueryExecutor.inDatabaseFolder(
path: 'db.sqlite',
logStatements: true,
));
with
AppDatabase()
: super(LazyDatabase(() async {
final dbFolder = await getDatabasesPath();
final file = File(p.join(dbFolder, 'db.sqlite'));
return NativeDatabase.createInBackground(file);
}));
You'd also have to add imports:
import 'package:drift/native.dart';
import 'package:path/path.dart' as p;
import 'package:sqflite/sqflite.dart' show getDatabasesPath;
Yes, I think you are right :'(
But The phones that are affected I got someone to try to uninstall the app and reinstall it so they sure have the latest version of the database directly without a need to migrate from the old version I think but they still have a problem with that column!
also why did you put 2 times currrent++
on the loop and, let me know if that way will work to update and not losing users data
@singleton
@DriftDatabase(tables: [
Users,
Settings,
Wallets,
Transactions,
Categories,
], daos: [
UsersDao,
SettingsDao,
WalletsDao,
TransactionsDao,
CategoryDao,
], include: {
'categories.drift'
})
class AppDatabase extends _$AppDatabase {
static const Uuid uuid = Uuid();
AppDatabase.connect(DatabaseConnection connection)
: super.connect(connection);
AppDatabase()
: super(SqfliteQueryExecutor.inDatabaseFolder(
path: 'db.sqlite',
logStatements: true,
));
@override
int get schemaVersion => 6;
@override
MigrationStrategy get migration =>
MigrationStrategy(beforeOpen: _beforeOpen, onUpgrade: _onUpgrade);
Future<void> _beforeOpen(OpeningDetails details) async {
if (details.wasCreated) {
// create new settings row on database first created
await into(settings).insert(
const Setting(
isFirstLaunch: true,
id: 1,
),
);
} else if (details.hadUpgrade && details.versionBefore == 3) {
// create a backup after users updating
getIt<ISettingsService>().backup();
}
}
Future<void> _onUpgrade(Migrator m, int from, int to) async {
if (from < 7) {
// get backup to my server for signed users
await getIt<SettingsService>().backup();
// let's get a backup first!
final List<Transaction> transactionsList = (await customSelect(
'SELECT * FROM transactions',
readsFrom: {transactions}).get())
.map((e) => transactions.map(e.data))
.toList();
final List<User> usersList =
(await customSelect('SELECT * FROM users', readsFrom: {users}).get())
.map((e) => users.map(e.data))
.toList();
final List<Wallet> walletsList =
(await customSelect('SELECT * FROM wallets', readsFrom: {wallets})
.get())
.map((e) => wallets.map(e.data))
.toList();
final List<Category> categoriesList = (await customSelect(
'SELECT * FROM categories',
readsFrom: {categories}).get())
.map((e) => categories.map(e.data))
.toList();
final List<Setting> settingsList =
(await customSelect('SELECT * FROM settings', readsFrom: {settings})
.get())
.map((e) => settings.map(e.data))
.toList();
// Older migrations were broken, reset the database in v7 to fix this.
final reversedEntities = allSchemaEntities.toList().reversed;
for (final entity in reversedEntities) {
await m.drop(entity);
}
// Re-create them now
await m.createAll();
// no add all data to database
await batch((batch) {
batch.insertAll(settings, settingsList);
batch.insertAll(users, usersList);
batch.insertAll(categories, categoriesList);
batch.insertAll(wallets, walletsList);
batch.insertAll(transactions, transactionsList);
});
return;
}
for (var current = from; current < to; current++) {
switch (current) {
// nothing here as there's nothing to do for now
default:
throw StateError('Unhandled migration from version $from');
}
current++; // why you adding another increase in here
}
}
}
How is it still I get this error from some devices after that upgrade which is supposed to drop the database tables and make a new one!, this becomes really weird
Fatal Exception: io.flutter.plugins.firebase.crashlytics.FlutterError: DatabaseException(no such column: transaction_in_details (code 1 SQLITE_ERROR[1]): , while compiling: UPDATE "transactions" SET "value" = ?, "transaction_in_details" = ?, "description" = ?, "date" = ?, "repeated_period" = ?, "category_id" = ?, "wallet_id" = ?, "attachments" = ?, "priority" = ?, "contact" = ?, "location" = ?, "synced" = ? WHERE "id" = ?;) sql 'UPDATE "transactions" SET "value" = ?, "transaction_in_details" = ?, "description" = ?, "date" = ?, "repeated_period" = ?, "category_id" = ?, "wallet_id" = ?, "attachments" = ?, "priority" = ?, "contact" = ?, "location" = ?, "synced" = ? WHERE "id" = ?;' args [1340.0, null, null, 1661062693, null, 52, 554c520c-93f9-4f19-b41f-9ef43e162eb2, [], 0, null, {"lat":null,"lng":null,"address":null}, 1, 54cf6657-5439-4339-9132-409361dceafd]
at .wrapDatabaseException(exception_impl.dart:11)
at SqfliteDatabaseMixin.txnApplyBatch.<fn>(database_mixin.dart:699)
at BasicLock.synchronized(basic_lock.dart:33)
at SqfliteDatabaseMixin.txnSynchronized(database_mixin.dart:490)
at _SqfliteDelegate.runBatched(drift_sqflite.dart:77)
at Batch._commit(batch.dart:199)
at DatabaseRepository.transactionsBackedup(database_repository.dart:99)
at SettingsService.backup(settings_service.dart:202)
at AppDatabase._onUpgrade(database.dart:88)
at GeneratedDatabase.beforeOpen.<fn>(db_base.dart:129)
at DelegatedDatabase._runMigrations(engines.dart:458)
at DelegatedDatabase.ensureOpen.<fn>(engines.dart:426)
at Selectable.getSingleOrNull(query.dart:240)
at SettingsService.getSavedSettings(settings_service.dart:108)
at SettingsCubit.getSavedSettings(settings_cubit.dart:34)
Fatal Exception: io.flutter.plugins.firebase.crashlytics.FlutterError: DatabaseException(no such column: transaction_in_details (code 1 SQLITE_ERROR): , while compiling: UPDATE "transactions" SET "value" = ?, "transaction_in_details" = ?, "description" = ?, "date" = ?, "repeated_period" = ?, "category_id" = ?, "wallet_id" = ?, "attachments" = ?, "priority" = ?, "contact" = ?, "location" = ?, "synced" = ? WHERE "id" = ?;) sql 'UPDATE "transactions" SET "value" = ?, "transaction_in_details" = ?, "description" = ?, "date" = ?, "repeated_period" = ?, "category_id" = ?, "wallet_id" = ?, "attachments" = ?, "priority" = ?, "contact" = ?, "location" = ?, "synced" = ? WHERE "id" = ?;' args [10.0, null, null, 1684484028, null, 18, 7b5de104-3bae-4ab7-bceb-470bb69f6886, null, 1, null, null, 1, 14cff190-f61d-11ed-958a-13ab8a76bf30]
at .wrapDatabaseException(exception_impl.dart:11)
at SqfliteDatabaseMixin.txnApplyBatch.<fn>(database_mixin.dart:699)
at BasicLock.synchronized(basic_lock.dart:33)
at SqfliteDatabaseMixin.txnSynchronized(database_mixin.dart:490)
at _SqfliteDelegate.runBatched(drift_sqflite.dart:77)
at Batch._commit(batch.dart:199)
at DatabaseRepository.transactionsBackedup(database_repository.dart:99)
at SettingsService.backup(settings_service.dart:202)
at AppDatabase._onUpgrade(database.dart:88)
at GeneratedDatabase.beforeOpen.<fn>(db_base.dart:129)
at DelegatedDatabase._runMigrations(engines.dart:458)
at DelegatedDatabase.ensureOpen.<fn>(engines.dart:426)
at Selectable.getSingleOrNull(query.dart:240)
at SettingsService.getSavedSettings(settings_service.dart:108)
at SettingsCubit.getSavedSettings(settings_cubit.dart:34)
@singleton
@DriftDatabase(tables: [
Users,
Settings,
Wallets,
Transactions,
Categories,
], daos: [
UsersDao,
SettingsDao,
WalletsDao,
TransactionsDao,
CategoryDao,
], include: {
'categories.drift'
})
class AppDatabase extends _$AppDatabase {
static const Uuid uuid = Uuid();
AppDatabase.connect(DatabaseConnection connection)
: super.connect(connection);
AppDatabase()
: super(SqfliteQueryExecutor.inDatabaseFolder(
path: 'db.sqlite',
logStatements: true,
));
@override
int get schemaVersion => 7;
@override
MigrationStrategy get migration =>
MigrationStrategy(beforeOpen: _beforeOpen, onUpgrade: _onUpgrade);
Future<void> _beforeOpen(OpeningDetails details) async {
if (details.wasCreated) {
// create new settings row on database first created
await into(settings).insert(
const Setting(
isFirstLaunch: true,
id: 1,
),
);
} else if (details.hadUpgrade && details.versionBefore == 3) {
// create a backup after users updating
getIt<ISettingsService>().backup();
}
}
Future<void> _onUpgrade(Migrator m, int from, int to) async {
if (from < 7) {
try {
// get backup to my server for signed users
await getIt<ISettingsService>().backup();
} catch (e, stack) {
print('$e $stack');
getIt<FirebaseCrashlytics>()
.recordError(e, stack, fatal: true, printDetails: true);
}
// let's get a backup first!
final List<Transaction> transactionsList = (await customSelect(
'SELECT * FROM transactions',
readsFrom: {transactions}).get())
.map((e) => transactions.map(e.data))
.toList();
final List<User> usersList =
(await customSelect('SELECT * FROM users', readsFrom: {users}).get())
.map((e) => users.map(e.data))
.toList();
final List<Wallet> walletsList =
(await customSelect('SELECT * FROM wallets', readsFrom: {wallets})
.get())
.map((e) => wallets.map(e.data))
.toList();
final List<Category> categoriesList = (await customSelect(
'SELECT * FROM categories',
readsFrom: {categories}).get())
.map((e) => categories.map(e.data))
.toList();
final List<Setting> settingsList =
(await customSelect('SELECT * FROM settings', readsFrom: {settings})
.get())
.map((e) => settings.map(e.data))
.toList();
// Older migrations were broken, reset the database in v7 to fix this.
final reversedEntities = allSchemaEntities.toList().reversed;
for (final entity in reversedEntities) {
await m.drop(entity);
}
// Re-create them now
await m.createAll();
try {
// no add all data to database
await batch((batch) {
batch.insertAll(settings, settingsList);
batch.insertAll(users, usersList);
batch.insertAll(categories, categoriesList);
batch.insertAll(wallets, walletsList);
batch.insertAll(transactions, transactionsList);
});
} catch (e, stack) {
print('$e $stack');
await getIt<ISettingsService>().setupBackup();
getIt<FirebaseCrashlytics>()
.recordError(e, stack, fatal: true, printDetails: true);
}
return;
}
}
}
Here's my transactions table too
part of '../database.dart';
class Transactions extends Table {
TextColumn get id =>
text().nullable().clientDefault(() => AppDatabase.uuid.v4())();
RealColumn get value => real()();
TextColumn get transactionInDetails => text().nullable()(); // this field is exist here, how is not in the database?!
TextColumn get description => text().nullable()();
DateTimeColumn get date => dateTime().withDefault(Constant(DateTime.now()))();
IntColumn get repeatedPeriod => intEnum<RepetitionPeriod>().nullable()();
TextColumn get categoryId =>
text().nullable().customConstraint('NULL REFERENCES categories(id)')();
TextColumn get walletId =>
text().nullable().customConstraint('NULL REFERENCES wallets(id)')();
TextColumn get attachments =>
text().map(const TransactionsAttachmentsConverter()).nullable()();
IntColumn get priority => intEnum<TransactionPriority>().nullable()();
TextColumn get contact => text().map(const ContactConverter()).nullable()();
TextColumn get location => text().map(const LocationConverter()).nullable()();
BoolColumn get synced =>
boolean().withDefault(const Constant(false)).nullable()();
@override
Set<Column> get primaryKey => {id};
}
Describe the bug
I made an update on a table ( adding a column ), I did the migration correctly and tested it and everything went well on my phone ( Xiaomi 11t Android version 13 )
I published the update to find many phones crashing due to the new column not existing!
So I fixed this by upgrading the database version and did the following ( fixed the creating/updating ...etc ) but still I don't know why the AlterTable didn't work
now I have another issue that didn't exist before this database upgrade :(
I let users backup their data on my server by getting those data as JSON and saving it.
To install those data again I am using the following function that was working just fine and still my personal phone, but not on another like ( Huawei P30 pro )
The error I get is the following ( from using insertAllOnConflictUpdate )