Backup(filename, password) not working in 4.0.2

[slodki]

tested with m_db->Backup(newFileName.GetFullPath(), new_password); code.

v3.5.9:

• unencrypted -> Backup(filename, wxEmptyString) :ok:
• encrypted -> Backup(filename, wxEmptyString) :ok:
• unencrypted -> Backup(filename, password) :ok:

v4.0.2:

• unencrypted -> Backup(filename, wxEmptyString) :ok:
• encrypted -> Backup(filename, wxEmptyString) :ok:
• unencrypted -> Backup(filename, password) - new file is created, looks like encrypted, but cannot be opened with the same password...

This can be similar to #27 but for Backup function IMHO.

[utelle]

I'll take a look what's going on. However, this may take a few days, since I'm currently very busy in my job.

[utelle]

In fact, the issue is mainly caused by a restriction of the SQLite3 backup API: the number of reserved bytes per database page can't be changed during a backup operation. Since the official SQLite3 version does not include an encryption extension, the limitation is not mentioned in the SQLite3 documentation, but only in the source code of the backup API itself. Therefore I wasn't aware of this restriction.

The problem is not related to #27. The rekey feature uses a special version of the VACUUM command to convert between different encryption ciphers, while the backup API just copies database pages from the source database to the target database.

Please find below a table describing with which encryption cipher combinations the backup API can be used.

Backup  To	SQLite3	wxSQLite3	wxSQLite3	wxSQLite3	SQLCipher v1	SQLCipher v2+
From	Plain	AES-128	AES-256	ChaCha20 Poly1305	AES-256	AES-256 SHA1
SQLite3 Plain	:ok:	:ok:	:ok:	:x:	:x:	:x:
wxSQLite3 AES-128	:ok:	:ok:	:ok:	:x:	:x:	:x:
wxSQLite3 AES-256	:ok:	:ok:	:ok:	:x:	:x:	:x:
wxSQLite3 ChaCha20 Poly1305	:ok: :small_red_triangle_down:	:question: :small_red_triangle_down:	:question: :small_red_triangle_down:	:ok:	:x:	:x:
SQLCipher v1 AES-256	:ok: :small_red_triangle_down:	:ok: :small_red_triangle_down:	:ok: :small_red_triangle_down:	:x:	:ok:	:x:
SQLCipher v2+ AES-256 SHA1	:ok: :small_red_triangle_down:	:question: :small_red_triangle_down:	:question: :small_red_triangle_down:	:x:	:x:	:ok:

Symbol	Description
:ok:	Works
:question:	Not yet tested, but probably works
:x:	Does not work
:small_red_triangle_down:	Keeps reserved bytes per database page

Note: It is recommended to use the same encryption cipher for the source and target database.

[utelle]

However, I agree that wxSQLite3 should not create unusable backup database files without issuing a warning message or throwing an exception. I have to further investigate why the backup API does not signal a failure although the backup API implementation checks whether the number of reserved bytes per database page of the source and target database do match or not.

[slodki]

New default cipher selected for new wxSQLite3 is the most problematic one... Adding the warning or exception is important, because change brakes previous behavior of Backup/Restore functions. I need cipher compatible with Android, to exchange encrypted db PC<->Android. I'll test old version of SQLCipher. Can I set default cipher at app initialization step for all future open/create db operations?

[utelle]

Well, I'm aware of that changing the default cipher is a big step. However, I think it is justifiable almost 12 years after having introduced the wxSQLite3 encryption extension. And therefore I also changed the major version number to make clear that the new version introduces major new features.

Regarding the warning/exception I first have to find out why the backup API proceeds although the numbers of reserved bytes of source and target database do not match. As soon as I have a solution I will publish it, of course.

The default cipher can be selected at compile time, as before, by setting the symbol CODEC_TYPE to one of the values CODEC_TYPE_AES128, CODEC_TYPE_AES256, CODEC_TYPE_CHACHA20, or CODEC_TYPE_SQLCIPHER (see config.gcc or wx_setup.props).

If you want compatibility with SQLCipher you most likely need to set symbol WXSQLITE3_USE_SQLCIPHER_LEGACY as well, otherwise wxSQLite3 will exclude bytes 16 to 23 of the database header from encryption to enhance compatibility with SQLite. (AFAIK a future version of SQLCipher will also introduce the option to exclude part of the database header from encryption, but it is still in pre-release.)

Using the function wxsqlite3_config and wxsqlite3_config_cipher you can also set the global defaults of the encryption extension. Just call these functions before opening any database connection and pass a NULL pointer for the SQLite3 database handle. Prefix the parameter names with default: to affect the default values (a more detailed description can be found here).

Each database connection creates a copy of the global cipher parameter tables. That is, the defaults an active database connection uses can be set separately for each connection.

[utelle]

Solving the problem with the backup API will not be trivial, unfortunately. If the source database is not encrypted, the number of reserved bytes per page is typically zero. If a password is specified on invocation of the wxSQLite3 Backup method, the new default cipher ChaCha20-Poly1305 tries to reserve 32 bytes per page for the HMAC. However, the SQLite3 backup API just resets the number of reserved bytes to zero, without allowing the cipher to complain and to set an error code.

Most likely this is the reason that for example SQLCipher does not officially support the SQLite3 backup API. SQLCipher recommends to use their export function to create database backups.

In non-legacy mode (that is, database header bytes 16 to 23 are always unencrypted) it would be possible to implement a variant without HMAC - and therefore without the requirement to reserve any bytes per database page. However, for example in case of SQLCipher cipher the resulting backup database would not be compatible with the original SQLCipher library.

To allow using the backup API for plain databases with a cipher which makes use of reserved bytes, it would be necessary to adjust the plain database reserving the required number of bytes per page, first.

I'm not sure which approach should be used:

1. Simply disallow use of backup for cipher combinations marked as "not working"
2. Resort to cipher variants without reserved bytes (aka no HMAC), if possible, allowing to support most combinations marked as "not working" (but incompatible with legacy cipher databases)
3. Find a way to adjust the number of reserved bytes for the source database (could be an expensive operation in respect to time and disk space), before attempting the backup

[utelle]

Commit 9d616e1e74b2c300191498cea37a032e8b81781c attempts to solve the issue to a certain degree. A backup operation will now be aborted, if the cipher combination for source and target database can't be used. For non-legacy ciphers a fallback solution with reduced security is implemented. That is, if a password is given for the backup operation for a plain unencrypted database, the new default cipher ChaCha20 will be used, but without HMAC and page nonce.

[utelle]

As far as I can tell the current solution works. Maybe it is not ideal. Certainly it would be preferable to get a regularly encrypted backup database even if the source database is not encrypted at all. However, in my opinion it is not a typical use case to create encrypted backup databases from unencrypted databases.

The typical use case - using the same cipher scheme for source and target database - is covered by the current implementation. Therefore I'm closing this issue.