Unable to open database with enabled vfs=win32-longpath option

[schorle80]

I need to use databases in paths that are longer than 1024 characters. Therefore sqlite has different VFSes (https://www.sqlite.org/vfs.html).

But if i set vfs=win32-longpath in my connecton string i cannot open any database that is encrypted. Also dbs in paths shorter than 260 characters cannot be opened any more. I use RC4 and chacha20 encryption.

But when i remove encryption from the database it works without any problems with paths shorter than 260 characters and also with paths with more than 1024 characters.

So it must be something wrong when there is an encryption for the db. I use an URI connection string like this: "file:///E:/DatabaseFilename.db?vfs=win32-longpath&cipher=chacha20&key=XXXX"

Is there a bug in the SQLite3MultipleCiphers implementation? Or am i doing something wrong?

[utelle]

I need to use databases in paths that are longer than 1024 characters. Therefore sqlite has different VFSes (https://www.sqlite.org/vfs.html).

I'm aware of that.

But if i set vfs=win32-longpath in my connecton string i cannot open any database that is encrypted.

The VFS win32-longpath does not support encryption on its own, so you can't create or open an encrypted database with this VFS alone.

On initialization SQLite3 Multiple Ciphers creates a VFS shim for the default VFS of the operating system (for Windows that is the VFS win32). The name of this VFS shim that supports encryption is multipleciphers-win32.

If encryption support for other VFSs is required, a VFS shim has to be created for those VFSs, too.

Also dbs in paths shorter than 260 characters cannot be opened any more. I use RC4 and chacha20 encryption.

The problem does not depend on the encryption scheme or the path length, but only on the VFS in use.

But when i remove encryption from the database it works without any problems with paths shorter than 260 characters and also with paths with more than 1024 characters.

Of course, the VFS win32-longpath supports unencrypted databases with long as well as short paths.

So it must be something wrong when there is an encryption for the db. I use an URI connection string like this: "file:///E:/DatabaseFilename.db?vfs=win32-longpath&cipher=chacha20&key=XXXX"

I agree that it would be nice to select a VFS with encryption support without requiring to set up a VFS shim manually. In the current implementation a VFS shim needs to be created in advance.

Is there a bug in the SQLite3MultipleCiphers implementation? Or am i doing something wrong?

The SQLite3 Multiple Ciphers implementation is correct, at least as far as I know. However, unfortunately the documentation lacks the information what needs to be done to be able to use other VFSs than the default one with encryption support.

The SQLite3 Multiple Ciphers API includes the function

int sqlite3mc_vfs_create(const char* zVfsReal, int makeDefault);

that can be used to create a new VFS shim with encryption support, In your use case you have to call

int rc = sqlite3mc_vfs_create("win32-longpath", 0);

This creates a VFS shim with encryption support using the VFS win32-longpath for the real file access. The name of the VFS shim will be multipleciphers-win32-longpath - and that name can then be used in the vfs URI parameter (vfs=multipleciphers-win32-longpath).

I will complement the documentation in the near future. Additionally, I will check whether I can implement an easier way to use any of the available VFSs with encryption support.

[schorle80]

Oh great. Now it works perfectly. Thank you very much.

[utelle]

Oh great. Now it works perfectly. Thank you very much.

You are welcome. So, I will close the issue.

In the meantime I experimented a bit. As a result the next release will allow to specify a VFS name with encryption support without the need to call sqlite3mc_vfs_create(). Simply using a VFS name with the "multipleciphers-" prefix will instantiate the VFS shim if if doesn't already exist. Calling sqlite3mc_vfs_create() will only be necessary, if a certain VFS shim with encryption support should be made the default VFS.

[schorle80]

@utelle Maybe you have also experiences with this Problem. In windows everything works fine with long paths and VFS set (win32-longpath). But today i tried to use long paths in linux (Ubuntu 20.04.5 LTS (Focal Fossa)) with my .Net5 application.

I tried without setting a special VFS but when the path is longer than 512 characters i get an InvalidDatabaseException when trying to open the database.

All documented VFSes for unix seems not to solve the problem.

Do you have an idea? Or do you know if there exists a limitation for sqlite in path length in unix?

[utelle]

Maybe you have also experiences with this Problem. In windows everything works fine with long paths and VFS set (win32-longpath). But today i tried to use long paths in linux (Ubuntu 20.04.5 LTS (Focal Fossa)) with my .Net5 application.

I tried without setting a special VFS but when the path is longer than 512 characters i get an InvalidDatabaseException when trying to open the database.

That is a hard coded limitation in the SQLite sources. For Unix-like systems there is a symbol MAX_PATHNAME defined:

#define MAX_PATHNAME 512

This number is fixed and is used for all VFSs.

All documented VFSes for unix seems not to solve the problem.

Right. All Unix VFSs use the same limit for pathnames.

Do you have an idea? Or do you know if there exists a limitation for sqlite in path length in unix?

AFAIK the limitation to 512 characters in SQLite is arbitrary. Most Linux systems should support pathname with up to 4096 characters in total.

So, IMHO there are 2 options:

1. You use symbolic links to shorten the effective pathname length, so that the limit of 512 characters is not exceeded, or
2. You redefine the symbol MAX_PATHNAME by changing it in the sources to

   #define MAX_PATHNAME 4096

   For most Linux file systems the maximum total pathname length is 4096.

[schorle80]

So, IMHO there are 2 options:

1. You use symbolic links to shorten the effective pathname length, so that the limit of 512 characters is not exceeded, or
2. You redefine the symbol MAX_PATHNAME by changing it in the sources to

   
   #define MAX_PATHNAME 4096

We already use a self compiled version because we also need RC4 encryption . So i set MAX_PATHNAME to 4096 in the source. Now it works. Thanks.

In the meantime I experimented a bit. As a result the next release will allow to specify a VFS name with encryption support without the need to call sqlite3mc_vfs_create(). Simply using a VFS name with the "multipleciphers-" prefix will instantiate the VFS shim if if doesn't already exist. Calling sqlite3mc_vfs_create() will only be necessary, if a certain VFS shim with encryption support should be made the default VFS.

I also tried your newest version (1.6.0) with your extension without the need to call sqlite3mc_vfs_create(). It works perfectly. Thank you very much for your great and fast support.

[utelle]

So, IMHO there are 2 options:

1. You use symbolic links to shorten the effective pathname length, so that the limit of 512 characters is not exceeded, or
2. You redefine the symbol MAX_PATHNAME by changing it in the sources to

   #define MAX_PATHNAME 4096

We already use a self compiled version because we also need RC4 encryption.

I assume you know that SQLite3 Multiple Ciphers supports the RC4 variant that was included in prior versions of System.Data.SQLite.

So i set MAX_PATHNAME to 4096 in the source. Now it works. Thanks.

I asked about MAX_PATHNAME on the SQLite forum. If we are lucky, its value will be configurable in a future SQLite version. However, I'm not too optimistic as another minor change I asked for in the past never happened. If the next SQLite version does not make MAX_PATHNAME configurable, I can add a patch on my own.

I also tried your newest version (1.6.0) with your extension without the need to call sqlite3mc_vfs_create(). It works perfectly. Thank you very much for your great and fast support.

You are welcome.

[schorle80]

Now i have another question to the same subject. We have the problem that we have to load/unload our assemblies (C#, .NetFramework) dynamically from different versions in different AppDomains. For every version of our assembly there is also the according sqlite-multicipher-dll for this version.

Unfortunately it is not possible to unload native dlls (also the sqlite-multicipher-dll). So if a sqlite-multicipher-dll first was loaded in an earlier version which does not support your new feature yet (creating vfs without calling _sqlite3mc_vfscreate), we will get an error if we load and use our actual assembly which assumes that the vfs will be created automatically because in this case the older sqlite-multicipher-dll is used.

I think due to compatibility reasons the only solution is to call _sqlite3mc_vfscreate in our new version because older versions of your sqlite-dll already has this method since a longer time. So our new versions will also work with an older sqlite-multicipher-dll.

Now my question. Do you think it will be a problem if we call _sqlite3mc_vfscreate multiple times in the same process in different AppDomains? Because it will be difficult to know if _sqlite3mc_vfscreate was already called by some AppDomain.

[utelle]

We have the problem that we have to load/unload our assemblies (C#, .NetFramework) dynamically from different versions in different AppDomains. For every version of our assembly there is also the according sqlite-multicipher-dll for this version.

Can't you update older assembly versions to use a more recent SQLite3MC DLL? Isn't that a legitimate requirement to be able to replace an old assembly by a new one - to fix bugs or to introduce new features?

Unfortunately it is not possible to unload native dlls (also the sqlite-multicipher-dll). So if a sqlite-multicipher-dll first was loaded in an earlier version which does not support your new feature yet (creating vfs without calling _sqlite3mc_vfscreate), we will get an error if we load and use our actual assembly which assumes that the vfs will be created automatically because in this case the older sqlite-multicipher-dll is used.

Well, yes, that is certainly a problem. However, the older SQLite3MC versions weren't able to use the VFS "win32-longpath" with encryption support, unless the VFS was explicitly created by the application in advance.

I think due to compatibility reasons the only solution is to call _sqlite3mc_vfscreate in our new version because older versions of your sqlite-dll already has this method since a longer time. So our new versions will also work with an older sqlite-multicipher-dll.

Of course you can always create VFSs explicitly. However, I assume that your older assemblies didn't do that. So, you have to update your assembles anyway, and in that course you could update the SQLite3MC DLL as well.

Now my question. Do you think it will be a problem if we call _sqlite3mc_vfscreate multiple times in the same process in different AppDomains? Because it will be difficult to know if _sqlite3mc_vfscreate was already called by some AppDomain.

In principal, you can call sqlite3mc_vfs_create multiple times, but at the moment this will create memory leaks, because sqlite3mc_vfs_create doesn't check whether the same VFS was already created in a prior call. This is an oversight, and I will adjust the code for the next release.

[schorle80]

Can't you update older assembly versions to use a more recent SQLite3MC DLL? Isn't that a legitimate requirement to be able to replace an old assembly by a new one - to fix bugs or to introduce new features?

For us there is no need to update older versions because new features or bugfixes are always introduced with a newer version or new service packs

In principal, you can call sqlite3mc_vfs_create multiple times, but at the moment this will create memory leaks, because sqlite3mc_vfs_create doesn't check whether the same VFS was already created in a prior call. This is an oversight, and I will adjust the code for the next release.

We have decided to check with _sqlite3_vfsfind if the vfs already exists (multipleciphers-win32-longpath) and only if it not exists we will call _sqlite3mc_vfscreate . So we do not call it multiple times and there should not be any memory leaks.

Thanks again for your help.

[utelle]

We have decided to check with _sqlite3_vfsfind if the vfs already exists (multipleciphers-win32-longpath) and only if it not exists we will call _sqlite3mc_vfscreate . So we do not call it multiple times and there should not be any memory leaks.

Sure, by checking first whether the VFS already exists, you can easily avoid memory leaks.