Unity iOS Support

[Heged19]

Hi,

Firstly, thank you for your project which is very helpful. I use it with Xamarin without problem. But now I try to use it in a Unity3D project and I have some complication with iOS platform. Indeed, the iOS part of SqlitePCL.raw specify a Xamarin.iOS target which is not the target in a Unity iOS project. So library cannot be find. I tried a lot of things like a custom provider, and when building is ok the call to sqlite is throwing linking exception. That's why I'm posting my issue here, maybe you have a solution for me, or maybe the Unity compatibility for such platform is a future feature?

Thank you for your time and your answer!

[ericsink]

I don't have any experience with Unity. So it supports iOS but is not Xamarin-like? What TFM does it use?

In any case, we probably need to get it to use the same approach that is used for Xamarin.iOS.

[bricelam]

Unity transforms .NET assemblies into C++ via IL2CPP. Then compiles the code in Xcode to produce iOS binaries--no Xamarin involved. It looks like any .a files under Assets/Plugins/iOS are included the Xcode project.

Unity also has no built-in support for NuGet. You'd have to see how/if something like NuGetForUnity handles TFMs, RIDs, and native assets.

I'm sure it's all possible, lol, it's just a very different world than we're used to...

[ericsink]

Okay wow.

[bricelam]

Personally, I hope Unity can get up-to-speed on NuGet, take part in the new TFMs, and get involved in whatever the future of .NET AOT is. This would sure make life a lot easier for library authors. 🙂

[Heged19]

Thank you for your quick answer. I think TFM is MonoTouch, Unity is based on Mono, but Xamarin has changed TFM to Xamarin.iOS. So your library is not directly compatible with Unity iOS. I am going to try tomorrow to change your TFM to MonoTouch, and include the sqlite3.a lib in the iOS plugin folder...

[bricelam]

@indiesaudi Any guidance here? (See also #347 for another Unity scenario that falls short.) We get a few questions around this on the dotnet/efcore repo too.

[Heged19]

I have already seen the issue on dotnet/efcore, I think it is the same. And if we can resolve it, it will be very helpful for those who want to use EF on Unity (with iOS target). Problems are different between iOS and Android, because on iOS we try to link SQlite with the native one, I think that's done with Mono.Data.SQlite. I will try to investigate their code to find any clues.

[Heged19]

@ericsink can you explain me how the link is done (with Xamarin.iOS) with the dynamic provider. Because I don't see any dependancies with Xamarin. If I understand well, the green bundle create a dynamic provider which try to load the "sqlite3" library. I think this part is ok, because with the flag NativeLibrary.WHERE_PLAIN, I have no problem on load, but with flag NativeLibrary. WHERE_RUNTIME_RID I have an exception on load (not found). So the load seems to be okay. Error appear when I try to use Sqlite, I have an exception NotSupportedException : "Linked away".

I use Microsoft.Data.Sqlite as wrapper, I will try with sqlite-pcl-net if I see any difference.

I've also investigated Mono.Data.Sqlite, but they do it differently. They use the Interrop sqlite dll, and bind all functionality for iOS target.

[ericsink]

I have never gotten the dynamic provider working with iOS. On that platform, I only use static linking with DllImport("__Internal").

[corytrese]

I am also currently working on this.

[Heged19]

I’ve tried to use DllImport. I had a not matching method so I copied the internal provider and commented this method. Compilation was ok so that means linking with native iOS library too. But I have the « Linked away » exception again... I’ve also tried to set linking settings in Xcode to static but that creates error because some Unity libraries need to be dynamic... @ericsink do you do some specific code with Xamarin iOS to link with the native library? @corytrese do you have any clue on this subject? Have you tried other things? Thanks to all for your help and contribution!

[ericsink]

My SQLitePCLRaw.lib package for iOS contains a method which is called by the bundle package to prevent the linker from stripping away the assembly. It calls:

SQLitePCL.lib.embedded.Init();

[Heged19]

I have seen this method, but it seems to not work without Xamarin. Is it normal that the method is empty?

[ericsink]

Yes, that method is empty.

[corytrese]

I've made some progress today getting DllImport("__Internal") style bindings to work on iOS. Proof of concept of at least sqlite3_open_v2 and sqlite3_close working correctly (outside of SQLitePCLRaw.)

My use case is "compiling custom SQLite3+SEE for Unity on all platforms" which ends up being very similar to SQLitePCLRaw's approach.

[Heged19]

Ok so you are not using SqlitePclRaw? My need is to have a common library between all platforms, included for Hololens, and SqlitePclRaw work on it except iOS. If I cannot found any solution I will use two libraries for Sqlite and use IOC to have different implemantions between platforms.

[corytrese]

Here is how I'm binding in iOS for IL2CPP

    [DllImport("__Internal")]
    private static extern SQLiteErrorCode sqlite3_open_v2([MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(Utf8Marshaler))] string filename, out IntPtr ppDb, SQLiteOpenOptions flags, IntPtr zvfs);

    [DllImport("__Internal")]
    private static extern SQLiteErrorCode sqlite3_close(IntPtr db);

    [DllImport("__Internal", EntryPoint = "sqlite3_config")]
    private static extern SQLiteErrorCode sqlite3_config_3(SQLiteConfiguration op, long i1, long i2);

Would be possible to add an IL2CPP-iOS Provider to SqlitePclRaw using this type of approach? We'd have to consult with the wizard @ericsink

As for my personal situation, I started with SqlitePclRaw initially for my project but found that it didn't have iOS IL2CPP support so I started researching what that would take. Probably the same stuff you're doing.

Also, worth noting that "IL2CPP does not support marshaling delegates that point to instance methods to native code" which has made these:

https://www.sqlite.org/c3ref/collation_needed.html

very hard / impossible to use.

[ericsink]

Yes, I could add a provider specifically for IL2CPP.

But the snippet you posted looks almost exactly like one of the existing providers. It seems unlikely that the difficulties you all are experiencing are at this layer.

For example, the approach in the snippet will have exactly the same kinds of issues with respect to the linker as discussed above. The linker problem, although frustrating, is relatively straightforward compared to the marshaling issues.

That said, the issue of marshaling delegates would be a separate problem, and perhaps one that IL2CPP simply cannot handle.

[corytrese]

Forgive my ignorance -- perhaps the existing providers can handle IL2CPP in Unity -- if the solution is in there, I could not get it to work.

I'm not experiencing any difficulties -- I haven't had any of the described linker issues with IL2CPP. The marshaling delegates issue is certainly a separate problem; I just wanted to bring it up for completeness. Unity does support the [MonoPInvokeCallback] on static methods, so it is I think possible to implement, I just don't know how.

Thanks again for all your work on SqlitePclRaw, truly an amazing library.

[ericsink]

Mostly I'm hoping you folks will forgive MY ignorance with respect to Unity. :-)

There may still be differences in your snippet that I don't understand. But as far as I can tell, the approach is the same one I'm using.

It seems like @Heged19 was pretty close to getting this to work, except perhaps for the linking problem.

[corytrese]

Worth noting, if it wasn't clear before -- IL2CPP [DllImport("Internal")] is linking against the source code in Unity -- not a library or .a file -- for iOS. When Unity translates IL to C++ source code and packages it as an XCode project, it injects any provided C code (in our case, sqlite3.c) so that Internal will find it.

Perhaps I am simply missing the provider that would support that because it isn't obviously named "Unity IL2CPP Provider for iOS" 👍

[ericsink]

@corytrese That's interesting.

So if you're just adding sqlite3.c to your project directly, then you don't need my e_sqlite3 lib package at all.

So maybe just try this provider:

https://www.nuget.org/packages/SQLitePCLRaw.provider.internal/

With this line to initialize it:

SQLitePCL.raw.SetProvider(new SQLitePCL.SQLite3Provider_internal());

[Heged19]

The internal provider is the one I tried, but I didn’t add any library. I think the native one is found (in CoreData framework), because compilation in XCode is ok. When linked method is not found, compilation is not working and method name is displayed. The problem (for my case) is when I try to call a method at runtime, that tells « linked away ». I understand that may be cause by dynamic linking (not supported on iOS except for native). That’s why I’ve tried static linking in XCode settings, but this one is not working with Unity. In a generated project for XCode by Unity, the Unity project is enbedded as a framework. But I saw we can add a sub framework, so maybe I will try to place my sqlite code in this sub framework and try static linking on this one, and keep dynamic to the unity framework. But if it’s working that means some hack after every generated project, not the best...

[corytrese]

What I did to get the internal provider to work was drop the sqlite3.c + sqlite3.h + sqlite3ext.h files into /Assets/Plugins/ and build the Unity project with the IL2CPP Runtime in Player Settings. That works in Unity 2019 LTS for Win32/iOS/Android/Linux/Mac OS X and I assume Nintendo Switch and any other IL2CPP targets like HoloLens.

Alternatively, if you want to use Mono (not available on iOS anyway) you can use a different provider and inject the library (.so, .dylib, .dll) but honestly I'd just use IL2CPP at this point when packaging SQLite3. The performance is dramatically better in some situations -- you are basically skipping the C# to C (P/Invoke, JNI) bridge between your Unity code and SQLite3, instead replacing it with just C++ calling C++ that calls C. That's nice.

[Heged19]

Thanks for the tip! I have tried but still the same error “linked away” for me... What do you use as wrapper? I use Microsoft.Data.Sqlite, but that’s maybe my problem if it works for you! I will try tomorrow another one.

[Heged19]

I have tried with Sqlite-pcl-net instead of Microsoft.Data.Sqlite, there is no more "linked away" error but I have a new one. A NotSupportedException on the constructor SqliteConnection : "ResolveArgumentsInternal-IL2CPP does not support inspection of attribute constructor arguments at run time". Has someone the same error or any idea on the subject?

@corytrese do you try only the compilation or have you try on runtime and call Sqlite methods? Do you use a wrapper like me?

[corytrese]

I am using a Sqlite backed in our game -- so compiling, running 2,700 unit tests, playing the game on iOS/Android/Linux/Mac OS X/Win32 so I know it can be made to work.

However, note that I'm not using Sqlite-pcl-net or Microsoft.Data.Sqlite. I am using a wrapper, but it just does [DllImport] and manages the sqlite3 methods so I can call them from C#.

Are you getting that error on iOS? This is what I think you are hitting -- this regression that came in 2018 (tightening IL2CPP rules)

https://issuetracker.unity3d.com/issues/customattributedata-dot-cpp-il2cpp-does-not-support-inspection-of-attribute-constructor-arguments-at-run-time

https://docs.microsoft.com/en-us/dotnet/api/system.reflection.customattributedata.constructorarguments?view=netcore-2.0

[Heged19]

Thanks for your answer @corytrese ! Yes I'm hitting iOS, I've also seen the regression, so I think it will be complicated to resolve the problem. Do you use a custom wrapper or is it a wrapper I could try?

[bricelam]

@ericsink and I had a thought: Does using SQLitePCLRaw.bundle_green work in Unity on iOS? This bundle uses the system version of libsqlite3, so you shouldn't need to statically link a version of SQLite into your app.

[Heged19]

I cannot use SQLitePCLRaw.bundle_green on Unity iOS because it depends on Xamarin.iOS. But there is something I don't understand. In this bundle the iOS part use dynamic provider, but @ericsink told me:

I have never gotten the dynamic provider working with iOS. On that platform, I only use static linking with DllImport("__Internal").

I have tried to use, without the bundle, the dynamic provider and the internal one, without success. Internal seems to work with a low level wrapper (@corytrese ) but not with wrapper like Microsoft.Data.Sqlite or SQlite-pcl-net.

[ManuBera]

Hey everybody, is there maybe any news to this issue? I've been trying for weeks to get Sqlite running in a Unity iOS build (works on every other platform) and my whole project depends on this :/ Not matter what I try, Xcode is telling me "DllNotFoundException: Unable to load DLL 'e_sqlite3". I've tried initializing with SQLitePCL.Batteries.Init() and SQLitePCL.Batteries_V2.Init(), but to no avail. I would be very thankful for any help.

[corytrese]

The answer is something like this:

"You must modify small portions of SQLNado source to use __Internal and then link the C and C++ sources into IL2CPP via a source-native plugin on your iOS build."

[Heged19]

Hi, sorry but I forgot to give some news on the subject. I have found a solution last year.

First, you need to upgrade your Unity with a 2022 version, there is an improvement in the il2cpp build, previous versions was creating some errors on the link with SQLite (it was regression in Unity).

Then you cannot use the Batteries.Init(). You have to set the following provider instead: SQLitePCL.raw.SetProvider(new SQLitePCL.SQLite3Provider_sqlite3()); Don’t forget to add the matching library in your plug-in folder: SQLitePCLRaw.provider.sqlite3 And NOT this one: SQLitePCLRaw.provider.e_sqlite3

All is working good for me on iOS with these modifications (iPad, iPhone …) Tell me if it is good for you too!

[corytrese]

Note that linking the C source directly for iOS works on all LTS versions back to 2017 or before.

Using the IL2CPP plugin layer is also 100% identical between iOS/Android/Standalone, works on Nintendo Switch and lets you fully customize SQLite3 with things like encryption, compression and platform optimizations. Using the provided iOS SQLite isn't really something I'd want to do for a production game (in Unity or any other C++ engine.)

[ManuBera]

First of all, thank you so much for your quick replies, I'm really desperate with my project and I really appreciate this!

@Heged19 I've just tried it with Unity version 2022.1.20f1, but I get the same exception: "Unable to load DLL 'e_sqlite3'. Tried the load the following dynamic libraries: Unable to load dynamic library '/e_sqlite3' because of 'Failed to open the requested dynamic library (0x06000000) dlerror() = dlopen(/e_sqlite3, 5): image not found". Maybe I'm using the wrong dylib file? Can you remember what dylib you were using?

@corytrese I'm sorry, but I'm afraid this is beyond my skill level (I usually just use C# to script Unity). Could you maybe give me some more hints on how to achieve this? Or give me a nudge in the right direction? I'm a bit lost here :/

[Heged19]

Like I said in my previous message, you need to add the SQLitePCLRaw.provider.sqlite3.dll in your Unity project and init SQLite with the code SQLitePCL.raw.SetProvider(new SQLitePCL.SQLite3Provider_sqlite3()); instead of SQLitePCL.Batteries_V2.Init(); You do not have any e_sqlite in my instructions

[ManuBera]

I used the line of code you provided, but I can't remove SQLitePCLRaw.provider.e_sqlite3.dll because it is a dependency of Microsoft.Data.Sqlite.

But I just noticed that when trying to build my dll I get an error on using var connection = new SqliteConnection(_databaseConfig.Name);:

System.TypeInitializationException has been thrown: Method 'sqlite3_load_extension' in type 'SQLitePCL.SQLite3Provider_e_sqlite3' from assembly 'SQLitePCLRaw.provider.e_sqlite3, Version=2.0.6.1341, Culture=neutral, PublicKeyToken=9c301db686d0bd12' does not have an implementation.

I'm a bit confused as to why "e_sqlite3" is still involved, because I only use the line SQLitePCL.raw.SetProvider(new SQLitePCL.SQLite3Provider_sqlite3()); and nothing else.

[ericsink]

Try Microsoft.Data.Sqlite.Core instead of Microsoft.Data.Sqlite.

[ManuBera]

@ericsink Thank you so much! That solved the problem and I can build my Dll and execute without exception. My goal is to use it together with an ORM like EFCore or Dapper and now I get a PlatformNotSupportedException because iOS seems to have a problem with Dapper as well, which of course has nothing to do with my sqlite problem, but that means right now I can't quickly check if it now runs on iOS :/ I'm so frustrated with building to iOS....