[UNITY ISSUE] Database creation with Esent fail on Unity/mono

[liszto]

Whe I try to load a file by using the ESENT database system it always failed with this error :

Illegal index definition // Microsoft.Isam.Esent.Interop.EsentIndexInvalidDefException: Illegal index definition
  at Microsoft.Isam.Esent.Interop.Api.Check (System.Int32 err) [0x0000a] in D:\Project\Designer\Designer\Assets\Scripts\Bim-Library\Scripts\Xbim\Common\Esent.Interop\Api.cs:2972 
  at Microsoft.Isam.Esent.Interop.Api.JetCreateIndex (Microsoft.Isam.Esent.Interop.JET_SESID sesid, Microsoft.Isam.Esent.Interop.JET_TABLEID tableid, System.String indexName, Microsoft.Isam.Esent.Interop.CreateIndexGrbit grbit, System.String keyDescription, System.Int32 keyDescriptionLength, System.Int32 density) [0x00015] in D:\Project\Designer\Designer\Assets\Scripts\Bim-Library\Scripts\Xbim\Common\Esent.Interop\Api.cs:1219 
  at Xbim.IO.Esent.EsentEntityCursor.CreateTable (Microsoft.Isam.Esent.Interop.JET_SESID sesid, Microsoft.Isam.Esent.Interop.JET_DBID dbid) [0x000f3] in D:\Project\Designer\Designer\Assets\Scripts\Bim-Library\Scripts\Xbim\Common\Xbim.IO\Esent\EsentEntityCursor.cs:109 
  at Xbim.IO.Esent.PersistedEntityInstanceCache.CreateDatabase (System.String fileName) [0x0002f] in D:\Project\Designer\Designer\Assets\Scripts\Bim-Library\Scripts\Xbim\Common\Xbim.IO\Esent\PersistedEntityInstanceCache.cs:124 
UnityEngine.Debug:Log(Object)
Xbim.IO.Esent.PersistedEntityInstanceCache:CreateDatabase(String) (at Assets/Scripts/Bim-Library/Scripts/Xbim/Common/Xbim.IO/Esent/PersistedEntityInstanceCache.cs:130)
Xbim.IO.Esent.PersistedEntityInstanceCache:ImportStep(String, Stream, Int64, ReportProgressDelegate, Boolean, Boolean, Int32) (at Assets/Scripts/Bim-Library/Scripts/Xbim/Common/Xbim.IO/Esent/PersistedEntityInstanceCache.cs:743)
Xbim.IO.Esent.PersistedEntityInstanceCache:ImportStep(String, String, ReportProgressDelegate, Boolean, Boolean, Int32) (at Assets/Scripts/Bim-Library/Scripts/Xbim/Common/Xbim.IO/Esent/PersistedEntityInstanceCache.cs:736)
Xbim.IO.Esent.EsentModel:CreateFrom(String, String, ReportProgressDelegate, Boolean, Boolean, Nullable`1) (at Assets/Scripts/Bim-Library/Scripts/Xbim/Common/Xbim.IO/Esent/EsentModel.cs:446)
Xbim.Ifc.IfcStore:Open(String, XbimEditorCredentials, Nullable`1, ReportProgressDelegate, XbimDBAccess, Int32) (at Assets/Scripts/Bim-Library/Scripts/Xbim/Common/Xbim.Ifc/IfcStore.cs:313)

All is fine with the MemoryModel so far. I try on 2 files of 1Mo and 103Mo, both are fine on MemoryModel but failed with ESENT. Any idea concerning this issue ?

I think it will be something related to Unity once again but just in case I prefer to ask you, if it sounds familiar :/

[SteveLockley]

Do you have very long path names to the file you are creating as the esent db?

[liszto]

I don't think the path is too long but there is special character like "é" ignored by the Path.GetTempPath() so my final path is : C:\Users\AURLIE~1\AppData\Local\Temp\Xbim.4487f519-1f54-47fd-87f0-4c03b089f211

I removed the configmanager calls from the GetXbimTempDirectory() cause it didn't work on Unity Engine (it seems) so it will use the temp folder which is fine for us.

EDIT : I tried with this path : D:\Project\Designer\Designer\Xbim.faa1a987-10c1-4dd1-aa30-1100441e3c0c

Same issue, so I don't think it's path length related (sadly)

Just to be sure, I updated the EsentInterop.dll to the last update available on there github (February 2018) and the result is the same (that was just a test, I hadn't a lot of expectation on it).

I try to understand this error but there is no documentation on all the ESENT database system ? :(

I continue my investigation on this ESENT error, I debug step by step the database generation, it succeed by creating the first index encountered which is :

using (var transaction = new Transaction(sesid))
            {
                Api.JetCreateTable(sesid, dbid, ifcEntityTableName, 8, 100, out tableid);

                JET_COLUMNID columnid;
                //Add the primary key, Entity Label
                var columndef = new JET_COLUMNDEF { coltyp = JET_coltyp.Long, grbit = ColumndefGrbit.ColumnNotNULL };
                Api.JetAddColumn(sesid, tableid, colNameEntityLabel, columndef, null, 0, out columnid);

                // Identity of the type of the object : 16-bit integer looked up in IfcType Table
                columndef = new JET_COLUMNDEF { coltyp = JET_coltyp.Short, grbit = ColumndefGrbit.ColumnMaybeNull };
                Api.JetAddColumn(sesid, tableid, colNameIfcType, columndef, null, 0, out columnid);

                //The properties of the entity
                columndef = new JET_COLUMNDEF { coltyp = JET_coltyp.LongBinary, grbit = ColumndefGrbit.ColumnMaybeNull };
                //if(EsentVersion.SupportsWindows7Features) columndef.grbit |= Windows7Grbits.ColumnCompressed;
                Api.JetAddColumn(sesid, tableid, colNameEntityData, columndef, null, 0, out columnid);

                //Flag to say if this class is to be indexed by type
                columndef = new JET_COLUMNDEF { coltyp = JET_coltyp.Bit, grbit = ColumndefGrbit.None };
                Api.JetAddColumn(sesid, tableid, colNameIsIndexedClass, columndef, null, 0, out columnid);

                //Primary Key index
                var labelIndexDef = string.Format("+{0}\0\0", colNameEntityLabel);
                Api.JetCreateIndex(sesid, tableid, entityTableLabelIndex, CreateIndexGrbit.IndexPrimary, labelIndexDef, labelIndexDef.Length, 100);
                Api.JetCloseTable(sesid, tableid);
                transaction.Commit(CommitTransactionGrbit.LazyFlush);
            }

but it seems to fail on this one which the second to be indexed :

 //Now create a table for the indexed properties
            using (var transaction = new Transaction(sesid))
            {
                Api.JetCreateTable(sesid, dbid, ifcEntityIndexTableName, 8, 100, out tableid);
                JET_COLUMNID columnid;
                // Identity of the type of the object : 16-bit integer looked up in IfcType Table
                var columndef = new JET_COLUMNDEF { coltyp = JET_coltyp.Short, grbit = ColumndefGrbit.ColumnNotNULL };
                Api.JetAddColumn(sesid, tableid, colNameIfcType, columndef, null, 0, out columnid);
                // Name of the secondary key : for lookup by a property value of the object that is a foreign object
                columndef = new JET_COLUMNDEF { coltyp = JET_coltyp.Long, grbit = ColumndefGrbit.ColumnNotNULL };
                Api.JetAddColumn(sesid, tableid, colNameSecondaryKey, columndef, null, 0, out columnid);
                //Add the entity Label
                Api.JetAddColumn(sesid, tableid, colNameEntityLabel, columndef, null, 0, out columnid);

                //Add the primary key, Entity Type, Index label and Entity Label 
                var labelIndexDef = string.Format("+{0}\0{1}\0{2}\0\0", colNameIfcType, colNameSecondaryKey, colNameEntityLabel);
                Api.JetCreateIndex(sesid, tableid, entityTableLabelIndex, CreateIndexGrbit.IndexPrimary, labelIndexDef, labelIndexDef.Length, 100);
                Api.JetCloseTable(sesid, tableid);
                transaction.Commit(CommitTransactionGrbit.LazyFlush);
            }

The further that I can go is this line :

#if !MANAGEDESENT_ON_WSA // Not exposed in MSDK
[DllImport(EsentDll, CharSet = EsentCharSet, ExactSpelling = true)]
public static extern int JetCreateIndex(IntPtr sesid, IntPtr tableid, string szIndexName, uint grbit, string szKey, uint cbKey, uint lDensity);

I continue my investigation...

[liszto]

Ok I pushed further my test and I succeeded to at least make progress the IfcStore.Open() to the full progress bar (100%) and after it seems to freeze...

Basically I replaced JetCreateIndex with JetCreateIndex2 and this one avoid the Illegal Index Definition. For example these lines in EsentEntityCursor (and every other ...Cursor classes) :

Api.JetCreateIndex(sesid, tableid, entityTableLabelIndex, CreateIndexGrbit.IndexPrimary, labelIndexDef, labelIndexDef.Length, 100);

are replaced by :

JET_INDEXCREATE[] jET_INDEXCREATs = new JET_INDEXCREATE[1];
jET_INDEXCREATs[0] = new JET_INDEXCREATE();
jET_INDEXCREATs[0].grbit = CreateIndexGrbit.IndexPrimary;
jET_INDEXCREATs[0].szKey = labelIndexDef;
jET_INDEXCREATs[0].cbKey = labelIndexDef.Length;
jET_INDEXCREATs[0].szIndexName = entityTableLabelIndex;
jET_INDEXCREATs[0].ulDensity = 100;
Api.JetCreateIndex2(sesid, tableid, jET_INDEXCREATs, jET_INDEXCREATs.Length);

Now I will try to fix the final loading part that them to freeze, but if you have any idea of why the "JetCreateIndex2" is working and not "JetCreateIndex" it will be a pleasure to hear why ? 👍

Another thing that I noticed with ESENT vs Memory is the loading time :s in our Engine it takes 1min roughly for a 1.8Mo file vs 2sec for the memory model

EDIT : Quick edit, I found why it freezes at the end in fact the IfcStore.Open ends well but when I try to access to anything by using LINQ it freeze due to in fact a database generated with only empty field... So the JetCreateIndex2 works well cause no JET_Error but in fact it just generates empty field, column, etc... @SteveLockley I think it will be all my researches for today on this issue with ESENT with or without any progress depending on how I look my progress :/

[liszto]

Still on the database issue, I can't understand why the PersistedEntityInstanceCache is empty whereas when I look at the database file generated in temp seems to have all the informations needed. I know it's a binary one but I can see some clear information like the bim file used to generate it, property field etc...

I'm still using the CreateIndex2 cause CreateIndex still return Illegal Index Definition :/

Any ideas ?

[liszto]

Another update, if someone as information. Cause the value that create the exception are :

sesid : JET_SESID(0x5c5e09a0)
tableid : JET_TABLEID(0x612b3020)
indexName : EntByLabel
grbit : IndexPrimary
keyDescription : +IfcType\0SecondaryKey\0EntityLabel\0\0
keyDescriptionLength : 35
density : 100

Result from the JET_Err : Illegal index definition => Microsoft.Isam.Esent.Interop.EsentIndexInvalidDefException: Illegal index definition

There is multiple reason for this one : This screenshot come from this documentation page : https://msdn.microsoft.com/en-us/library/gg269324(v=exchg.10).aspx because I can't find any documentation error page for the "CreateIndex"....

and I can't figure out which one could be the problem. I try to edit some of them just to try to exclude some of them but nothing change anything :/

in the following code part :

//Now create a table for the indexed properties
using (var transaction = new Transaction(sesid))
{

[liszto]

Quick edit, I found at least one issue in the mono/unity engine with Marshalling, I don't know if this issue is related with my database creation trouble but it happens on the same line than my "JetCreateIndex" related issue in Unity that I created and validated by their QA team : https://issuetracker.unity3d.com/issues/strings-are-not-fully-read-when-slash-0-null-character-is-used

[dipendra210]

Thank you for your posting.

I'm trying to build EDBViewer on CentOS. I have troubleshoot to build ESENT.dll Let me know the version of mono, wine, ESENT, EsentInterop.dll.

Would you share video that building it step by step?

Regards.

[andyward]

My understanding is that liszto is building on Unity on Mono on Windows so unlikely to help on CentOS

Esent is native unmanaged DLL on all Windows machines, provided as standard by Microsoft. It's the same database tech used by Microsoft Exchange, which I'd guess you're interested in it for EDBViewer

I'm no great expert on wine or Linux/Windows interop, but I can't see how you'd get this working under CentOS.

EsentInterop.dll is just a set of P/invoke wrappers to marshal between the managed .net and unmanaged x86/x64 code. We use the latest 1.9.4 from https://www.nuget.org/packages/ManagedEsent/ and https://github.com/Microsoft/ManagedEsent

[dipendra210]

Andyward, Thank you for your posting.

• I was confused with Unity on Linux.
• I realized it is impossible to run ESENT.dll on Linux machine.
• My trouble is I don't know the EDB file structure. EDB file structure is private for last version.

[andyward]

My guess is that EDB is an Esent database file.

At this point you're probably better off finding an Exchange or Esent specific group on Stackoverflow

[andyward]

@liszto What's the latest on this. Have you been able to work around? Did v5 Xbim help?

[liszto]

I need to recheck this asap. I try to do this, this morning !

[liszto]

I tried to open an ifc file with a 100Mo size with the dll from xBim 5.0 it seems to work. I tried with a threshold ultra low something like 1 and opened any other ifc files and it seems to work too.

[andyward]

Great! Assuming you using the Xbim.Essentials package? Worth noting that depending on the environment, the 100MB threshold may be ignored in v5 - it only kicks in when XBIM.IO.Esent is available.

So just to verify this before we close, because IfcStore now dynamically uses different model strategies it might be worth forcing it to use the Esent model by adding IfcStore.ModelProviderFactory.UseEsentModelProvider(); when you start the program.

(Normally I'd recommend IfcStore.ModelProviderFactory.UseHeuristicModelProvider(); which provides the v4 behaviour and switching at 100MB)

[liszto]

I'm currently using the Xbim.Essentials package based on the last build (or almost he last cause I didn't check recently) I'm using 5.0.213 (not the 5.0.216)

I want to try but :

• I don't see in the methods UseEsentModelProvider nor UseHeuristicModelProvider but just Use or UseMemoryModelProvider it's because I don't have ESENT ?

Edit : It's normal I have remove the ESENT dll during my first setup of xBim 5. I'll add it again in the project and I test it again. My bad !

[andyward]

That'll be because Xbim.IO.Esent is not referenced. I suspect that's 'by design' as XBIM.IO.Esent is only compatible with net47 targets. It'll be skipped in netstandard based environments.

So I guess that's 'case closed' for now. It works, but you're using the MemoryModel for all operations under Unity (regardless of model size). Obviously that may have performance / resource issues in larger models, especially as you can't persist to an XBIM format

I think we've got another ticket open to look at other persistent stores for IModels: #59

[liszto]

Just to add a final note to this ticket : Using UseEsentModelProvider still trigger the issue even with last xBim revision which is logical cause the issue cas coming from ESENT himself with unity/mono environment.

Otherwise it works well :)

Thansk again @andyward !