Dictionary<List<byte[]>, ICustomInterface> not serializing properly

[flycast]

I have been building my skills today. Thank you so much for this extensive package.

I have a dictionary that uses a key of List<byte[]> and a value of my custom Interface so: Dictionary<List<byte[]>, IDispenseEntity>

The interface is implemented by 7 or so classes that implement other interfaces as well.

I have successfully serialized a List of the keys and I have successfully serialized a list of the the values. My issue seems to be that when I try to serialize a dictionary<List<byte[]>,IDispenseEntity> I just get the keys without any values. No errors.

Three questions:

1. Do you have any advice for this scenario?

2. In my case I have classes that implement IDispenseEntity. I am keeping all these in the dictionary as the value IDispenseEntity. When I serialize these I want the properties that are not called out in the ICustomInterface to be serialized as well. Is there something I need to do specifically for this?

3. What happens in the case of a class that has static fields?

[Mike-E-angelo]

Haha yeah I have to say @flycast when I saw that you are using byte[] as keys/elements I get a little nervous. A dictionary with a list of elements as the key. If I understand correctly that would have to be keyed on reference rather than the equality? I am unclear how that would work. It's almost like you would be better converting that list of bytes to a string and use that as the key instead.

However, I am interested in seeing what's happening here. If you can provide a sample model set as you are having trouble with it, I can take a look into it for you. For the IDispenseEntity provide 2-3 implementations. The simpler the better. :)

Edit: Doh! I forgot to attend to your questions.

1. I would definitely consider using a string for the key of your dictionary. This is simpler to understand and to process. If you use a byte array I believe the equality used for the key is based on the actual reference to the byte array and not the human-readable value it represents.
2. When serializing interfaces we serialize it as an object and emit the class name as an exs:type attribute so we know how to read it back later. If this is not happening please do provide a simple example and I can further investigate for you.
3. Static fields are not considered as candidates. What you could do is wrap a static field within an instance property and assign via there, but this is not recommended. Static fields are exactly that and run the risk of threading/state issues. I try to stay away from them as much as possible due to the complications they introduce.

[flycast]

Yeah, it’s a code smell isn’t it? Thanks for mentioning it. I actually stubbed my toe on that issue a few weeks ago. This is an add in for Solidworks. SW uses an immutable array of bytes as their unique ID for entities in their software. I considered converting them to something else but in the end I just implemented IEqualityComparer:

        public bool Equals(IDispenseEntity x, IDispenseEntity y)
        {
            if (x.Key.Count != y.Key.Count) return false;
            for (int i = 0; i < Key.Count; i++)
            {
                if (!x.Key[i].SequenceEqual(y.Key[i])) return false;
            }
            return true;
        }

I’d be happy to send a stripped down version of some of these files, I just don’t want to post it publicly.

[flycast]

@Mike-E-angelo : I found the issue. I had a property in one of my entities that was a type:

public IMathTransform SolidworksMathTransform { get; set; }

That is a SolidWorks type. It was somehow causing a memory issue. Some of the time Visual Studio would shut down debugging because it noticed a memory access violation:

System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'

Solidworks was originally written in the early 1990's and I often have issues with COM Interop - something I don't understand except that is is part of the black magic of legacy code. SolidWorks is heavily maintained so I can kind of tell where old legacy code is left alone and newer technologies are being used.

I think what I need to do is carefully go through what I am serializing and exclude anything I really don't need.

I REALLY appreciate your help!

One last question:

When using the fluent way of excluding properties how would I exclude a property of IDispenseEntity:

var xmlSer = new ConfigurationContainer()
    .Type<Dictionary<List<byte[]>, IDispenseEntity>>()
    .Member(a => a)
    .Create();

I understand how when the type is a simple class, just not when I need to exclude a member of a value in a dictionary.

[create-issue-branch[bot]]

Branch issues/other/i565 created!

[Mike-E-angelo]

Cool glad to hear you got that figured out @flycast and good luck with those COM components. 😁

As for ignoring dictionary values, that is a tricky issue. You can ignore the value but what about the key? My recommendation is that if you can, serialize a list instead. Here's a sketch:

https://github.com/ExtendedXmlSerializer/home/blob/6b3a62e9240e1b0f8136ef3ceb0768b66548444f/test/ExtendedXmlSerializer.Tests.ReportedIssues/Issue565Tests.cs#L12-L62

In here, I've created a Model class that stores the key. This is beneficial as the key is only calculated once, whereas with a byte array, it has to be processed each and every time. The string is a calculated representation of that byte array, and serves as a stored pointer, so it should be much faster when any lookups are made.

Additionally, I've made use of the EmitWhen method to elide a particular type if needed. Then, once that is all read back in, I create a dictionary from it. So stored as a list, read/used as a dictionary. Let me know if you have any questions around this and I'll see if I can answer them.

[flycast]

I really appreciate the thoughts and advice. Are you concerned about the List<byte[]> performance in ExtendedXMLSerialization or just in general? Are there issues I am not catching on to beyond impliminting IEqualityComparer? I ask for my own learning.

[Mike-E-angelo]

Sure thing. The issue is not serialization but runtime performance. I could be mistaken but I am under the impression that a string is essentially a computed hash for a set of bytes. This hash is further stored as an integer which is its hashcode. So doing a lookup on that vs. doing a SequenceEquals should be much faster. Performance is a tricky thing however so I could be mistaken, but I am pretty sure that is how it works. :)

[flycast]

Yeah. Now that I am getting Serialize to work I am getting errors deserializing. "Same key already exists". I have not made changes to the key to make it a List as that is some deeper surgery that might have some unintended consequences. I was hoping to avoid that if possible.

Here is the error:

Exception thrown: 'System.ArgumentException' in mscorlib.dll
System.ArgumentException: An item with the same key has already been added.
   at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at Dispensing.Classes.EntityDatabase.Add(List`1 key, IDispenseEntity value)
   at ExtendedXmlSerializer.ReflectionModel.DictionaryAddDelegates.Add(IDictionary dictionary, DictionaryEntry entry)
   at ExtendedXmlSerializer.ReflectionModel.DictionaryAddDelegates.Add(Object dictionary, Object item)
   at ExtendedXmlSerializer.Core.ListAdapter`1.System.Collections.IList.Add(Object value)
   at ExtendedXmlSerializer.ContentModel.Collections.CollectionInnerContentHandler.IsSatisfiedBy(IInnerContent parameter)
   at ExtendedXmlSerializer.ContentModel.Content.InnerContentReader.Get(IFormatReader parameter)
   at ExtendedXmlSerializer.ContentModel.Members.Extensions.GetIfAssigned[T](IReader`1 this, IFormatReader reader)
   at ExtendedXmlSerializer.ExtensionModel.Xml.Read.Get(XmlReader parameter)
   at ExtendedXmlSerializer.ExtensionMethodsForSerialization.Deserialize[T](IExtendedXmlSerializer this, XmlReaderSettings settings, Stream stream)
   at Dispensing.DocumentHandler.HandleStorageReadAvailable(IXDocument doc)

Even though I wrote an equality comparer of my own to compare values rather than instances ExtendedXmlSerializer is using the default. I am guessing here but I think the byte[] is getting serialized and then unserialized in a way that causes the default equality method to think two are the same?

[create-issue-branch[bot]]

Branch issues/other/i565 created!

[Mike-E-angelo]

Ah indeed that's a tricky part as the Comparer property is a read-only property, so it is not considered as a candidate for content. Additionally, I believe we have custom activation for Dictionary objects.

I would have to see the model to assist further, but if you are able to specify your own dictionary you can do something like this:

https://github.com/ExtendedXmlSerializer/home/blob/cdf14979034f523267968a0446f5f3d07390863a/test/ExtendedXmlSerializer.Tests.ReportedIssues/Issue565Tests_Extended.cs#L11-L38

[flycast]

If it's not one thing it's another. Now I am getting incomplete deserialization. The stream is writing to the Solidworks file on file save. I also (for troubleshooting) serialize and write to a text file. When I load the SW file at a certain point an event fires that tells me I can read from the third party storage in the file. That s a stream. When I read I am only getting 6140 bytes and the xml appears to have an issue - deserialization does not complete. I am trying to determine for sure if the issue is:

Did my stream not completely write? Is my stream not completely reading?

First I need to determine if I have completely written to the file.

[Mike-E-angelo]

Yeah if you are running into streaming issues I would first start out with loading the entire string at once (basically not using streams until you get it working), and ensure it looks correct in memory. If so, load the string in via the Deserialize<T>(string content) extension method and see what happens.

[flycast]

I have the issue narrowed down I believe. When I was working with smaller datasets the complete xml was being saved. When it was deserialized it deserialized complete. Once I started getting larger datasets then the xml was truncated when saving. I know this because as I tried to save large datasets the final file size did not grow. When I started deserializing since the xml was truncated the xml deserialization would throw various errors attributable to the xml being truncated. The errors were sometimes the same and sometimes different so it was confusing.

I have started looking in the direction of the Solidworks stuff - perhaps there is a data size limit?

Thanks for all your help.

[Mike-E-angelo]

That's a good question... I am not aware of any limits with ExtendedXmlSerializer as it does use the native System.Xml.XmlReader/System.Xml.XmlWriter to do its reading/writing. I should also mention that in addition to the Deserialize extension method above there is a complementing public string Serialize extension method. I would recommend using that first before dealing with any streams as they do lead to problems as you are experiencing. Usually, this has to do with calling Flush and other oddities. That extension method accounts for all of that and provides the content as intended. Or should. 😁

[flycast]

@Mike-E-angelo - I have learned a lot more. I hope I can be clear here.

I continue to have issues with my serializing. I was on a quest to see what the Solidworks file that I am serializing to has inside it after I serialize and save that file. When I did this I noticed the stream length is much larger than the data I am expecting so I read the stream byte by byte and appended it to a string with some commas:

var textReader = new StreamReader(deserializeStream);
deserializeStream.Position = 0;
while (!textReader.EndOfStream)
{
    contents += textReader.Read() + ", ";
}

The string variable contents ends up huge. It holds 234,612 commas so about the same number of bytes. All of the string looks like my current and complete data is at the front and then there is either old data with a 65279 at the beginning or a bunch of zeros. When I tried to convert this to text using this converter I'd get a error:

Error: Error: Invalid UTF-8 detected

Narrowing that down I am finding the value 65279 appearing in numerous places in the string. Ultimately what I discovered is that the stream is a LOT longer than my data and that a 65279 seems to appear right before old data.

When I looked closer at the XML that I thought was being truncated I discovered that the XML is actually complete but there is garbage data after my closing tag. My good, valid data goes from character 1 - 14733 and then there is old data directly after that. Here is an example of what the serializer is deserializing:

Starts: <?xml version="1.0" encoding="utf-8"?><EntityDatabase><Item xmlns="https://ext

 //I clipped the data to save your eyes

Then around character 14700 close to the end:

<X>0.057737096938368315</X><Y>-0.0036400421496311855</Y><Z>0</Z></StartPoint></Value></Item></EntityDatabase>lserializer.github.io/v2" exs:item="unsignedByte">sDYAAAEAAAD/

// and then a lot more characters

Notice the ending tag </EntityDatabase>?

I am not sure how deserializing is supposed to work but what is happening is the data that the deserializer is getting is longer than the actual data. It seems like the deserializer is missing the ending XML tag. Plausible?

[Mike-E-angelo]

Hi @flycast I go back to using streams and how finicky they can be to use. Are you sure you are flushing them properly? Are you encountering this issue when you use the extension methods I mentioned earlier? If you are encountering this issue with those extension methods I would say there is a bug worth further exploring here. If you are not, then the problem is probably in how the streams are being handled.

[flycast]

@Mike-E-angelo - Thanks for getting back to me so quick.

I thought I was using the extension method:

var serializeStream = new ConfigurationContainer()
    .EnableImplicitTyping(typeof(EntityDatabase))
    .Create();

Debug.WriteLine($"Attempting to write {_db.Count} entities to {EntityStreamName} on save");
serializeStream.Serialize(str, _db);

I started having a weird error that looked like it was coming from xCad. I set Visual Studio to decompile code not my own. The error was actually coming from this code in ExtendedXmlSerializer.ExtensionModel.Xml.InstanceFormatter:

public string Get(object parameter) { Stream stream = _stream(); using (System.Xml.XmlWriter writer = _factory.Get(stream)) { _serializer.Serialize(writer, parameter); writer.Flush(); stream.Seek(0L, SeekOrigin.Begin); return new StreamReader(stream).ReadToEnd(); } }

Writer.Flush() is the one causing the exception. The exception is:

System.Runtime.InteropServices.COMException (0x80030005): Access Denied. (Exception from HRESULT: 0x80030005 (STG_E_ACCESSDENIED))
   at System.Runtime.InteropServices.ComTypes.IStream.Read(Byte[] pv, Int32 cb, IntPtr pcbRead)
   at Xarial.XCad.Toolkit.Data.ComStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadToEnd()
   at ExtendedXmlSerializer.ExtensionModel.Xml.InstanceFormatter.Get(Object parameter)
   at ExtendedXmlSerializer.ExtensionMethodsForSerialization.Serialize(IExtendedXmlSerializer this, Stream stream, Object instance)
   at Dispensing.DocumentHandler.HandleStorageWriteAvailable(IXDocument doc) in C:\Users\erics\source\repos\TestingEFClassLibrary\TestingEFClassLibrary\DocumentHandler.cs:line 251

[Mike-E-angelo]

No worries @flycast. My hunch is that you have something going on with _factory.Get(stream). The extension method does not use streams and handles all of that for you. You should be able to do something like:

var subject = new ConfigurationContainer().EnableImplicitTyping(typeof(EntityDatabase)).Create();
string content = subject.Serialize(instance);

If content has some funny business going on inside of it, then that means it's something in ExtendedXmlSerializer. Otherwise it could be how the stream is handled. Streams are funny things so let's see if you are able to still see the issue with the above.

[flycast]

OK @Mike-E-angelo .

At every step I confirmed what was being deserialized by deserializing to a string and examining the string. It looked exactly as expected with no garbage before or after the XML.

I have very carefully confirmed that the stream handling and storage in xCad with .Net XmlSerializer works by starting with serializing very simple string and working up to a test class that had a single string property. They all serialized and unserialized well. At every step I confirmed what was being deserialized by deserializing to a string and examining the string. It looked exactly as expected with no garbage before or after the XML.

Next I switched to a more complicated class and ExtendedXmlSerializer.

I added int, double and bool properties to the class (code below).

If it serializes and deserializes properly I get a small bit of text prepended before the opening tag:

<my opening xml tag>

If I serialize and deserialize just a single instance of TestSerialization class it serializes and deserializes fine. There is the prepended text though.

If I serialize and deserialize a small List with 5 instances it still serializes and deserializes just fine. IT still has the prepended test.

If I go to a large quantity of data then either the serialization or the deserialization quits working, not sure which.

What happens is the text that gets deserialized is now missing data at the front or the rear. Example of tring to perform the test on 1,000 elements. This is what I get:

stSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>975</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>976</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>977</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>978</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>979</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>980</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>981</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>982</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>983</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>984</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>985</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>986</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>987</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>988</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>989</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>990</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>991</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>992</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>993</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>994</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>995</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>996</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>997</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>998</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>999</MyInteger></ns1:TestSerialization></List>stSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>964</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>965</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>966</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>967</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>968</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>969</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>970</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>971</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>972</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>----------</MyString><MyBool>false</MyBool><MyDouble>-1.7976931348623157E+308</MyDouble><MyInteger>973</MyInteger></ns1:TestSerialization><ns1:TestSerialization><MyString>**********</MyString><MyBool>true</MyBool><MyDouble>1.7976931348623157E+308</MyDouble><MyInteger>974</MyInteger></ns1:TestSerialization><ns1:Te>/

I get about 6145 characters. This number is the limit I keep seeing over and over when I start serializing and deserializing lists over some size limit.

My serializing code:

                    using (Stream str = storage.TryOpenStream(EntityStreamName, true))
                    {

                        if (str is not null)
                        {
                            try
                            {
                                InventoryBuilder inventoryBuilder = new InventoryBuilder(_app);

                                List<TestSerialization> test = new List<TestSerialization>();
                                for (int i = 0; i < 500; i++)
                                {
                                    var newElement = new TestSerialization();
                                    if (i % 2 == 0)
                                    {
                                        newElement.MyString = "**********";
                                        newElement.MyDouble = Double.MaxValue;
                                        newElement.MyBool = true;
                                        newElement.MyInteger = i;
                                    }
                                    else
                                    {
                                        newElement.MyString = "----------";
                                        newElement.MyDouble = Double.MinValue;
                                        newElement.MyBool = false;
                                        newElement.MyInteger = i;
                                    }
                                    test.Add(newElement);
                                }

                                var serializer = new ConfigurationContainer()
                                    .EnableImplicitTyping(typeof(List<TestSerialization>))
                                    .Create();

                                serializer.Serialize(str, test);
                            }
                            catch (Exception ex)
                            {
                                Debug.WriteLine("Something bad happened when trying to write to the SW file.");
                                Debug.WriteLine(ex);
                            }

Test class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace Dispensing
{
    [Serializable]
    public class TestSerialization
    {
        public string MyString { get; set; }

        public bool MyBool { get; set; }

        public double MyDouble { get; set; }

        public int MyInteger { get; set; }
    }
}

I am still working with Solidworks to see if we can tell how much was written. I hold out zero hope though. Around 2015 they stopped storing their file format in plain text and went to some encrypted or compressed format. Also, Solidwoks has terrible support once you get past the value added reseller level to Solidworks itself, terrible.

[Mike-E-angelo]

Thank you for the additional context and information @flycast. I took your provided code and made a test for our suite which seems to pass just fine. You can see this here:

https://github.com/ExtendedXmlSerializer/home/blob/e7604b656b4955ed1819922e8b5da1eb4ee58d95/test/ExtendedXmlSerializer.Tests.ReportedIssues/Issue565Tests_Extended_II.cs#L13-L44

I upped the count to 2,000 elements just to be safe and those seem to cycle with a serialization/deserialization just fine.

So, I think it's safe to say that the problem you are experiencing is in the stream and whatever is occurring in storage.TryOpenStream(EntityStreamName, true). Again my recommendation would be to would try to use the extension methods outlined above first to see if you can get this working with that and then from there move into using the streams. It is unclear from your report, but have you been able to use the extension methods at all? I would be interested in hearing if you are able to use them in a way to get you sorted here.

[flycast]

@Mike-E-angelo - Thanks for your help as always.

I am writing third party data to SolidWorks files. SolidWorks has a very specific way they require you to do this. Solidworks was written starting in the early 90's so it's has been much, much easier to use the xCad add in - not only for the third party writing but for a lot of other reasons. I do see your point though - ExtendedXmlSerializer works without the complication of Solidworks. xCad works without the complication of ExtendedXmlSerializer. Solidworks prevents the user from examining their files.

I'm kind of stuck.

I have noticed that I can write a string to the SW file just fine. I think what I'll try is to serialize to a string and then write the string. On the read side read the string and then deserialize the xml string into the class.

Worst case scenario I'll write my data to a stand alone xml file. I really hate this idea because that pairs a file up with the Solidworks file and that sucks. At least I could persist the data I need though.

[Mike-E-angelo]

No problem @flycast I am mostly concerned obviously with discerning if there is a problem here with ExtendedXmlSerializer and if there is something that needs to be addressed on our end. As for the other integrations, I am afraid I cannot be of much assistance as I have never heard of them before until your post.

I will say that streams are tricky beasts. I actually stay away from them as much as possible they are so difficult to use and I always seem to encounter the same issues as you are describing (or worse). Usually, it has to do with proper disposing/flushing, but not always. You may also be simply experiencing a bug with one of the two integration points.

Have you tried reddit, perhaps? Sometimes when I have problems with a product I do a quick glance there to see if there's a community that can assist. Looks like there is one for SolidWorks here: https://www.reddit.com/r/SolidWorks/

xCad is more dubious. Looks like there's one for a cryptocurrency w/ the same name. But that SolidWorks sub looks promising, with 38k members and looks rather active. That might be a good starting point to reach out and see if someone with a better understanding of your scenario can provide more valuable feedback for you.

If you do manage to better define your issue and there is something I can do to further assist, please do let me know and I'll see if I can help you out. 👍

reddit

Learning community for all things SOLIDWORKS. • r/SolidWorks

ALL posts related to SOLIDWORKS are welcome. Share what you know. Learn what you don't. 100% Pirate Free Sub. Zero Tolerance

[flycast]

@Mike-E-angelo - I missed your last post. Thanks for the pointer to the Reddit SW community. Since we have legal licenses we use the official Dessault community for SolidWorks.

xCad can be found here.

Perhaps I should start another issue on this: I thought I had my issue all worked out by reading the stream as text and using the result to deserialize from. Everything seemed to be working properly but now I see that there is an issue with the List<byte[]>.

This gets a tad confusing. This is serializing/deserializing a dictionary and my dictionary key is a List<byte[]>. The dictionary key serializes/deserializes perfectly. Here is the XML for the dictionary key:

<Key>
    <Capacity>2</Capacity>
    <Array
        xmlns:exs="https://extendedxmlserializer.github.io/v2" exs:item="unsignedByte">AQIDBA==
    </Array>
    <Array
        xmlns:exs="https://extendedxmlserializer.github.io/v2" exs:item="unsignedByte">sDYAAAEAAAD//v8AAAAAABoAAAA=
    </Array>
</Key>

The object that is the dictionary value has a property that is also called Key. Property Key is also a List<byte[]> and happens to have the exact same values as the dictionary key. However, the XML looks different:

    <Key>
            <Capacity>2</Capacity>
            <Array exs:item="unsignedByte">AQIDBA==</Array>
            <Array exs:item="unsignedByte">sDYAAAEAAAD//v8AAAAAABoAAAA=</Array>
    </Key>

What I find strange is when the Object::Key deserializes back into the class instance the last entry in Key (Key[last]) ends up null. In my test data I have six entries in the dictionary. IN all sixe the Object::Key reads fine except for the last entry. In the last entry it is always null.

The obvious difference between the two xml is the second set does not have the xmlns:exs="https://extendedxmlserializer.github.io/v2" in it. I'm betting this is something in the config spec I am missing when I write it. Here is my current code for writing and reading:

Writing:

private void WriteToSolidWorksStream(IStorage storage, string streamName, object objToWrite)
{
    Type type = objToWrite.GetType();
    using (Stream str = storage.TryOpenStream(streamName, true))
    {
        Debug.WriteLine($"Attempting to write to: {streamName}");
        if (str is not null)
        {
            try
            {

                var serializer = new ConfigurationContainer()
                    .EnableImplicitTyping(type)
                    .Create();

                string content = serializer.Serialize(objToWrite);
                Debug.WriteLine($"Converted {content.Length} characters to write.");
                byte[] bytes = Encoding.ASCII.GetBytes(content);
                str.Write(bytes, 0, bytes.Length);
                Debug.WriteLine($"Wrote to {streamName} without exceptions.");
            }
            catch (Exception ex)
            {
                Debug.WriteLine("Something bad happened when trying to write to the SW file.");
                Debug.WriteLine(ex);
            }
        }
        else
        {
            Debug.WriteLine($"Failed to open stream {streamName} to write to.");
            throw new StreamDoesNotExistException();
        }
    }
}

Reading:

private T ReadFromSolidWorksStream<T>(IStorage storage, string streamName, string endingXMLTag)
{
    using var deserializeStream = storage.TryOpenStream(streamName, false);
    if (deserializeStream != null)
    {
        Debug.WriteLine($"Opened the stream {streamName} ok.");

        string characters = "";
        deserializeStream.Position = 0;
        byte[] buffer = new byte[deserializeStream.Length];
        int length = Convert.ToInt32(deserializeStream.Length);
        deserializeStream.Read(buffer, 0, length);
        characters = Encoding.UTF8.GetString(buffer);
        int index = characters.IndexOf(endingXMLTag);
        characters = characters.Substring(0, index + endingXMLTag.Length);
        Debug.WriteLine($"Retrieved {characters.Length} from the stream.");

        deserializeStream.Position = 0;
        var deserializer = new ConfigurationContainer()
            .EnableImplicitTyping(typeof(T))
            .Create();

        try
        {
            object obj = deserializer.Deserialize<T>(characters);
            Debug.WriteLine($"Deserialized the characters into the class {typeof(T)} ok.");
            return (T)obj;
        }
        catch (Exception ex)
        {
            Debug.WriteLine(ex);
            throw;
        }
    }
    else
    {
        Debug.WriteLine($"Failed to open stream {streamName} to read from.");
        throw new StreamDoesNotExistException($"{streamName} did not exist.");
    }
} 

[Mike-E-angelo]

Sure @flycast if you can provide some more specific code here with just the ExtendedXmlSerializer-based code intact and the SolidWorks code removed, along with an example model class that you are using that reproduces the error I can take a further look into it for you.

[Mike-E-angelo]

For reference @flycast I put together a simple test here based on your explanation and it cycles as expected with 6 items:

https://github.com/ExtendedXmlSerializer/home/blob/a4a8bf408867b1a29131d635f9c8079ea6b79b86/test/ExtendedXmlSerializer.Tests.ReportedIssues/Issue565Tests_Extended_III.cs#L13-L110

The only consideration is the List<byte[]> being used as a key and I have that resolving w/ a comparer as discussed earlier.

[flycast]

Some days weeks months I just want to give up programming. I created a minimum reproducible example and the problem did not happen. Went back to my project and had the issue. Out of sheer frustration I Restarted everything and the issue is gone. I'll close this and before I start another I will:

Reboot. CReate a minimum reproducible example to test just writing/reading to a text file

If after that I still have the issue then I'll reach out.

Thanks a ton for being so patient.

[Mike-E-angelo]

HA! The ol' when-in-doubt-reboot! I have not had to do that in a while, but seeing on how you are wrestling with code from a few decades ago I am not too surprised that did the trick. :)

No worries about the issues. If you are able to limit something down to showcase a potential problem I will certainly take a look. In any case, thank you for using ExtendedXmlSerializer. 👍