Serialization error using ClientValueObject in AddListItem using IncrementalDeploy

[netarrow]

Brief description

Generating a Intranet using Incremental Deployment, If I use AddListItem valorizing records using FieldValue containing type like FieldUrlValue, FieldUserValue or LookupFieldValue I receive the following error:

SPMeta2.Exceptions.SPMeta2ModelDeploymentException: There was an error while provisioning definition. Check ModelNode prop. ---> System.Runtime.Serialization.SerializationException: Type 'Microsoft.SharePoint.Client.FieldUrlValue' with data contract name 'FieldUrlValue:http://schemas.datacontract.org/2004/07/Microsoft.SharePoint.Client' is not expected. Consider using a DataContractResolver if you are using DataContractSerializer or add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to the serializer.

If I remove the incremental deployment everything work fine.

SPMeta2 model

We are inside a foreach loop which generate diferent customers Note: I am using spmeta-easy-deploy library [..] var customerModel = SPMeta2Model.NewWebModel(web => {

                web.AddList(customerIndex, list =>
                {

                    list.AddListItem(new ListItemDefinition()
                    {
                        Title = customer.Url,
                        Values = new List<FieldValue>()
                        {
                            new FieldValue() { FieldName = "Cliente", Value = new FieldUrlValue() { Description = customer.Name, Url = customer.Url } },
                            new FieldValue() { FieldName = "Title", Value = customer.Name },
                            [..]
                            new FieldValue() { FieldName = "Venditore", Value = FieldUserValue.FromUser(customer.Venditore.Mail) },
                            new FieldValue() { FieldName = "Area", Value = customer.AreaComm.ID },
                        }
                    });
                });
            });

            customerModel.SetIncrementalProvisionModelId($"Customer.{customer.Account}");
            customerModel .DeployModelTo(connInfo, true);

        }
    }

[..]

Potential Fix

Doing some search inside the source code I found that inside the class MD5HashCodeServiceBase, in this part: https://github.com/SubPointSolutions/spmeta2/blob/f54e29be0724aecc3ee90fbd7c58fdccc4f8eb82/SPMeta2/SPMeta2/Services/HashCodeServiceBase.cs#L70

Are added the know types for serialization iterating the Properties of the object to serialize, but maybe this implementation does not consider the different kind of object I can assign to the property Value (which is object type) inside the FieldValue entity:

new FieldValue() { FieldName = "Venditore", Value = FieldUserValue.FromUser(customer.Venditore.Mail)}

What do you think? Any suggestion or experience with this problem?

Thank you

SharePoint API

Which version of SharePoint runtime do you use?

• O365

SPMeta2 API

SPMeta2.Diagnostic.DiagnosticInfo SPMeta2FileVersion:[1.2.17191.0958] SPMeta2FileLocation:[C:\Users\matt\Source\Workspaces\xxxxx\Prod\1.0_salesItaly\xxxxxxx\bin\x64\Debug\SPMeta2.dll] Convert(p.IsCSOMDetected):[False] Convert(p.IsSSOMDetected):[False]

[avishnyakov]

Hey @netarrow, really appreciate for such a detailed explanation on the issues. That's a valid one, indeed. Assessing it right, give us a few days to evaluate a solution.

As for potential fixes and why this happens.

SPMeta2 has built-in serialization to JSON/XML capabilities. That's done with default .NET serializers. In turn, they require "known types" to serialize C# object.

Serialization API is exposed via SPMeta2Model class, with FromXXX/ToXXX methods to suport JSON/XML serialization. Down the road, it delegates serialization into specific services passing along "known types". By default, these would be SPMeta2 definition classes which meant to be in the model itself. However, for specific cases, we expose SPMeta2Model. RegisterKnownType() method to register a custom type which may not be in the model. That's how standard definition comes in, and that's how custom types could be registered to enable serialization capabilities with custom classed.

Hope that makes sense so far.

Now, incremental provision part. In this mode, SPMeta2 calculates hashes of every model tree node, then saves it via "storage provider" - to memory, file system or SharePoint. During this, we calculate hashed, default is md5, and we perform serialization to get it faster. You pointed this right, MD5HashCodeServiceBase does serialization without factoring in potential custom types which could be in the model. It should respect, expose custom type registration in some way so we would be able to register them and carry on. Essentially, misdesign from our side.

While we are looking into a solution to fix this, here is what can be done to keep you moving forward:

• custom class inherited from MD5HashCodeServiceBase
• override GetHashCode the way you need
• register into ServiceContainer ( https://github.com/SubPointSolutions/spmeta2/blob/master/SPMeta2/SPMeta2/ServiceContainer.cs) - I am following up on this

It may move you forward for a while until we fix it.

Another way around - avoid using custom types, switch to string based values for FieldValue, avoid SharePoint types, use string-based representations of the field values until we fix this.

I hope this provides more context, and you may also see something else we missed. Ultimately, we gonna fix this issue by exposin API for registering custom known types. That should be it.

[netarrow]

Hi @avishnyakov, thank you for the fast feedback.

I noticed that the Md5HasCodeServiceBase is hard referenced inside the DefaultIncrementalModelTreeTraverseService

 public DefaultIncrementalModelTreeTraverseService()
        {
            _hashService = new MD5HashCodeServiceBase();

            CurrentModelHash = new ModelHash();
            IgnoredModelNodes = new List<ModelNode>();

            DefaultDefinitionFullPathSeparator = "/";
            DefaultDefinitionIdentityKeySeparator = ";";

            DefaultPersistenceModelIdPrefix = "spmeta2.incremental_state";

            EnableCaching = true;
        }

So I cannot override type registration with my custom HashService.

I have seen you talked about string representation of the SharePoint type; in the specific case of the FieldUrlValue and FieldUserValue I was unable to find one. In particular FieldUrlValue I did not find how to specify separately Url and Description, for the FieldUserValue passing directly the email is not accepted as valid value.

Thank you

[avishnyakov]

With string representation of FieldUrlValue, we might look into reflector/ilspy to see what is the logic to form string value. Can't say from memory as well.

Yes, we saw hard references to Md5HasCodeServiceBase and inability to register custom types within MD5HashCodeServiceBase. That's why assessing codebase, looking for a broader solution.

Either way will be testing and including your example into current unit test/regression codebase to avoid this issue pop up in the future.

What's your timeframe for this fix? When do you need it?

[netarrow]

Hi, at the moment I forked and cloned SpMeta2 and using my version of SpMeta2 with inside the loosely coupled version of dependency between Md5HasCodeServiceBase and DefaultIncrementalModelTreeTraverseService

I applied inside the HasService the type registration and I am moving forward now without error in Incremental, and is very fast! :)

It would be usefull if is possible a release of the official Nuget Package with just the hot fix to decouple the two dependency https://github.com/netarrow/spmeta2/commit/3379b23cbb9eb7d0d9d08f815595c90353dce059, to keep clean the Solution without the external custom SPMeta2.dll.

In the next month me and my team will work a lot on intranet generation with SpMeta2, so I hope to contribute with other case scenario, feedback, issue and enhancement.

Thank you for support!

[avishnyakov]

Sweet, sounds like a plan. The change looks good, happy to hear that it unblocks you guys.

Incremental would be 2/5/10 times faster depending on the nature of the model. There are some edges cases but in general, it shines.

[avishnyakov]

@netarrow , going to make all references to MD5HashCodeServiceBase via ServiceContainer plus expose type registration methods.

That allows either adding custom types for serialization or complete service replacement. Don't see that we should invest more time into making default service work with custom value types; recursive walk with potential cycle detection, well, not at this very moment.

Testing implementation, fx should be up on Monday or over the weekend,

[SubPointSupport]

Completed, testing. Refactored base and default implementation for this service, replaced referencies across codebase to fetch the instance from service container as needed.

The API looks as following:

var hashService = ServiceContainer.Instance.GetService<HashCodeServiceBase>();

hashService.KnownTypes.Add(typeof(MyType));
hashService.RegisterKnownTypes(new []{ typeof(MyType) });

Alternatively, this service can be replaced with your own one within service container. Sourced can be found here:

https://github.com/SubPointSolutions/spmeta2/blob/dev/SPMeta2/SPMeta2/Services/HashCodeServiceBase.cs#L11

https://github.com/SubPointSolutions/spmeta2/blob/dev/SPMeta2/SPMeta2/Services/MD5HashCodeServiceBase.cs#L12

@netarrow, let us know how it all suits you.