svcutil generation - XmlElement vs XmlArray

[mryandot]

I am trying to generate a WSDL client for Workday's Revenue Management service, specifically Get_Customer_Invoices (links to WSDL and XSD on the page).

The XSD indicates that the Get_Customer_Invoices_Response element should have (among other things) zero or more Response_Data elements of type Customer_Invoice_Response_DataType, each containing zero or more Customer_Invoice elements of type Customer_InvoiceType.

The XML that is returned by the services looks like:

...
<wd:Get_Customer_Invoices_Response xmlns:wd="urn:com.workday/bsvc" wd:version="v32.1">
...
<wd:Response_Data>
  <wd:Customer_Invoice>
    <wd:Customer_Invoice_Data>
      <wd:Customer_Invoice_ID>CUSTOMER_INVOICE-6-1</wd:Customer_Invoice_ID>
      <wd:Submit>0</wd:Submit>

The code generated for the Response_Data property on Get_Customer_Invoices_ResponseType is:

        [System.Xml.Serialization.XmlArrayAttribute(Order=5)]
        [System.Xml.Serialization.XmlArrayItemAttribute("Customer_Invoice", typeof(Customer_InvoiceType), IsNullable=false)]
        public Customer_InvoiceType[][] Response_Data
        { // ...

First of all, it experiences the issues described by #1274. After applying the workaround (make it typeof(Customer_InvoiceType[])) I get back a response with no errors, but also no response data. If I mock up a response and serialize it as-generated, I get:

<Get_Customer_Invoices_ResponseType xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Response_Data xmlns="urn:com.workday/bsvc">
    <Customer_Invoice>
      <Customer_InvoiceType>
        <Customer_Invoice_Data>
          <Customer_Invoice_ID>1</Customer_Invoice_ID>
...

Note there is now a new level, <Customer_InvoiceType>. I'm not finding a way to adjust the XmlArrayAttribute to not emit that level. Instead, it appears the fix is to:

• Add the missing Customer_Invoice_Response_DataType with a Customer_Invoice property of type Customer_InvoiceType[]
• Change the Response_Data property to type Customer_Invoice_Response_DataType[]
• Flag both properties with XmlElement instead of XmlArray and XmlArrayItem.

I'm assuming this is intended as an optimization (avoid the extra serialization class), but it ends up breaking the results. I was hoping just to create a single client library with the generated API classes, but the final Reference.cs with only the WSDL files for the modules we use is nearly 35 MB, so not really something I want to fix that many times by hand (there are 652 instances of this issue with Response_Data properties alone, and I'm guessing the other properties would have similar issues).

I also checked with xsd.exe and svcutil.exe; the same optimization is used by all three.

[StephenBonikowsky]

@mryandot Sorry for the delay in responding to your issue, we will get to it as soon as we can.

[dasetser]

@mryandot Thanks for providing enough information for us to investigate this. It looks like the XML returned from the service does not match the current wsdl, and the behavior of the tools look correct (other than the issue described in 1274).

As you noted, the difference is the Customer_InvoiceType level. This wrapping element is required because it has two children, and the data could be ambiguous to deserialize if you don't have it. If you look at the wsdl the definition of Customer_InvoiceType is:

<xsd:complexType name="Customer_InvoiceType">
    <xsd:sequence>
        <xsd:element name="Customer_Invoice_Reference" type="wd:Customer_InvoiceObjectType" minOccurs="0" />
        <xsd:element name="Customer_Invoice_Data" type="wd:Customer_Invoice_WWS_DataType" minOccurs="0" />
    </xsd:sequence>
</xsd:complexType>

Each Customer_InvoiceType object has two nullable children. The Customer_Invoice_Reference, and Customer_Invoice_Data. Here's a scenario where you run into problems if you don't have the wrapping element:

If you have two Customer_InvoiceType objects, one of which only has the Data element, and one of which only has the Reference element. Without the wrapping element it's serialized as:

<wd:Customer_Invoice> 
    <wd:Customer_Invoice_Reference>
...
    </wd:Customer_Invoice_Reference>
    <wd:Customer_Invoice_Data>
...
    </wd:Customer_Invoice_Data>
</wd:Customer_Invoice>

This is the same XML you would get if you were to serialize a single Customer_InvoiceType object that included both a reference and data. You need the wrapping element to show where one Customer_InvoiceType object and the next begins.

As far as working around this issue, it sounds like you already have one idea. Another idea to avoid having to make the same modification many times would be to download the wsdl, modify it to match what the service is expecting, and use the tool to generate a proxy from that modified version.

[mryandot]

I disagree this is a problem with the XSD. The XSD is perfectly valid and matches the XML generated. In general Workday defines their types as a sequence occurring exactly once with each element in the sequence occurring zero or more times. That means a response may contain data, data may contain invoices, invoices may contain a reference or data (or nothing), and so on. The top level you refer to is the response/message body.

Ignore the specifics of this case and look at the example in the NestingLevel documentation. Let's say I'm working with the national forestry service and they expect a forest to contain trees, trees to contain branches, and branches to contain leaves. The example given doesn't generate that, it generates a forest that contains a TreeArray which contains trees. That level is not necessary if you treat the Forest as an array of trees.

What I don't know is how to have the serializer do that, other than defining all of the intermediate levels and telling it to treat the arrays as repeated elements.

    public class Forest {[XmlElement] public Tree[] Trees { get; set; } }
    public class Tree {[XmlElement] public Branch[] Branches { get; set; } }
    public class Branch {[XmlElement("Leaf")] public string[] Leaves { get; set; } }

In the base example we just have a Forest class where my example requires Forest, Tree, and Branch to be defined, and this is why I'm guessing this is an over-optimization issue; the utility doesn't see the need to create those intermediate classes, but without them we get an extra element in the XML that shouldn't be there.

Back to the Workday example, how would I encode multiple invoices? The way the XSD describes:

<wd:Get_Customer_Invoices_Response xmlns:wd="urn:com.workday/bsvc" wd:version="v32.1">
  <wd:Response_Data>
    <wd:Customer_Invoice>
      <wd:Customer_Invoice_Reference>
        <!-- reference elements here -->
      </wd:Customer_Invoice_Reference>
    </wd:Customer_Invoice>
    <wd:Customer_Invoice>
      <wd:Customer_Invoice_Data>
        <!-- data elements here -->
      </wd:Customer_Invoice_Data>
    </wd:Customer_Invoice>
  </wd:Response_Data>
  <!-- Could be additional Response_Data elements here as well -->
</wd:Get_Customer_Invoices_Response>

While I get this problem may not be worth solving as it would either require changes to the XmlSerializer or a bunch of additional classes being defined, I still think it is a limitation of the tool and not a problem with the given XML, XSD, or WSDL.

[mconnew]

Hi @mryandot. I'm really sorry this was my mistake. I helped @dasetser with this issue a few days ago and it looks like I misread or misunderstood something at the time. What you want to do is possible with XmlSerializer, we just need to find the right magic sauce to make it happen. To try and get you a solution faster, I'm going to make a guess right now as to what I believe the solution is, but if it doesn't work let me know and I'll spend the time to create a fully tested solution for you.

[System.Xml.Serialization.XmlElementAttribute("Customer_Invoice")]
public Customer_InvoiceType[] Response_Data
{  // ...

According to the documentation here it says:

The field named Employees returns an array of Employee objects. In this case, the XmlElementAttribute specifies that the resulting XML will not be nested (which is the default behavior of items in an array).

The changes are to make it a single depth array and declare it to only have the XmlElementAttribute. This basically says "for each item in this array, emit an XmlElement with this name". Btw, you can replace the array with List<Customer_InvoiceType> if you want it more friendly than an array.

Please let me know if this works. I'm taking a few days off but I'll be checking my email to follow up on this issue if this doesn't solve it.

@dasetser, we should investigate the scope of a fix in dotnet-svcutil.

[mryandot]

If I'm reading you correctly, you're talking about what I have confirmed works. The issue as far as my convenience goes is that when it optimizes arrays, it leaves out required intermediate types, so to go from the current generated code (abbreviated):

public class Get_Customer_Invoices_ResponseType {
    [System.Xml.Serialization.XmlArrayAttribute(Order=5)]
    [System.Xml.Serialization.XmlArrayItemAttribute("Customer_Invoice", typeof(Customer_InvoiceType[]), IsNullable=false)]
    public Customer_InvoicesType[][] Response_Data  { get; set; }
}

to the working code:

public class Get_Customer_Invoices_ResponseType {
    [System.Xml.Serialization.XmlElementAttribute(Order=5)]
    public Customer_Invoice_Response_DataType[] Response_Data  { get; set; }
}

public class Customer_Invoice_Response_DataType {
    [System.Xml.Serialization.XmlElementAttribute(Order = 0)]
    public Customer_InvoiceType Customer_Invoice { get; set; }
}

...I also have to generate the Custer_Invoice_Response_DataType class, which is pretty much what you see there, so really not all that bad individually.

Unfortunately I have to do this across thousands of classes, and have to pull directly from the WSDL/XSD because I don't have the proper names for the intermediate types in the generated code. Which takes me back to hacking together a temporary cheap replacement for svcutil, if possible, until I can get a fix. I would really like to avoid that. 😒

[mconnew]

I just generated the classes and here are some snippets of the generated code a little cleaned and reformatted up to make reading easier:

// ...
        [System.Xml.Serialization.XmlArrayAttribute(Order=5)]
        [System.Xml.Serialization.XmlArrayItemAttribute("Customer_Invoice", typeof(Customer_InvoiceType), IsNullable=false)]
        public Customer_InvoiceType[][] Response_Data
        {
            get { return this.response_DataField; }
            set { this.response_DataField = value; }
        }
// ...

    [System.Xml.Serialization.XmlTypeAttribute(Namespace="urn:com.workday/bsvc")]
    public partial class Customer_InvoiceType
    {
        private Customer_InvoiceObjectType customer_Invoice_ReferenceField;
        private Customer_Invoice_WWS_DataType customer_Invoice_DataField;

        [System.Xml.Serialization.XmlElementAttribute(Order=0)]
        public Customer_InvoiceObjectType Customer_Invoice_Reference
        {
            get { return this.customer_Invoice_ReferenceField; }
            set { this.customer_Invoice_ReferenceField = value; }
        }

        [System.Xml.Serialization.XmlElementAttribute(Order=1)]
        public Customer_Invoice_WWS_DataType Customer_Invoice_Data
        {
            get { return this.customer_Invoice_DataField; }
            set { this.customer_Invoice_DataField = value; }
        }
    }

So all the classes that you need have been generated already. There are two changes needed. First, replace the two attributes on Response_Data with [System.Xml.Serialization.XmlElementAttribute(Order=5, "Customer_Invoice")]. Second, change the array from 2D to 1D. There's no adding of any new types needed. No new data types needed to be created.

[mryandot]

My test is just using a few customer invoices with IDs, but when I do what you suggest I lose the Request_Data element, which should be a child of Get_Customer_Invoices_ResponseType and wrap all of the Customer_Invoice elements. This is what's generated by my test:

<?xml version="1.0" encoding="IBM437"?>
<Get_Customer_Invoices_ResponseType xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Customer_Invoice xmlns="urn:com.workday/bsvc">
    <Customer_Invoice_Data>
      <Customer_Invoice_ID>1</Customer_Invoice_ID>
    </Customer_Invoice_Data>
  </Customer_Invoice>
  <Customer_Invoice xmlns="urn:com.workday/bsvc">
    <Customer_Invoice_Data>
      <Customer_Invoice_ID>2</Customer_Invoice_ID>
    </Customer_Invoice_Data>
  </Customer_Invoice>
  <Customer_Invoice xmlns="urn:com.workday/bsvc">
    <Customer_Invoice_Data>
      <Customer_Invoice_ID>3</Customer_Invoice_ID>
    </Customer_Invoice_Data>
  </Customer_Invoice>
</Get_Customer_Invoices_ResponseType>

Just to confirm, I changed the Response_Data element to:

        [System.Xml.Serialization.XmlElementAttribute("Customer_Invoice", Order = 5)]
        public Customer_InvoiceType[] Response_Data
// ...

[mconnew]

The xsd type wd:Get_Customer_Invoices_ResponseType is declared (with other children removed, cleaned up etc) as:

            <xsd:complexType name="Get_Customer_Invoices_ResponseType">
                <xsd:sequence>
                    <!-- Other elements removed for brevity -->
                    <xsd:element name="Response_Data" type="wd:Customer_Invoice_Response_DataType" minOccurs="0" maxOccurs="unbounded" />
                </xsd:sequence>
                <xsd:attribute ref="wd:version"/>
            </xsd:complexType>

So there is an array of Response_Data elements as maxOccurs is unbounded. The xsd type of Response_Data is Customer_Invoice_Response_DataType which is declared (again, cleaned up etc) as:

            <xsd:complexType name="Customer_Invoice_Response_DataType">
                <xsd:sequence>
                    <xsd:element name="Customer_Invoice" type="wd:Customer_InvoiceType" minOccurs="0" maxOccurs="unbounded" />
                </xsd:sequence>
            </xsd:complexType>

This type then contains an array of Customer_Invoice elements of type Customer_InvoiceType. So this is the root cause of the 2D array, because in the schema it's declared as there being an array of arrays. So the original generated data type was correct, it should be:

        public Customer_InvoiceType[][] Response_Data
        {
            get { return this.response_DataField; }
            set { this.response_DataField = value; }
        }

So the trick looks to be to use a different different XML element name for each of the array dimensions. The first dimension should create <Response_Data> and the second dimension should create <Customer_Invoice>. This is achieved with the NestingLevel property you mentioned earlier. So your code should look something like this:

        [System.Xml.Serialization.XmlArrayAttribute(Order=5)]
        [System.Xml.Serialization.XmlArrayItemAttribute("Response_Data", NestingLevel = 0, IsNullable = false)]
        [System.Xml.Serialization.XmlArrayItemAttribute("Customer_Invoice", NestingLevel = 1, IsNullable = false)]
        public Customer_InvoiceType[][] Response_Data
        {
            get
            {
                return this.response_DataField;
            }
            set
            {
                this.response_DataField = value;
            }
        }

Can you give that a try and see if it generates the correct XML?

[mconnew]

Are you sure there can be multiple Response_Data elements in the response? This looks suspiciously like it was intended to have a single response value but some setting somewhere was set in such a way as to generate the wsdl saying there can be multiple. If there's only ever one Response_Data element, then I would expect making a change in the WSDL to remove the maxOccurs attribute (or set it to 1) to generate what you want.

[mryandot]

I get two layers of Response_Data.

<?xml version="1.0" encoding="IBM437"?>
<Get_Customer_Invoices_ResponseType xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Response_Data xmlns="urn:com.workday/bsvc">
    <Response_Data>
      <Customer_Invoice>
        <Customer_Invoice_Data>
          <Customer_Invoice_ID>1</Customer_Invoice_ID>
        </Customer_Invoice_Data>
      </Customer_Invoice>
    </Response_Data>
  </Response_Data>
</Get_Customer_Invoices_ResponseType>

I'm not sure what was intended or what their implementation allows, so I pretty much have to go on the WSDL as a contract. If they might send me more than one I need to be ready to handle it. If it was just this element I'd work around it and move on; my problem is the pattern repeats throughout their API. How do I distinguish ones that shouldn't be repeated from ones that should?

[StephenBonikowsky]

@mconnew

[StephenBonikowsky]

It looks like this has the same behavior on full framework and will require a deeper dive to understand what is going on. We can't get to it for 3.0/3.1, adding it to the 5.0 release.

[udlose]

@mconnew @StephenBonikowsky

This bug still exists in .NET 5. Any idea when it will be fixed?

[wondertalik]

Probably i have related issue. Have xsd generated by dynamics 365. As you can see all fields of EmployeeCard have minOccurs=0. Example of response.

To generate wcf client i used both Visual Studio Conected Services and dotnet-svcutil.

    /// <remarks/>
        [System.Xml.Serialization.XmlElementAttribute(Order=0)]
        public string Key
        {
            get
            {
                return this.keyField;
            }
            set
            {
                this.keyField = value;
            }
        }

        /// <remarks/>
        [System.Xml.Serialization.XmlElementAttribute(Order=1)]
        public string No
        {
            get
            {
                return this.noField;
            }
            set
            {
                this.noField = value;
            }
        }

The problem is that the generated code is searching by order index, but in general some info cant be without any data. In this case dynamics 365 returned xml without some element and deserialized object is wrong tottaly. To solve this problem i had to change Order to ElementName.

        /// <remarks/>
        [System.Xml.Serialization.XmlElementAttribute(ElementName = "No")]
        public string No
        {
            get
            {
                return this.noField;
            }
            set
            {
                this.noField = value;
            }
        }

The problem that i have more than 100 services with problem like this and its imposible to do it manually for all definitions.

How can i say to generate with ElementName instead of Order attribute?

About versions.

  "ProviderId": "Microsoft.VisualStudio.ConnectedService.Wcf",
  "Version": "15.0.40203.910",
  "GettingStartedDocument": {
    "Uri": "https://go.microsoft.com/fwlink/?linkid=858517"
  },

dotnet-svcutil --help
Microsoft (R) WCF Service Model Proxy Generation Tool for .Net Core platform
[Microsoft.Tools.ServiceModel.Svcutil, Version 2.0.2]

dotnet --version
5.0.203