XML namespaces in generated YAML metadata files for nested elements confuse Deserializer

holtkamp commented 9 years ago

When generating YAML metadata files to be used with JMS Serializer, it seems that for EACH element the XML namespace is defined in the resulting YAML files. Also for nested elements that already 'inherit' their namespace from the root namespace.

For nested elements this seems to confuse the JMS Serializer => the element is not properly detected and the property remains empty. By removing the XML namespace JMS Serializer detects the element again.

For example:

Ideal\Messages\AcquirerErrorRes:
    xml_root_name: AcquirerErrorRes
    xml_root_namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
    properties:
        version:
            expose: true
            access_type: public_method
            serialized_name: version
            accessor:
                getter: getVersion
                setter: setVersion
            xml_attribute: true
            type: string
        createDateTimestamp:
            expose: true
            access_type: public_method
            serialized_name: createDateTimestamp
            xml_element:
                namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
            accessor:
                getter: getCreateDateTimestamp
                setter: setCreateDateTimestamp
            type: Goetas\Xsd\XsdToPhp\XMLSchema\DateTime

The createDateTimestamp property will not be populated properly.

By removing the snippet

            xml_element:
                namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'

serialization works again and the createDateTimestamp property is properly populated with a DateTime object.

The used XSD is here: https://github.com/juriansluiman/SlmIdealPayment/blob/master/data/xsd/AcceptantAcquirer.xsd, can provide example XML instances if required

demonkoryu commented 9 years ago

This should be worked over, I'm stumbling into the same problem.

goetas commented 9 years ago

Can you provide an example XML too? Are you using my jms serializer fork, right?

kevinmlong commented 8 years ago

I'm also running into this issue...any suggestions for how to resolve this, without having to manually handle this?

goetas commented 8 years ago

Quoting myself

Can you provide an example XML too? Are you using my jms serializer fork, right?

kevinmlong commented 8 years ago

I can't provide the xsd as it's not public, but looks like @holtkamp provided a sample. I am using your fork of the jms serializer.

kevinmlong commented 8 years ago

By commenting out the xml-element:namespace, things begin to work, but this seems to have an adverse effect on nested children elements. I receive a PHP warning stating Node no longer exists and cites lines 233 and 251 of XMLDeserializationVisitor.php

holtkamp commented 8 years ago

Ok, also for my own administration an example for integrating with the iDEAL Payment system. Apologies for the long comment, but I prefer to be complete ;).

XSD2PHP Converting XSD to PHP objects to be populate using based on this XML Schema File:

php bin/xsd2php convert:php https://raw.githubusercontent.com/juriansluiman/SlmIdealPayment/master/data/xsd/AcceptantAcquirer.xsd --ns-map='http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1;Ideal/Messages/' --ns-map='http://www.w3.org/2000/09/xmldsig#;XmlDigitalSignature' --ns-dest='Ideal/Messages/;src/Ideal/Messages/' --ns-dest='XmlDigitalSignature/;src/W3/'

XSD2YML Converting XSD to YML metadata to be used by JMS Serializer using based on this XML Schema File:

php bin/xsd2php convert:jms-yaml https://raw.githubusercontent.com/juriansluiman/SlmIdealPayment/master/data/xsd/AcceptantAcquirer.xsd --ns-map='http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1;Ideal/Messages/' --ns-map='http://www.w3.org/2000/09/xmldsig#;XmlDigitalSignature' --ns-dest='Ideal/Messages/;yml/Ideal/Messages/' --ns-dest='XmlDigitalSignature/;yml/W3/'

Results in the YAML metadata that will be used by Serializer (note the XML namespace of the last element, this should be different than the default namespace, coming to that later):

Ideal\Messages\DirectoryReq:
    xml_root_name: DirectoryReq
    xml_root_namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
    properties:
        version:
            expose: true
            access_type: public_method
            serialized_name: version
            accessor:
                getter: getVersion
                setter: setVersion
            xml_attribute: true
            type: string
        createDateTimestamp:
            expose: true
            access_type: public_method
            serialized_name: createDateTimestamp
            xml_element:
                namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
            accessor:
                getter: getCreateDateTimestamp
                setter: setCreateDateTimestamp
            type: Goetas\Xsd\XsdToPhp\XMLSchema\DateTime
        merchant:
            expose: true
            access_type: public_method
            serialized_name: Merchant
            xml_element:
                namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
            accessor:
                getter: getMerchant
                setter: setMerchant
            type: Ideal\Messages\DirectoryReq\MerchantAType
        signature:
            expose: true
            access_type: public_method
            serialized_name: Signature
            xml_element:
                namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
            accessor:
                getter: getSignature
                setter: setSignature
            type: XmlDigitalSignature\Signature

XML2PHP No lets try to deserialize the following XML instance using JML Serializer:

<?xml version="1.0" encoding="utf-8"?>
<!--URL: https://idealtest.rabobank.nl/ideal/iDEALv3-->
<DirectoryReq xmlns="http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1" version="3.3.1"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1 https://raw.githubusercontent.com/juriansluiman/SlmIdealPayment/master/data/xsd/AcceptantAcquirer.xsd http://www.w3.org/2000/09/xmldsig# https://raw.githubusercontent.com/cloudregistry/EPP-Launch-Phase-Extension-Specification/master/xsd/xmldsig-core-schema.xsd">
    <createDateTimestamp>2013-06-20T18:54:36.000Z</createDateTimestamp>
    <Merchant>
        <merchantID>002093965</merchantID>
        <subID>0</subID>
    </Merchant>
    <Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
        <SignedInfo>
            <CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
            <SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256" />
            <Reference URI="">
                <Transforms>
                    <Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                </Transforms>
                <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256" />
                <DigestValue>0R00+jQyZlSLrTA+1gSmsH9vtbzbNbIIoWzh9SDJXGc=</DigestValue>
            </Reference>
        </SignedInfo>
        <SignatureValue>
            H5FBG+1ceWVginZdK1bT77g1HL0p8/OXJ1FTw2QWNBOzkKgg1q0L7OvbQTsW3i+RP1TVfvXr35X+C3KUQbZNrnm 59w6J765T+XMpNyPCU5MgN6aQTV1SqXQMFpDlsdxMxeDHamRwoKs+g9hga/mTUv5xEedkVhTZGuQ2aiUeAec=
        </SignatureValue>
        <KeyInfo>
            <KeyName>7D665C81ABBE1A7D0E525BFC171F04D276F07BF2</KeyName>
        </KeyInfo>
    </Signature>
</DirectoryReq>

With Serializer 0.16 this results in PHP object where only the XSD simpleTypes seem to be populated:

Ideal\Messages\DirectoryReq Object
(
    [version:Ideal\Messages\DirectoryReq:private] => 3.3.1
    [createDateTimestamp:Ideal\Messages\DirectoryReq:private] => 
    [merchant:Ideal\Messages\DirectoryReq:private] => 
    [signature:Ideal\Messages\DirectoryReq:private] => 
)

With Serializer 0.16 and removing the extra XML namespace:

xml_element:
                namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'

this results in a PHP object that also has the XSD complexTypes populated:

With the suggested Serializer "jms/serializer": "xsd2php-dev as 0.18.0" we got a apparently properly populated PHP object:

Ideal\Messages\DirectoryReq Object
(
    [version:Ideal\Messages\DirectoryReq:private] => 3.3.1
    [createDateTimestamp:Ideal\Messages\DirectoryReq:private] => DateTime Object
        (
            [date] => 2013-06-20 18:54:36.000000
            [timezone_type] => 2
            [timezone] => Z
        )

    [merchant:Ideal\Messages\DirectoryReq:private] => Ideal\Messages\DirectoryReq\MerchantAType Object
        (
            [merchantID:Ideal\Messages\DirectoryReq\MerchantAType:private] => 002093965
            [subID:Ideal\Messages\DirectoryReq\MerchantAType:private] => 0
        )

    [signature:Ideal\Messages\DirectoryReq:private] => 
)

Only, as we can see with "jms/serializer": "xsd2php-dev as 0.18.0" that at least the <merchant/> element is processed properly. But another issue arises. It seems the XML namespace of theelement is incorrect in the generated YAML, it should behttp://www.w3.org/2000/09/xmldsig#```, therefore that property is not detected and populated properly.

Changing that part fixes everything:


Ideal\Messages\DirectoryReq:
    xml_root_name: DirectoryReq
    xml_root_namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
    properties:
        version:
            expose: true
            access_type: public_method
            serialized_name: version
            accessor:
                getter: getVersion
                setter: setVersion
            xml_attribute: true
            type: string
        createDateTimestamp:
            expose: true
            access_type: public_method
            serialized_name: createDateTimestamp
            xml_element:
                namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
            accessor:
                getter: getCreateDateTimestamp
                setter: setCreateDateTimestamp
            type: Goetas\Xsd\XsdToPhp\XMLSchema\DateTime
        merchant:
            expose: true
            access_type: public_method
            serialized_name: Merchant
            xml_element:
                namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'
            accessor:
                getter: getMerchant
                setter: setMerchant
            type: Ideal\Messages\DirectoryReq\MerchantAType
        signature:
            expose: true
            access_type: public_method
            serialized_name: Signature
            xml_element:
                namespace: 'http://www.w3.org/2000/09/xmldsig#'
            accessor:
                getter: getSignature
                setter: setSignature
            type: XmlDigitalSignature\Signature

Resulting PHP:

Concluding, the workaround is to use the Goetas fork of JMS Serializer, we should try to get https://github.com/schmittjoh/serializer/pull/301 merged.

Regarding the generation of the YAML files, this might be a new issue? XML Schema Definitions that involve multiple namespaces are not recognized properly? Looking at this line in the XSD, we see that the element is referenced, maybe this is not supported yet by the XSD2YML operation?:

 <xs:element ref="ds:Signature"/>

PS: note that a CustomTypeHandler is required to properly deal with the XSD:SimpleType and XSD:AnySimpleType types in the XML Digital Signing schema, also described in the documentation

kevinmlong commented 8 years ago

So I'm having some issues with nested namespaces as well. The deserializer seems to be fine with an element like this:

<ns1:Root xmlns:n1="..." xmlns:ns2="..." xmlns:ns3="...">
     <blah>...</blah>
     <blah2>...</blah2>
     <blah3>...</blah3>
</ns1:Root>

but chokes on something like this:

<ns1:Root xmlns:n1="..." xmlns:ns2="..." xmlns:ns3="...">
     <ns2:blah>...</ns2:blah>
     <ns3:blah2>...</ns3:blah2>
     <ns2:blah3>...</ns2:blah3>
</ns1:Root>

When implementing the the fix mentioned above (remove xml_element:namespace) the first example works fine (it doesn't without this fix). However, the second example just leaves blank nodes like:

<Root>
     <blah/>
     <blah2/>
     <blah3/>
</Root>

and shows numerous errors

PHP Warning:  JMS\Serializer\XmlDeserializationVisitor::visitProperty(): Node no longer exists in .../vendor/jms/serializer/src/JMS/Serializer/XmlDeserializationVisitor.php on line 233
PHP Warning:  JMS\Serializer\XmlDeserializationVisitor::visitProperty(): Node no longer exists in .../vendor/jms/serializer/src/JMS/Serializer/XmlDeserializationVisitor.php on line 251

holtkamp commented 8 years ago

Just to rule things out, are you sure you "feed" the Serializer with the metadata for all involved namespaces? Otherwise the Serializer might not know how to deal with them...

 $serializerBuilder = SerializerBuilder::create();
        $serializerBuilder->setMetadataDirs(
            array(
                'Vendor1\MessageNamespace1' => APPLICATION_ROOT.'/yml/Vendor1/Messages',
                'Vendor2\MessageNamespace2' => APPLICATION_ROOT.'/yml/Vendor2/Messages',
            )
        );

kevinmlong commented 8 years ago

Was hoping I was doing something wrong there, but I definitely am seeding the serializer correctly.

$serializerBuilder = SerializerBuilder::create();
$serializerBuilder->addMetadataDir('src/jms/path1','Path1');
$serializerBuilder->addMetadataDir('src/jms/path1','Path1\sub-path2');
$serializerBuilder->addMetadataDir('src/jms/path2/sub-path1','Path2\sub-path1');
$serializerBuilder->addMetadataDir('src/jms/path2/sub-path2','Path2\sub-path2');
$serializerBuilder->addMetadataDir('src/jms/path2/sub-path3','Path2\sub-path3');
$serializerBuilder->addMetadataDir('src/jms/path3/sub-path1','Path3\sub-path1');

I was really hoping this project was going to help with creating the classes based on the XSD (which it seems it does fine), but the serialization aspect was very important, but seems to be broken, or I'm just not doing it right...

Unfortunately, I can't share the XSDs as they are not public, but in general, there are 5 namespaces and some elements contain elements from a different namespace as previously shown above...

kevinmlong commented 8 years ago

@holtkamp - maybe I am confused. where are you removing the:

xml_element:
     namespace: 'http://www.idealdesk.com/ideal/messages/mer-acq/3.3.1'

from in the yaml files?

kevinmlong commented 8 years ago

So...I looked again and figured out where you are doing that. It would seem that the fix includes not having xml_element:namespace in the yaml file if the element's namespace is the same as the parent? Can this be added into the library?

holtkamp commented 8 years ago

yes indeed, in the YAML files, it seems when generating these YAML metadata files, not "all" namespaces are recognized correctly. You can try to generate the YAML files and then manually check whether these xml_element:namespace fields have the proper value. I suspect something goes wrong there when multiple namespaces are involved / elements are referenced in the XML Schema file.

Maybe you can try to isolate the part of the XML Schema file that goes wrong, otherwise the author of this library is still searching for the needle in the haystack ;).

On 26 September 2015 at 22:03, kevinmlong notifications@github.com wrote:

So...I looked again and figured out where you are doing that. It would seem that the fix includes not having xml_element:namespace in the yaml file if the element's namespace is the same as the parent? Can this be added into the library?

— Reply to this email directly or view it on GitHub https://github.com/goetas/xsd2php/issues/73#issuecomment-143491296.

goetas commented 8 years ago

i spent some time trying to reproduce the bug... but to me currently it looks working. any change to submit a failing test case?

goetas / xsd2php

XML namespaces in generated YAML metadata files for nested elements confuse Deserializer #73