[Serializer] when normalizing, order the properties in a way controllable by the user

[dkarlovi]

Description
Currently, when normalizing, the serializer seems to order the properties in the same way it discovers them (guessing, reflection-based?). If possible, ot would make sense to use the order as supplied by the user.

Example

With classes:

class A {
    public $a;
    public $b;
    public $c;
}

class B extends A {
    public $d;
    public $e;
}

with config:

<?xml version="1.0" ?>
<serializer>
    <class name="B">
        <attribute name="a"/>
        <attribute name="b"/>
        <attribute name="c"/>
        <attribute name="d"/>
        <attribute name="e"/>
    </class>
</serializer>

I would expect it to come out normalized in the exact order specified in the mapping.

Instead, it comes out as:

d:
e:
a:
b:
c:

[Einenlum]

@dkarlovi I never encountered a case where the order of the properties mattered, in a json, yaml or xml. Do you have an example?

[dkarlovi]

@Einenlum The order is for humans. If you have an API (say, built on API Platform), it makes sense for you to be specific in the order of the properties (for example, the more common ones on top, group similar properties together, etc).

Since the order exists and is not arbitrary anyway (it doesn't change between refreshes), it would make sense to allow the user to control it.

[Dranac]

If you serialize data as CSV you'll expect to be able to choose the order of the fields (at least an order you can expect or an explanation about the generated order).

In my case i have something like :

$serializer->serialize(
    $collectionOfData,
    'csv',
    [
        'attributes' => [
            'id',
            'name',
            'description',
            'year',
            'actors'       => [
                'type',
                'name',
            ],
            'location'     => [
                'address', 
                'city', 
                'country', 
                'zipCode',
            ],
            'sizing'       => [
                'length',
                'volume',
                'surface',
                'outflow',
                'other',
            ],
        ],
    ]
);

It will generate a csv like this :

id, name, description, location.address, location.zipCode, location.city, location.country, sizing.length, sizing.volume, sizing.surface, sizing.outflow, sizing.other, year

Why does sizingkeeps its order while location does not ? Why does field yeargoes at the end while id,name,description are at the right place ?

[maidmaid]

@Dranac here, a workaround to generate your csv with the ordered fields (described in the private orderproperty) :

namespace App\Serializer\Normalizer;

use App\Entity\Foo;
use Symfony\Component\Serializer\Normalizer\NormalizerInterface;
use Symfony\Component\Serializer\Normalizer\ObjectNormalizer;

class CsvFooNormalizer implements NormalizerInterface
{
    private $normalizer;

    private $order = ['a', 'b', 'c'];

    public function __construct(ObjectNormalizer $normalizer)
    {
        $this->normalizer = $normalizer;
    }

    public function normalize($objects, $format = null, array $context = array()): array
    {
        $data = [];
        foreach ($objects as $object) {
            $ordered = array_merge(array_flip($this->order), $this->normalizer->normalize($object, $format, $context));
        }

        return $data;
    }

    public function supportsNormalization($data, $format = null): bool
    {
        return is_array($data) && $data[0] instanceof Foo && 'csv' === $format;
    }
}

[dkarlovi]

@maidmaid creating a custom normalizer for each class you want to serialize just to manually set the order is not great, not to mention the performance implications this approach might have, doesn't seem like a good workaround, let alone what I'd consider a solution.

[brooksvb]

+1 this issue because I am in the exact situation: Wanting to reorder the properties for a CSV.

[HRvojeIT]

@dkarlovi I never encountered a case where the order of the properties mattered, in a json, yaml or xml. Do you have an example?

Amazon MWS Feed xml format!

[gomcodoctor]

Same issue here.. need to have specific order of properties in csv

[Surf-N-Code]

+1

Workaround: Change the order of your getters and setters within your entity. https://stackoverflow.com/a/48442956/9036543

[arjanfrans]

Our system has clients which require a fixed order of XML fields. They are using some funny XML parsers or configurations which rely on the order. Unfortunately we have to consider the order. It worked fine using the JMS parser before, which supports this feature. After upgrading to Symfony parser we had set the order of getters and setters right (as mentioned by @Surf-N-Code).

However, this leads to some complications when for example extending a class. If I want to have a different order for a field in my child class, I'd have to copy all the functions.

It would be nice to see a solution like the JMS has, an annotation which allows you to specify the order.

[dunglas]

We could use the order defined in XML files, but it will not work for annotations (as annotations use the reflection API under the hood, the order will be the same as currently).

Also, the order doesn't matter when serializing JSON documents (except for humans) because by the spec JSON objects are unordered.

The order matters however only if you use XML or CSV.

To be honest, I think that when the order matters and isn't the same as the one of the properties (or accessors) in the class, using a dedicated DTO matching the excepted output would be cleaner than introducing an "order" or a "weight" key on the annotation, but I'm not strongly opposed to adding such key anyway.

Regarding other config formats than annotations, I´le not sure if changing the order to match the one in the config file would be allowed by our BC promise.

[dkarlovi]

@dunglas we can always opt in. But also, since this order is not deliberate, it does seem like it wouldn't be part of it.

[dunglas]

Why isn't it deliberate? Having the same order than the one is the class looks rational to me. Opt-in will require to introduce an "order" attribute even in XML and YAML (which is fine, I guess).

[dkarlovi]

Why isn't it deliberate? Having the same order than the one is the class looks rational to me.

In case of using configuration it's definitely not expected (hence this issue) since you talk about the serialization in the configuration, not the class. In case of annotations, I agree with you.

Opt-in will require to introduce an "order" attribute even in XML and YAML (which is fine, I guess).

Not necessarily, I can see something like USE_CONFIGURATION_ORDER which would flip the switch, your configuration order is reflected as is, you don't need an "order" attribute there, I think. Same reasoning as with routing, the route order with annotations needs it, but not others.

[dkarlovi]

Alternatively, we can use

ORDER_BY: configuration|class

[maidmaid]

I faced again the same need more than 1 year after my first comment :smile:

This time, I solved it by creating a normalizer decorator handling sort. So, as any decorator, it can be easily configurated in the DI with your custom normalizers.

class SortedNormalizer implements NormalizerInterface
{
    private $decorated;
    private $sort;

    public function __construct(NormalizerInterface $decorated, array $sort)
    {
        $this->decorated = $decorated;
        $this->sort = array_flip($sort);
    }

    public function normalize($object, $format = null, array $context = []): array
    {
        $normalized = $this->decorated->normalize($object, $format, $context);

        uksort($normalized, function ($a, $b): int {
            return $this->sort[$a] ?? INF <=> $this->sort[$b] ?? INF;
        });

        return $normalized;
    }

    public function supportsNormalization($data, $format = null): bool
    {
        return $this->supportsNormalization($data, $format);
    }
}

Usage :

class BadlySorted
{
    public $c = 3;
    public $b = 2;
    public $a = 1;
}

$normalizer = new ObjectNormalizer();
$normalizer->normalize(new BadlySorted()); // returns C B A

$sort = ['a', 'b', 'c'];
$normalizer = new SortedNormalizer($normalizer, $sort);
$normalizer->normalize(new BadlySorted()); // returns A B C

[lajosthiel]

Workaround for Serializing to CSV

The change introduced in https://github.com/symfony/symfony/pull/24256 makes it possible to define the order of the fields when serializing to CSV by passing the 'csv_headers' option with a sequential array as value in which the data's keys are ordered in the desired sequence. The documentation (https://symfony.com/doc/5.2/components/serializer.html#the-csvencoder-context-options) is not very clear about this as it only states 'Sets the headers for the data' and one might expect it would set the labels for the header column and not determine the order of the header and content columns.

E.g.:

$this->serializer->serialize(
            [
                'c' => 3,
                'a' => 1,
                'b' => 2
            ],
            CsvEncoder::FORMAT,
            [
                CsvEncoder::HEADERS_KEY => ['a', 'b', 'c']
            ]);

returns

a,b,c
1,2,3

PR for clarifying the documentation: https://github.com/symfony/symfony-docs/pull/14609

[guilliamxavier]

One root cause of the issue is that PHP appends the parent properties after the child's ones (while e.g. C++ prepends the parent properties before the child's ones), which has bothered me in other contexts too 😠

For the Symfony serializer it seems that you can control the order by duplicating the parent properties and/or getters in the child (not a great solution...)

[guilliamxavier]

Update: It looks like PHP 8.1 is going to change its order: https://github.com/php/php-src/blob/578b67da49af51b2f796a48782e51ceb62860943/UPGRADING#L334-L341

Properties order used in foreach, var_dump(), serialize(), object comparison etc. was changed. Now properties are naturally ordered according to their declaration and inheritance. Properties declared in a base class are going to be before the child properties. This order is consistent with internal layout of properties in zend_object structure and repeats the order in default_properties_table[] and properties_info_table[]. The old order was not documented and was caused by class inheritance implementation details.

(https://github.com/php/php-src/commit/72c3ededed45fc8f2ce6f98d11f82adedc5e9763)

🙂

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[HRvojeIT]

Yes for shure!

Il giorno gio 21 apr 2022 alle ore 15:05 Carson: The Issue Bot < @.***> ha scritto:

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

— Reply to this email directly, view it on GitHub https://github.com/symfony/symfony/issues/27441#issuecomment-1105181865, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEWNSUNSN7C5FAPN6QJPJTVGFHALANCNFSM4FCS4TVA . You are receiving this because you commented.Message ID: @.***>

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[carsonbot]

Just a quick reminder to make a comment on this. If I don't hear anything I'll close this.

[gnutix]

Yes, this feature would still make sense.

[arjanfrans]

Still makes sense. I serialized some objects that get exported to external systems (which are not really that modern...) and require the XML data to be in a certain order.

[ageurts]

Still relevant indeed. Just implemented an API where the XML requires an order. Could circumvent the problem by changing the order of the properties, but when extending a class from another and adding properties to it, the order is all wrong.

[carsonbot]

Thank you for this suggestion. There has not been a lot of activity here for a while. Would you still like to see this feature?

[nesl247]

Yes.

[arjanfrans]

Yes.

[mrskhris]

Most definitely yes. XML is still being used, and often it's validated against DTDs that specify exact tag order.

[Hanmac]

most xml i work with, use <xsd:sequence> so the data needs in a specific order

that means i also can't move attributes into Traits because that would cause them to be discovered later and messed up the order

[rgpgf]

I'm writing a service that sends invoices to the equivalent of the IRS (government level) and I just caught on that they use xsd:sequence everywhere. An order declaration would be really useful.

[LukasGoTom]

it's been years. Imho "the order does not matter" is a bit indefensible

python used that argument for their dict structures... they eventually made them ordered too by default.

but I understand well that such a change would be a pain to implement.

[Crovitche-1623]

Anyone has a working workaround using attributes ?

[Hanmac]

My workaround, just redefining the protected attribute again.

Example, a Trait that is used in other classes:

trait MimeInfoTrait
{

    /**
     * @var Mime[]
     */
    #[SerializedPath("[MIME_INFO][MIME]")]
    protected array $mimes = [];
}

example where it is used:

class Product
{
    /**
     * @var PriceDetails[]
     */
    #[SerializedName("PRODUCT_PRICE_DETAILS")]
    protected array $priceDetails = [];

    use MimeInfoTrait;

    /**
     * @var Mime[]
     */
    #[SerializedPath("[MIME_INFO][MIME]")]
    protected array $mimes = [];

    #[SerializedName("USER_DEFINED_EXTENSIONS")]
    protected array $extensions = [];
}

Because I redefine the protected property $mimes again, the getProperties method returns them in the order: $priceDetails , $mimes, $extensions just like I want them for the sequence.