DDC-2332: [UnitOfWork::doPersist()] The spl_object_hash() generate not unique hash!

[doctrinebot]

Jira issue originally created by user fchris82:

I created fixtures and some data was inserted many times without calling the Task entity PrePersist event listener.

I printed the used and generated hash and I saw a Proxies\*_CG_*\Asitly\ProjectManagementBundle\Entity\User hash equal a Task entity hash!

[flack]

So let's hunt your last one collision :-)?

I'll see what I can do, but this one is really annoying to narrow down, it only occurs if I run my full suite of 778 tests. Might take a while :-)

Can you try do make consistency checks as described in #3037 (comment)?

So if I understand correctly you want me to check if at any point during the runtime there are discrepancies between $entityIdentifiers, $entityStates and $identityMap, and whether or not something writes to entityMap?

[mpdude]

Hm, that’s probably not practical.

Does the last one trigger the exception from #10785 ?

[flack]

Yes, looks like (I have to apply #10791 as well, otherwise I get dozens of #10785 exceptions). The immediate context is

1: create/persist/detach entity A 2: create/persist entity B which has an association to entity A

On 1) I get the error from addToIdentityMap (which is called from executeInserts) on 2) I get the the OID collision before calling $em->persist

But like I said, it doesn't happen if I run just this test class in isolation, so before step 1 there are probably a couple more missing. It doesn't happen in a testcase proper, but in setUpBeforeClass, and I suspect some deletion in a tearDownAfterClass from another test class leaves some inconsistency or something.

EDIT: To be more clear, I actually changed the exception into a debug_print_backtrace, otherwise it would stop at 1)

[mpdude]

On 1) I get the error from addToIdentityMap (which is called from executeInserts)

That is while trying to persist A for the first time?

So, your entity manager is not clean (clear()ed) at that time when the test starts? Do you provide IDs for the A entity yourself, or are those database-provided IDs (auto increments)? If the latter is the case, do you truncate the database so that the same IDs get re-assigned, but at the same time, you do not clear the EM so that it still has other (previous test, older) entities with particular IDs in its identity map?

[mpdude]

Here is code that would use a TRUNCATE TABLE to make two entities use the same ID, which corrupts the internal data structures on 2.15 and causes the exception to be raised with the change from #10785.

This might well be something you're doing over the course of your tests, spread out over different test cases or phases.

<?php

declare(strict_types=1);

namespace Doctrine\Tests\ORM\Functional\Ticket;

use Doctrine\ORM\Mapping as ORM;
use Doctrine\Tests\OrmFunctionalTestCase;

class IdReassignmentAfterTruncateTest extends OrmFunctionalTestCase
{
    public function testTruncatingTablesToReassignIdsIsNotSoGood(): void
    {
        $this->createSchemaForModels(SomeEntity::class);

        $entity = new SomeEntity();
        $this->_em->persist($entity);
        $this->_em->flush();

        $this->_em->getConnection()->executeStatement('TRUNCATE TABLE some_table');

        $entity = new SomeEntity();
        $this->_em->persist($entity);
        $this->_em->flush();
    }
}

/**
 * @ORM\Entity
 * @ORM\Table(name="some_table")
 */
class SomeEntity
{
    /**
     * @ORM\Id
     * @ORM\GeneratedValue()
     * @ORM\Column(type="integer")
     *
     * @var int
     */
    private $id;
}

[flack]

That is while trying to persist A for the first time?

Yes, it's a fresh object

So, your entity manager is not clean (clear()ed) at that time when the test starts?

Nope. I mean I could clear it in tearDownAfterClass if I wanted to, and it does fix (or rather mitigate) the issue, but I kind of prefer running the entire suite with the same em state. Tends to find some more obscure issues.

Do you provide IDs for the A entity yourself, or are those database-provided IDs (auto increments)? If the latter is the case, do you truncate the database so that the same IDs get re-assigned, but at the same time, you do not clear the EM so that it still has other (previous test, older) entities with particular IDs in its identity map?

All IDs are auto increments, yes. There is no clearing of the em, the only cleanup between tests is to delete/detach everything that is not needed anymore.

[flack]

Here is code that would use a TRUNCATE TABLE to make two entities use the same ID, which corrupts the internal data structures on 2.15 and causes the exception to be raised with the change from #10785.

This might well be something you're doing over the course of your tests, spread out over different test cases or phases.

No, I don't think so. The testsuite DB (SQLite in memory) gets setup once in the suite's bootstrap code, and after, there are no structural changes, all DB operations are done through EntityManager, there are no executeStatement calls or anything else that bypasses ORM and goes directly to DBAL (but I'll try to doublecheck to be sure)

[mpdude]

At the time this exeception is raised, two object instances of the same class exist that compete for the same @Id value. You should be able to see which class and ID that is.

At that time, where does the second object come from? How has it been created, where did it get its ID?

And, can you find where the first object was created and where it comes from? Set a breakpoint with a condition at the place where the objects are added to the identity map and watch out for that ID – so you should find the origin of the first object as well.

How come both are using the same identifier values?

[flack]

It took a while, but I figured it out now: It turns out there was one test that constructs and entity from an XML string. The XML comes from a hardcoded test asset and it contains a value of 32 for an association field in the entity. My code instantiates a new entity and applies the values from the XML, for associations it does that by calling $entity->linkToClassB = $em->getReference($value). The entity is never persisted, but just calling getReference is enough to poison $identityMap. After that other tests run until the point where there's 32 entries in the ClassB table, and then the exception happens.

So basically it is the same situation as in the testcase in #843. You can argue that this is a user error, but (I was meaning to write that in #843 as well), I have to say that getReference is quite a footgun in that case, because it will happily give you a reference to whatever nonsense you call it with, and then cause random-looking problems way later in completely different places. At least #10785 makes it possible to spot that there is a problem, but the problem might only occur on e.g. some long-running job on a production server and can be next to impossible to reproduce in unit tests (esp. since I bet almost everyone resets state between tests)

[Matt-PMCT]

@mpdude I had originally responded that I did not get an exception, but I was mistaken... (messed something up in my original run...) Your code does indeed throw an error message for me.

[Matt-PMCT]

@mpdude diving a bit deeper my problem seems to start with a class I have that extends another class.

In this case I have a "Role" class and then an "ApiRole" class that extends the "Role" class. Your error says I am adding an entity of the class "App\Entity\Role" with an ID of 1, but I think I am trying add an ApiRole, not a Role...

Now the new exception complains because my ApiRole has an ID of 1, which would collide with my Role that has an ID of 1 (different tables in the database...)

However, most interesting to me is, that it was an ApiRole that was colliding with a completely different Object later in the process...

My original ApiRole was just this:

If I remove the "extends" and copy all the code from the Roles Object the exception and collisions disappear. I was trying to avoid having duplicate code for two objects that essentially are the same but I wanted to store in different places. I'm not very good with how to use extends, so I'll look at that, maybe I was doing something dumb...

I think this proves that your addition is working and providing good feedback to the users 😁

[mpdude]

@Matt-PMCT could you share the code for both classes? I only need to see the “head” where @Entity etc is written, and the properties with annotations for the id and discriminator columns.

[Matt-PMCT]

@Matt-PMCT could you share the code for both classes? I only need to see the “head” where @Entity etc is written, and the properties with annotations for the id and discriminator columns.

ApiRole:

namespace App\Entity\Api;

use App\Entity\Role;
use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity(repositoryClass="App\Repository\Api\ApiRoleRepository")
 */
class ApiRole extends Role
{
    private $spare;

}

Role:

namespace App\Entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 * @ORM\Table(name="roles")
 */
class Role
{
    /**
     * @ORM\Column(type="smallint", options={"unsigned":true})
     * @ORM\Id
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    private $id;
    /**
     * @ORM\Column(type="string")
     */
    private $name;
    /**
     * @ORM\Column(type="string")
     */
    private $displayName;
    /**
     * @ORM\Column(type="datetime")
     */
    protected $dateTimeStart;
    /**
     * @ORM\Column(type="datetime", nullable=true)
     */
    protected $dateTimeEnd;

[mpdude]

Thanks!

Checking with https://www.doctrine-project.org/projects/doctrine-orm/en/2.15/reference/inheritance-mapping.html#entity-inheritance, you’re using entity inheritance without declaring it. That means the ORM will assume instances of both classes share the ID space and no two instances use the same id. But, in fact, both get their ID from different table auto-increments and so collisions may occur.

I am afraid the best we could do is to find out why the ORM did not reject your configuration in the first place, and I’ll take a look at that.

[Matt-PMCT]

@mpdude thanks a ton, I had no idea I needed to do anything special. Your code changes in #10785 directly led me to at least identify the problem. Seems like a solid improvement that I hope gets implemented, great work, and thank you!

[mpdude]

@Matt-PMCT could you please confirm that the exception added in #10463 would have spotted your entity configuration as invalid?

It will be an exception in 3.0, and is a deprecation warning starting in 2.15.

[Matt-PMCT]

@mpdude confirmed #10463 does spot it

[mpdude]

@Matt-PMCT try using a mapped superclass to hold the commonalities of both your Role classes.

[mpdude]

To summarize, we have

• 10785,

• 10791 and

• 10797

as actionable items to follow up on, and no other reproducible examples, leads or hints for this issue here.

[mpdude]

With

• 10785,

• 10791 and

• 10797

being solved/merged/closed, I don't see what else we could do here to improve the situation. My suggestion is to close this issue.