search

dmaicher / doctrine-test-bundle

Symfony bundle to isolate your app's doctrine database tests and improve the test performance
MIT License
1.08k stars 60 forks source link

Exception "May not alter the nested transaction with savepoints behavior while a transaction is open." #241

Closed nu111 closed 1 year ago

nu111 commented 1 year ago

Hi, I am on Symfony 5.4. I have followed the instructions to enable this bundle. During the installation process I allowed the recipe to run.

When I run my tests I get this error:

request.CRITICAL: Uncaught PHP Exception Doctrine\DBAL\ConnectionException: "May not alter the nested transaction with savepoints behavior while a transaction is open." at /var/www/html/vendor/doctrine/dbal/src/ConnectionException.php line 39

Any idea on what I am missing?

dmaicher commented 1 year ago

Interesting error :thinking: Can you show a stack trace for that exception?

nu111 commented 1 year ago 
[2023-02-04T13:53:30.152021+00:00] request.INFO: Matched route "app_user_list". {"route":"app_user_list","route_parameters":{"_route":"app_user_list","_controller":"App\\Controller\\UserController::list"},"request_uri":"http://localhost/api/v1.0/users/?limit=1","method":"GET"} []
[2023-02-04T13:53:30.178709+00:00] doctrine.INFO: Connecting with parameters array{"url":"<redacted>","dbname_suffix":"_test","driver":"mysqli","host":"<redacted>","port":3306,"user":"<redacted>","password":"<redacted>","driverOptions":[],"defaultTableOptions":[],"dama.keep_static":true,"dama.connection_name":"default","dbname":"<redacted>","charset":"utf8"} {"params":{"url":"<redacted>","dbname_suffix":"_test","driver":"mysqli","host":"<redacted>","port":3306,"user":"<redacted>","password":"<redacted>","driverOptions":[],"defaultTableOptions":[],"dama.keep_static":true,"dama.connection_name":"default","dbname":"<redacted>","charset":"utf8"}} []
[2023-02-04T13:53:30.182272+00:00] doctrine.DEBUG: Beginning transaction [] []
[2023-02-04T13:53:30.194672+00:00] request.CRITICAL: Uncaught PHP Exception Doctrine\DBAL\ConnectionException: "May not alter the nested transaction with savepoints behavior while a transaction is open." at /var/www/html/vendor/doctrine/dbal/src/ConnectionException.php line 39 {"exception":"[object] (Doctrine\\DBAL\\ConnectionException(code: 0): May not alter the nested transaction with savepoints behavior while a transaction is open. at /var/www/html/vendor/doctrine/dbal/src/ConnectionException.php:39)"} []
dmaicher commented 1 year ago

The full stack trace for the exception would be helpful

nu111 commented 1 year ago

Here it is! 

Testing App\Tests\CustomerTest
#0 /var/www/html/vendor/doctrine/dbal/src/Connection.php(1253): Doctrine\DBAL\ConnectionException::mayNotAlterNestedTransactionWithSavepointsInTransaction()
#1 /var/www/html/vendor/dama/doctrine-test-bundle/src/DAMA/DoctrineTestBundle/Doctrine/DBAL/StaticConnectionFactory.php(33): Doctrine\DBAL\Connection->setNestTransactionsWithSavepoints(true)
#2 /var/www/html/var/cache/test/Container48N8Zk2/getDoctrine_Dbal_DefaultConnectionService.php(22): DAMA\DoctrineTestBundle\Doctrine\DBAL\StaticConnectionFactory->createConnection(Array, Object(Doctrine\DBAL\Configuration), Object(Symfony\Bridge\Doctrine\ContainerAwareEventManager), Array)
#3 /var/www/html/var/cache/test/Container48N8Zk2/App_KernelTestDebugContainer.php(237): Container48N8Zk2\getDoctrine_Dbal_DefaultConnectionService::do(Object(Container48N8Zk2\App_KernelTestDebugContainer), true)
#4 /var/www/html/var/cache/test/Container48N8Zk2/getDoctrine_Orm_DefaultEntityManagerService.php(27): Container48N8Zk2\App_KernelTestDebugContainer->load('getDoctrine_Dba...')
#5 /var/www/html/var/cache/test/Container48N8Zk2/App_KernelTestDebugContainer.php(237): Container48N8Zk2\getDoctrine_Orm_DefaultEntityManagerService::do(Object(Container48N8Zk2\App_KernelTestDebugContainer), true)
#6 /var/www/html/var/cache/test/Container48N8Zk2/getDoctrine_Orm_DefaultEntityManager_ValidatorLoaderService.php(24): Container48N8Zk2\App_KernelTestDebugContainer->load('getDoctrine_Orm...')
#7 /var/www/html/var/cache/test/Container48N8Zk2/App_KernelTestDebugContainer.php(237): Container48N8Zk2\getDoctrine_Orm_DefaultEntityManager_ValidatorLoaderService::do(Object(Container48N8Zk2\App_KernelTestDebugContainer), true)
#8 /var/www/html/var/cache/test/Container48N8Zk2/getValidator_BuilderService.php(33): Container48N8Zk2\App_KernelTestDebugContainer->load('getDoctrine_Orm...')
#9 /var/www/html/var/cache/test/Container48N8Zk2/App_KernelTestDebugContainer.php(237): Container48N8Zk2\getValidator_BuilderService::do(Object(Container48N8Zk2\App_KernelTestDebugContainer), true)
#10 /var/www/html/var/cache/test/Container48N8Zk2/getValidator_Mapping_CacheWarmerService.php(24): Container48N8Zk2\App_KernelTestDebugContainer->load('getValidator_Bu...')
#11 /var/www/html/var/cache/test/Container48N8Zk2/App_KernelTestDebugContainer.php(237): Container48N8Zk2\getValidator_Mapping_CacheWarmerService::do(Object(Container48N8Zk2\App_KernelTestDebugContainer), true)
#12 /var/www/html/var/cache/test/Container48N8Zk2/getCacheWarmerService.php(28): Container48N8Zk2\App_KernelTestDebugContainer->load('getValidator_Ma...')
#13 /var/www/html/vendor/symfony/http-kernel/CacheWarmer/CacheWarmerAggregate.php(91): Container48N8Zk2\getCacheWarmerService::Container48N8Zk2\{closure}()
#14 /var/www/html/vendor/symfony/http-kernel/Kernel.php(585): Symfony\Component\HttpKernel\CacheWarmer\CacheWarmerAggregate->warmUp('/var/www/html/v...')
#15 /var/www/html/vendor/symfony/http-kernel/Kernel.php(787): Symfony\Component\HttpKernel\Kernel->initializeContainer()
#16 /var/www/html/vendor/symfony/http-kernel/Kernel.php(128): Symfony\Component\HttpKernel\Kernel->preBoot()
#17 /var/www/html/vendor/symfony/framework-bundle/Test/KernelTestCase.php(82): Symfony\Component\HttpKernel\Kernel->boot()
#18 /var/www/html/vendor/symfony/framework-bundle/Test/WebTestCase.php(46): Symfony\Bundle\FrameworkBundle\Test\KernelTestCase::bootKernel(Array)
#19 /var/www/html/tests/CustomerTest.php(78): Symfony\Bundle\FrameworkBundle\Test\WebTestCase::createClient()
#20 /var/www/html/vendor/phpunit/phpunit/src/Framework/TestCase.php(1608): App\Tests\CustomerTest->testOneFormat()
#21 /var/www/html/vendor/phpunit/phpunit/src/Framework/TestCase.php(1214): PHPUnit\Framework\TestCase->runTest()
#22 /var/www/html/vendor/phpunit/phpunit/src/Framework/TestResult.php(728): PHPUnit\Framework\TestCase->runBare()
#23 /var/www/html/vendor/phpunit/phpunit/src/Framework/TestCase.php(964): PHPUnit\Framework\TestResult->run(Object(App\Tests\CustomerTest))
#24 /var/www/html/vendor/phpunit/phpunit/src/Framework/TestSuite.php(684): PHPUnit\Framework\TestCase->run(Object(PHPUnit\Framework\TestResult))
#25 /var/www/html/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(653): PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
#26 /var/www/html/vendor/phpunit/phpunit/src/TextUI/Command.php(144): PHPUnit\TextUI\TestRunner->run(Object(PHPUnit\Framework\TestSuite), Array, Array, true)
#27 /var/www/html/vendor/phpunit/phpunit/src/TextUI/Command.php(97): PHPUnit\TextUI\Command->run(Array, true)
#28 /var/www/html/bin/phpunit(11): PHPUnit\TextUI\Command::main()
#29 {main}
dmaicher commented 1 year ago

Interesting. So it seems a transaction is already started before this is called:

https://github.com/dmaicher/doctrine-test-bundle/blob/master/src/DAMA/DoctrineTestBundle/Doctrine/DBAL/StaticConnectionFactory.php#L33

Could you try to debug what starts the transaction on that connection instance?

dmaicher commented 1 year ago

@nu111 also it would be interesting if you could test https://github.com/dmaicher/doctrine-test-bundle/pull/232 (using composer req dama/doctrine-test-bundle "dev-remove_post_connect_event as 7.99" --dev for example). There the whole flow changed a bit to prepare being compatible with Doctrine DBAL 4

nu111 commented 1 year ago

@dmaicher with #232 composer req dama/doctrine-test-bundle "dev-remove_post_connect_event as 7.99" --dev seems to work fine.

This is what mariadb general_log logged

230208 10:13:16     13 Connect <redacted> as anonymous on <redacted>
                    13 Query    SET NAMES utf8
                    13 Query    START TRANSACTION
                    13 Query    SAVEPOINT DAMA_TEST
                    13 Prepare  SELECT <redacted> FROM tblUser t0 WHERE t0.id = 1
                    13 Close stmt       
                    13 Prepare  SELECT <redacted> FROM tblUser t0 WHERE t0.email = ? LIMIT 2
                    13 Execute  SELECT <redacted> FROM tblUser t0 WHERE t0.email = 'testing@example.com' LIMIT 2
                    13 Close stmt       
                    13 Prepare  SELECT <redacted> FROM tblUser t0 WHERE t0.email = ? LIMIT 2
                    13 Close stmt       
230208 10:13:17     13 Prepare  UPDATE tblUser SET <redacted> WHERE id = ?
                    13 Execute  UPDATE tblUser SET <redacted> WHERE id = 1
                    13 Close stmt 
                    13 Query    ROLLBACK TO SAVEPOINT DAMA_TEST
                    13 Query    ROLLBACK
                    13 Quit
dmaicher commented 1 year ago

Ok interesting. The release of the approach in https://github.com/dmaicher/doctrine-test-bundle/pull/232 will take a bit longer though as I want to be sure it works with the final release of DBAL 4.0.

Using the latest stable release of this bundle: Could you try to find out what starts the transaction before this happens?

https://github.com/dmaicher/doctrine-test-bundle/blob/master/src/DAMA/DoctrineTestBundle/Doctrine/DBAL/StaticConnectionFactory.php#L33

dmaicher commented 1 year ago

Closing for now. Let me know whenever you have any more details on this issue.

Chaxwell commented 1 year ago

Giving a follow-up to this issue. I get the same error, and to provide an answer to your previous question there is the callstack in my application.

1) Tests\Controller\UniqueDocument\RiskGravityControllerRouteTest::testIndex
Doctrine\DBAL\ConnectionException: May not alter the nested transaction with savepoints behavior while a transaction is open.

/home/chaxwell/public_html/symfony/project/vendor/doctrine/dbal/src/ConnectionException.php:29
/home/chaxwell/public_html/symfony/project/vendor/doctrine/dbal/src/Connection.php:1306
/home/chaxwell/public_html/symfony/project/vendor/dama/doctrine-test-bundle/src/DAMA/DoctrineTestBundle/Doctrine/DBAL/StaticConnectionFactory.php:33
/home/chaxwell/public_html/symfony/project/var/cache/test/ContainerGvzHmd9/App_KernelTestContainer.php:698
/home/chaxwell/public_html/symfony/project/var/cache/test/ContainerGvzHmd9/App_KernelTestContainer.php:731
/home/chaxwell/public_html/symfony/project/vendor/symfony/var-exporter/Internal/LazyObjectState.php:97
/home/chaxwell/public_html/symfony/project/vendor/symfony/var-exporter/LazyGhostTrait.php:176
/home/chaxwell/public_html/symfony/project/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:207
/home/chaxwell/public_html/symfony/project/vendor/doctrine/doctrine-bundle/CacheWarmer/DoctrineMetadataCacheWarmer.php:42
/home/chaxwell/public_html/symfony/project/vendor/symfony/framework-bundle/CacheWarmer/AbstractPhpFileCacheWarmer.php:46
/home/chaxwell/public_html/symfony/project/vendor/symfony/http-kernel/CacheWarmer/CacheWarmerAggregate.php:96
/home/chaxwell/public_html/symfony/project/vendor/symfony/http-kernel/Kernel.php:526
/home/chaxwell/public_html/symfony/project/vendor/symfony/http-kernel/Kernel.php:709
/home/chaxwell/public_html/symfony/project/vendor/symfony/http-kernel/Kernel.php:122
/home/chaxwell/public_html/symfony/project/vendor/symfony/framework-bundle/Test/KernelTestCase.php:73
/home/chaxwell/public_html/symfony/project/vendor/symfony/framework-bundle/Test/WebTestCase.php:44
/home/chaxwell/public_html/symfony/project/tests/WebTestCase.php:28
/home/chaxwell/public_html/symfony/project/tests/Controller/UniqueDocument/RiskGravityControllerRouteTest.php:20
/home/chaxwell/public_html/symfony/project/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
/home/chaxwell/public_html/symfony/project/vendor/phpunit/phpunit/src/Framework/TestSuite.php:684
/home/chaxwell/public_html/symfony/project/vendor/phpunit/phpunit/src/Framework/TestSuite.php:684
/home/chaxwell/public_html/symfony/project/vendor/phpunit/phpunit/src/Framework/TestSuite.php:684
/home/chaxwell/public_html/symfony/project/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:651

Here's t he stack for the first (and only) call to beginTransaction

Doctrine\DBAL\Connection->beginTransaction (/home/chaxwell/public_html/symfony/project/vendor/doctrine/dbal/src/Connection.php:1343)
DAMA\DoctrineTestBundle\Doctrine\DBAL\PostConnectEventListener->postConnect (/home/chaxwell/public_html/symfony/project/vendor/dama/doctrine-test-bundle/src/DAMA/DoctrineTestBundle/Doctrine/DBAL/PostConnectEventListener.php:26)
Symfony\Bridge\Doctrine\ContainerAwareEventManager->dispatchEvent (/home/chaxwell/public_html/symfony/project/vendor/symfony/doctrine-bridge/ContainerAwareEventManager.php:63)
Doctrine\DBAL\Connection->connect (/home/chaxwell/public_html/symfony/project/vendor/doctrine/dbal/src/Connection.php:393)
Doctrine\DBAL\Connection->getDatabasePlatformVersion (/home/chaxwell/public_html/symfony/project/vendor/doctrine/dbal/src/Connection.php:446)
Doctrine\DBAL\Connection->detectDatabasePlatform (/home/chaxwell/public_html/symfony/project/vendor/doctrine/dbal/src/Connection.php:408)
Doctrine\DBAL\Connection->getDatabasePlatform (/home/chaxwell/public_html/symfony/project/vendor/doctrine/dbal/src/Connection.php:316)
Doctrine\Bundle\DoctrineBundle\ConnectionFactory->getDatabasePlatform (/home/chaxwell/public_html/symfony/project/vendor/doctrine/doctrine-bundle/ConnectionFactory.php:131)
Doctrine\Bundle\DoctrineBundle\ConnectionFactory->createConnection (/home/chaxwell/public_html/symfony/project/vendor/doctrine/doctrine-bundle/ConnectionFactory.php:109)
DAMA\DoctrineTestBundle\Doctrine\DBAL\StaticConnectionFactory->createConnection (/home/chaxwell/public_html/symfony/project/vendor/dama/doctrine-test-bundle/src/DAMA/DoctrineTestBundle/Doctrine/DBAL/StaticConnectionFactory.php:25)
ContainerGvzHmd9\App_KernelTestContainer->getDoctrine_Dbal_DefaultConnectionService (/home/chaxwell/public_html/symfony/project/var/cache/test/ContainerGvzHmd9/App_KernelTestContainer.php:698)
ContainerGvzHmd9\App_KernelTestContainer->getDoctrine_Orm_DefaultEntityManagerService (/home/chaxwell/public_html/symfony/project/var/cache/test/ContainerGvzHmd9/App_KernelTestContainer.php:731)
Symfony\Component\VarExporter\Internal\LazyObjectState->initialize (/home/chaxwell/public_html/symfony/project/vendor/symfony/var-exporter/Internal/LazyObjectState.php:97)
ContainerGvzHmd9\EntityManagerGhost1396729->__get (/home/chaxwell/public_html/symfony/project/vendor/symfony/var-exporter/LazyGhostTrait.php:176)
Doctrine\ORM\EntityManager->getMetadataFactory (/home/chaxwell/public_html/symfony/project/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:207)
Doctrine\Bundle\DoctrineBundle\CacheWarmer\DoctrineMetadataCacheWarmer->doWarmUp (/home/chaxwell/public_html/symfony/project/vendor/doctrine/doctrine-bundle/CacheWarmer/DoctrineMetadataCacheWarmer.php:42)
Symfony\Bundle\FrameworkBundle\CacheWarmer\AbstractPhpFileCacheWarmer->warmUp (/home/chaxwell/public_html/symfony/project/vendor/symfony/framework-bundle/CacheWarmer/AbstractPhpFileCacheWarmer.php:46)
Symfony\Component\HttpKernel\CacheWarmer\CacheWarmerAggregate->warmUp (/home/chaxwell/public_html/symfony/project/vendor/symfony/http-kernel/CacheWarmer/CacheWarmerAggregate.php:96)
Symfony\Component\HttpKernel\Kernel->initializeContainer (/home/chaxwell/public_html/symfony/project/vendor/symfony/http-kernel/Kernel.php:526)
Symfony\Component\HttpKernel\Kernel->preBoot (/home/chaxwell/public_html/symfony/project/vendor/symfony/http-kernel/Kernel.php:709)

I'm totally not an expert but I have tried to fast-fix this problem by simply moving the call of $connection->setNestTransactionsWithSavepoints(true); before the transaction begins.

From here : 

<?php

namespace DAMA\DoctrineTestBundle\Doctrine\DBAL;

use Doctrine\Bundle\DoctrineBundle\ConnectionFactory;
use Doctrine\Common\EventManager;
use Doctrine\DBAL\Configuration;
use Doctrine\DBAL\Connection;

class StaticConnectionFactory extends ConnectionFactory
{
    /**
     * @var ConnectionFactory
     */
    private $decoratedFactory;

    public function __construct(ConnectionFactory $decoratedFactory)
    {
        parent::__construct([]);
        $this->decoratedFactory = $decoratedFactory;
    }

    public function createConnection(array $params, Configuration $config = null, EventManager $eventManager = null, array $mappingTypes = []): Connection
    {
        $connection = $this->decoratedFactory->createConnection($params, $config, $eventManager, $mappingTypes);

        if (!StaticDriver::isKeepStaticConnections() || !isset($params['dama.keep_static']) || !$params['dama.keep_static']) {
            return $connection;
        }

-        // Make sure we use savepoints to be able to easily roll-back nested transactions
-        if ($connection->getDriver()->getDatabasePlatform()->supportsSavepoints()) {
-            $connection->setNestTransactionsWithSavepoints(true);
-        }
-
        return $connection;
    }
}

To here : 

<?php

namespace DAMA\DoctrineTestBundle\Doctrine\DBAL;

use Doctrine\DBAL\Event\ConnectionEventArgs;

class PostConnectEventListener
{
    public function postConnect(ConnectionEventArgs $args): void
    {
        // can be disabled at runtime
        if (!StaticDriver::isKeepStaticConnections()) {
            return;
        }

        // The underlying connection already has a transaction started.
        // We start a transaction on the connection as well
        // so the internal state ($_transactionNestingLevel) is in sync with the underlying connection.

+        $connection = $args->getConnection();
+
+        // Make sure we use savepoints to be able to easily roll-back nested transactions
+        if ($connection->getDriver()->getDatabasePlatform()->supportsSavepoints()) {
+            $connection->setNestTransactionsWithSavepoints(true);
+        }

        $args->getConnection()->beginTransaction();
    }
}

It works back again but I guess it would not in all situations, no idea.

I hope this helps :)

dmaicher commented 1 year ago

@Chaxwell thanks for the details!

Could you do me a favor and test https://github.com/dmaicher/doctrine-test-bundle/pull/232 ?

By running composer req --dev dama/doctrine-test-bundle "dev-remove_post_connect_event as 8.0" for example

For supporting Doctrine DBAL 4 I need to change the approach quite a bit and I believe this will also solve this issue.

Chaxwell commented 1 year ago

Seems like it's working just fine :) 

$ XDEBUG_MODE=off symfony php ./vendor/symfony/phpunit-bridge/bin/simple-phpunit.php --filter RiskGravityControllerRouteTest
Debug mode: off
Clearing the cache... Cache cleared !
PHPUnit 9.6.6 by Sebastian Bergmann and contributors.

Testing 
.......                                                             7 / 7 (100%)

Time: 00:19.555, Memory: 259.00 MB

OK (7 tests, 23 assertions)

Remaining indirect deprecation notices (62)
Done in 20.02s.

Data is rolledback as expected.

I will test with my whole suite and see if it's still working as expected.

dmaicher commented 1 year ago

@Chaxwell nice. Let me know if it works fine with the whole testsuite. On my projects everything works as expected.

I would then merge #232 and probably tag a new v8.0.0-BETA1 release to collect some more feedback

dmaicher commented 1 year ago

@Chaxwell did you have a chance to test this with your whole testsuite? :blush: