search

magento / magento2

Prior to making any Submission(s), you must sign an Adobe Contributor License Agreement, available here at: https://opensource.adobe.com/cla.html. All Submissions you make to Adobe Inc. and its affiliates, assigns and subsidiaries (collectively “Adobe”) are subject to the terms of the Adobe Contributor License Agreement.
http://www.magento.com
Open Software License 3.0
11.56k stars 9.32k forks source link

Integration tests failing during Magento setup #33802

Closed sergeynezbritskiy closed 1 year ago

sergeynezbritskiy commented 3 years ago

Preconditions (*)

  1. Fresh Magento 2.4.3 installation
  2. PHP 7.3 or PHP 7.4
  3. Composer version 2.1.5 (also tried with the highest composer 1 version
  4. no RabbitMQ

Steps to reproduce (*)

  1. Basically I follow the steps in this doc https://devdocs.magento.com/guides/v2.4/test/integration/integration_test_execution.html
  2. composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition
  3. bin/magento module:enable --all
  4. update the configuration indev/tests/integration/etc/install-config-mysql.php` (no RabbitMQ configuration)
  5. mysql -e "DROP DATABASE IF EXISTS magento_integration_tests"
  6. mysql -e "CREATE DATABASE magento_integration_tests"
  7. cd dev/tests/integration && php ../../../vendor/bin/phpunit

Expected result (*)

  1. Magento test instance should be configured in dev/tests/integration/tmp/sandbox-{hash} dir
  2. Tests defined in dev/tests/integration/phpunit.xml.dist should start running

Actual result (*)

  1. When running magento installation command, this command fails during the step "Disabling Maintenance Mode" with an error

    In ResourceConnection.php line 148:

[DomainException]
Connection "default" is not defined

    Log file is attached below

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

m2-assistant[bot] commented 3 years ago

Hi @sergeynezbritskiy. Thank you for your report. To help us process this issue please make sure that you provided the following information:

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

sergeynezbritskiy commented 3 years ago

Additional information regarding this issue:

  1. The issue is not reproducible on current 2.4-develop branch
  2. If I run integration tests second time and TESTS_CLEANUP config in phpunit.xml is set to disabled, the tests are running fine, so currently as a workaround I wrapped this command in try/catch in my Jenkinsfiles and run tests second time in catch 
    
script {
try {
    sh 'cd ${TMP_DIR}/dev/tests/integration && php ../../../vendor/bin/phpunit'
} catch (err) {
    sh 'cd ${TMP_DIR}/dev/tests/integration && php ../../../vendor/bin/phpunit'
}
}
sergeynezbritskiy commented 3 years ago

During the investigation I've found few issues:

  1. Some objects, which should be cached in object manager and shared btw objects, are not actually shared and there are multiple instances of the same class:
    • \Magento\Framework\App\ResourceConnection
    • \Magento\Framework\App\DeploymentConfig

When object manager first creates ResourceCollection object, it has a dependency DeploymentConfig $deploymentConfig with 

$data = [
    'cache_types' => [
        'compiled_config' => 1
    ]
]

And this object is cached, but then I see that another instances of ResourceConnection is created with another instance of deploymentConfig as its dependency. The data of this new deploymentConfig get updated multiple times during installation, but when we go to the step "Disable maintenance mode" I see that at some place there is a call to the old object of resource connection with an old instance of deployment config. And when we try to get db connection, there is no info about db connection in the old version of deployment config. https://github.com/magento/magento2/blob/2.4-develop/lib/internal/Magento/Framework/App/ResourceConnection.php#L141

  1. Second issue is that deployment config at some point gets cached inside the instance of DeploymentConfig, and every time we need reset it manually. so even if it doesn't return anything from cached $this->data, we presume that this configuration doesn't exist at all

I see 3 possible solutions:

  1. if we can't get db configuration from deployment config, then we need to reset data and try to get db configuration again. This is mostly a workaround for a single case, so doesn't solve the issue.
  2. ensure that shared objects are indeed shared all around the app. Currently I wasn't able to figure out why is it happening and how to solve is. Probably this should even work as it works now, so don't think that it would be a good idea to debug this.
  3. extend the functionality of \Magento\Framework\App\DeploymentConfig. Every time when the requested configuration was not found in $this->data or $this->flattenedData, we reload the configuration.

I think that the third solution is preferable in this case, but think that someone from Adobe team need to approve this solution, no ?

m2-assistant[bot] commented 3 years ago

Hi @engcom-Hotel. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

m2-assistant[bot] commented 3 years ago

Hi @engcom-Echo. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

engcom-Echo commented 3 years ago

Hi @sergeynezbritskiy We tried to reproduce this issue on 2.4-develop & were unable to do so. Also as per your comments, you have confirmed that issue is not reproducible on 2.4-develop, Hence we are closing this issue. We request you to create separate issues for any new issues identified during development/debugging. Thank you.

ihor-sviziev commented 3 years ago

@engcom-Echo, please double-check on 2.4.3, as this release wasn't yet merged to the 2.4-develop branch, or we can wait and then re-check again after merging.

t-heuser commented 3 years ago

I can confirm that this issue exists. I just updated to magento 2.4.3 and then tried to run integration tests. I get the exact same error as @sergeynezbritskiy stated in the issue.

The same bug also appears when trying to install magento with the file app/etc/config.php present. To reproduce try the following:

I think that these errors are related as the setup process for integration tests tries to install magento with an env.php and config.php present, so I think both errors are fixable at once.

Should I still create an extra issue for it or not, what do you think @ihor-sviziev?

ihor-sviziev commented 3 years ago

No need to create a separate issue. I reopened this one

m2-assistant[bot] commented 3 years ago

Hi @engcom-Lima. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

cmuench commented 3 years ago

@ihor-sviziev Is anyone at Adobe working on that issue. It's currently blocking the deployment of several (also commerce) projects, because our build pipelines are broken. That's a core issue which should be prioritized by the core team.

cmuench commented 3 years ago

We have that issue if the config.php is checked in and if we run that in the CI pipeline. php bin/magento setup:install --cleanup-database --db-host=localhost --db-user=root --db-password=root --db-name=magento

On the local machine we can run the command twice. The second run fixes the (maybe cache) issue. In the CI it's not the idea to run the installation process twice.

cmuench commented 3 years ago

New information. If we delete the app/etc/config.php in the pipeline, the installation finishes without a failure. But that's not a solution.

cmuench commented 3 years ago

We can confirm that the PR #33803 fixes the issue. We added the code changes a patch in our project. The pipeline is then "green".

github-jira-sync-bot commented 3 years ago

:white_check_mark: Jira issue https://jira.corp.magento.com/browse/AC-1405 is successfully created for this GitHub issue.

m2-assistant[bot] commented 3 years ago

:white_check_mark: Confirmed by @engcom-Alfa. Thank you for verifying the issue.
Issue Available: @engcom-Alfa, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

mrodespin commented 3 years ago

We are struggling with the same issue, and i can confirm that the PR #33803 fixes it.

astephanh commented 3 years ago

i too can confirm, that the PR #33803 fixes the issue.

jreinke commented 2 years ago

For a reason I do not have time to investigate for, I have removed the package magento/composer-root-update-plugin from my integration tests pipeline and it works now:

composer remove magento/composer-root-update-plugin

Maybe it will help someone!

BorisovskiP commented 2 years ago

Any updates on this issue? I can reproduce it in a pipeline on Magento 2.4.4.

ilnytskyi commented 2 years ago

I also confirm that https://github.com/magento/magento2/pull/33803/ fixes the issue with installation in pipelines in our case

pawan-mittal commented 2 years ago

Any updates on this issue? I can reproduce it in a pipeline on Magento 2.4.4.

engcom-Hotel commented 2 years ago

Hello @pawan-mittal,

The related PR https://github.com/magento/magento2/pull/33803 is in Merge in progress status.

Thanks

Vinai commented 2 years ago

Reproduced on 2.4.5

norgeindian commented 2 years ago

Does anybody by chance already have a patch file to fix this?

hostep commented 2 years ago

Here's one for Magento 2.4.5 which I've created earlier today, it's based on https://github.com/magento/magento2/pull/33803 but had to fix some merge conflicts, so hopefully those were solved correctly, but it seems to work on our end at least.

PR-33803-Framework.txt

norgeindian commented 2 years ago

@hostep , thanks a lot, awesome. Works fine for me.

peterjaap commented 2 years ago

Reproduced on 2.4.5-p1.

The patch provided by @hostep works.

peterjaap commented 2 years ago

The current pull request here does not work on 2.4.3 currently since the load() function differs between 2.4.3 and 2.4.4;

convenient commented 1 year ago

FWIW i've just got this on 2.4.7-beta1

My CI tooling can fix it by re-running composer install --no-interaction (which I discovered when playing with this comment https://github.com/magento/magento2/issues/33802#issuecomment-1112369298)

https://github.com/AmpersandHQ/magento-docker-test-instance/pull/12/files#diff-79a3628f21c70a32e21bfe89edb38a4f158b1707f6c1bea98083204d0934484fR143

hostep commented 1 year ago

This problem has indeed returned in 2.4.7-beta1, but also in 2.4.6-p2 (which is now available in pre-release and will be available from the 8th of August for everyone).

I've opened a new issue for this: https://github.com/magento/magento2/issues/37805

seepossible commented 10 months ago

Here's one for Magento 2.4.5 which I've created earlier today, it's based on #33803 but had to fix some merge conflicts, so hopefully those were solved correctly, but it seems to work on our end at least.

PR-33803-Framework.txt

@hostep your patch is very useful. Thank you