MFTF >=4.3.1 uses a Yandex\Allure class that doesn't exist

densen45 commented 1 year ago

Preconditions and environment

Magento version: 2.4.6 (installed via composer)

From composer.json:

"allure-framework/allure-phpunit": "^2"
"magento/magento2-functional-testing-framework": "^4.2.1" (currently 4.3.1 is being installed)

Steps to reproduce

Setup MFTF (https://developer.adobe.com/commerce/testing/functional-testing-framework/getting-started/)
Try to execute a test

or just visit the class Magento\FunctionalTestingFramework\Allure\Adapter\MagentoAllureAdapter and notice that the class Yandex\Allure\Codeception\AllureCodeception doesn't exist. This class is used and should serve as parent for the MagentoAllureAdapter.

Expected result

All used classes exist and MFTF tests can be executed.

Actual result

MFTF tests cannot be executed due to missing dependencies.

Additional information

Explicitly setting the MFTF version to 4.2.1 in composer.json and execute composer update solves this issue.

"magento/magento2-functional-testing-framework": "4.2.1" (without the ^)

Release note

No response

Triage and priority

[ ] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
[ ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
[X] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
[ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
[ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

m2-assistant[bot] commented 1 year ago

Hi @densen45. Thank you for your report. To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release
For more details, review the Magento Contributor Assistant documentation.
Add a comment to assign the issue: @magento I am working on this
To learn more about issue processing workflow, refer to the Code Contributions.

Join Magento Community Engineering Slack and ask your questions in #github channel. :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting. :clock10: You can find the schedule on the Magento Community Calendar page. :telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

m2-assistant[bot] commented 1 year ago

Hi @engcom-November. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

[ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
[ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
[ ] 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
[ ] 4. Verify that the issue is reproducible on 2.4-develop branch
Details
- Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
- If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
- If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
[ ] 5. Add label Issue: Confirmed once verification is complete.
[ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

engcom-November commented 1 year ago

Hi @densen45 , Thank you for reporting and collaboration. Verified the issue on Magento 2.4-develop instance and the issue is reproducible. Unable to execute MFTF tests with 4.3.1 version. Also changing / restoring back to "magento/magento2-functional-testing-framework": "^4.2.1" in composer.json followed by composer update is also not working. Workaround: Only by explicitly setting the MFTF version to 4.2.1 in composer.json and execute composer update solving this issue. "magento/magento2-functional-testing-framework": "4.2.1" (without the ^)

github-jira-sync-bot commented 1 year ago

:white_check_mark: Jira issue https://jira.corp.adobe.com/browse/AC-8996 is successfully created for this GitHub issue.

m2-assistant[bot] commented 1 year ago

:white_check_mark: Confirmed by @engcom-November. Thank you for verifying the issue.
Issue Available: @engcom-November, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

densen45 commented 6 months ago

The issue is getting worse.

The composer.json from M 2.4.7 requires magento/magento2-functional-testing-framework ^4.7 and this has the same issue as described above.

With Magento 2.4.7 it's no longer possible to explicitly downgrade Magento Functional Testing Framework to version 4.2.1, because this requires symfony/finder ^5.0, which requires symfony/string ^5.4, but M 2.4.7 requires symfony/string ^6.4.

composer.json

...
        "magento/magento2-functional-testing-framework": "4.2.1",
...
        "symfony/finder": "^5.4"

$ composer update
Gathering patches for root package.
Loading composer repositories with package information                                                                                                                                                          Updating dependencies                                 
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires magento/product-community-edition 2.4.7 -> satisfiable by magento/product-community-edition[2.4.7].
    - You can only install one version of a package, so only one of these can be installed: symfony/string[v5.1.10, ..., v5.4.39, v6.0.0, ..., v6.4.7, v7.0.0, ..., v7.0.7].
    - magento/magento2-functional-testing-framework 4.2.1 requires symfony/string ^5.4 -> satisfiable by symfony/string[v5.4.0, ..., v5.4.39].
    - magento/product-community-edition 2.4.7 requires symfony/string ^6.4 -> satisfiable by symfony/string[v6.4.0, ..., v6.4.7].
    - Root composer.json requires magento/magento2-functional-testing-framework 4.2.1 -> satisfiable by magento/magento2-functional-testing-framework[4.2.1].

Thus, I'm not able to execute any MFTF test in Magento 2.4.7. Has anyone else experienced this?

densen45 commented 2 months ago

Hey everyone, this issue is still a pain in my neck. We have invested a lot of time in creating MFTF tests, but since 2.4.7 we can no longer use them.

Guys, I'm sorry for marking you directly but I don't know how else to make progress here. Maybe you can help me :)

@okolesnyk you introduced the inheritance to the AllureCodeception class from Yandex, which does not exist (https://github.com/magento/magento2-functional-testing-framework/blob/d333743e9a0d7e722a3fbd0a2dec5e77e99fba17/src/Magento/FunctionalTestingFramework/Allure/Adapter/MagentoAllureAdapter.php#L42),
@soumyau you are the the last to make a real code change to the ‘MagentoAllureAdapter’ class,
@hostep and you are the last one who edited the MagentoAllureAdapter class.

Many thanks to all those who are willing to contribute to this issue.

hostep commented 1 month ago

It looks like you are correct.

This worked correctly in Magento 2.4.5 which comes with magento/magento2-functional-testing-framework 3.5.1 requires allure-framework/allure-codeception (~1.4.0||~1.5.0) But is broken in 2.4.6 and 2.4.7, latest comes with magento/magento2-functional-testing-framework 4.7.2 requires allure-framework/allure-codeception (^2.1)

The namespace of that AllureCodeception class changed from Yandex\Allure\Codeception to Qameta\Allure\Codeception between version 1 and 2 of allure-framework/allure-codeception by this commit: https://github.com/allure-framework/allure-codeception/commit/79a583a2fd13ce3b665d83edb2b51c37408be2dc

Part of the MFTF codebase was updated for the new namespace in https://github.com/magento/magento2-functional-testing-framework/commit/32615bc36c2c39608b7e57fcbdb5cd7187c5a8de, but not all of it.

Strange that Adobe's automatic tests didn't pick this up... Just running phpstan myself locally in an 2.4.7 project on the MFTF vendor codebase reveals this error next to a bunch of others:

$ vendor/bin/phpstan analyse --level=0 vendor/magento/magento2-functional-testing-framework/ | egrep ' not found.| unknown class '
 342/342 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

  17     Class Magento\Sniffs\Commenting\FunctionCommentSniff extends unknown class PHP_CodeSniffer\Standards\PEAR\Sniffs\Commenting\FunctionCommentSniff.
  82     Call to static method suggestType() on an unknown class PHP_CodeSniffer\Util\Common.
  247    Call to static method getConfigData() on an unknown class PHP_CodeSniffer\Config.
  368    Call to static method suggestType() on an unknown class PHP_CodeSniffer\Util\Common.
  383    Access to static property $allowedTypes on an unknown class PHP_CodeSniffer\Util\Common.
  16     Class Magento\Sniffs\Commenting\VariableCommentSniff extends unknown class PHP_CodeSniffer\Sniffs\AbstractVariableSniff.
  102    Call to static method suggestType() on an unknown class PHP_CodeSniffer\Util\Common.
  13     Class Magento\Sniffs\Files\LineLengthSniff extends unknown class PHP_CodeSniffer\Standards\Generic\Sniffs\Files\LineLengthSniff.
  42     Class Magento\FunctionalTestingFramework\Allure\Adapter\MagentoAllureAdapter extends unknown class Yandex\Allure\Codeception\AllureCodeception.
  193    Instantiated class Yandex\Allure\Adapter\Event\StepStartedEvent not found.
  213    Instantiated class Yandex\Allure\Adapter\Event\StepFailedEvent not found.
  216    Instantiated class Yandex\Allure\Adapter\Event\StepFinishedEvent not found.
  227    Instantiated class Yandex\Allure\Adapter\Event\TestCaseFailedEvent not found.
  241    Instantiated class Yandex\Allure\Adapter\Event\TestCaseBrokenEvent not found.
  246    Instantiated class Yandex\Allure\Adapter\Model\Step not found.
  249    Access to constant BROKEN on an unknown class Yandex\Allure\Adapter\Model\Status.
  334    Instantiated class Yandex\Allure\Adapter\Event\TestCaseFinishedEvent not found.
  349    Call to static method lifecycle() on an unknown class Yandex\Allure\Adapter\Allure.
  349    Instantiated class Yandex\Allure\Adapter\Event\AddAttachmentEvent not found.
  409    Call to static method getOutputDirectory() on an unknown class Yandex\Allure\Adapter\Model\Provider.
  16     Class Magento\FunctionalTestingFramework\Allure\Event\AddUniqueAttachmentEvent extends unknown class Yandex\Allure\Adapter\Event\AddAttachmentEvent.
  38     Instantiated class Yandex\Allure\Adapter\AllureException not found.
  50     Instantiated class Yandex\Allure\Adapter\AllureException not found.
  43     Class Magento\FunctionalTestingFramework\Config\Magento\FunctionalTestingFramework\Config\Dom not found.
  387    Class Magento\FunctionalTestingFramework\Extension\Comment not found.
  404    Class Magento\FunctionalTestingFramework\Extension\Comment not found.
  228    Instantiated class Magento\FunctionalTestingFramework\Suite\RuntimeException not found.
  43     Instantiated class Magento\FunctionalTestingFramework\StaticCheck\ActionGroupArgumentsCheck not found.

@KevinBKozan, @manjusha729: are you guys aware of this? Any plans on fixing this in a new version?

hostep commented 1 month ago

Looks like it was already reported before many months ago in https://github.com/magento/magento2-functional-testing-framework/issues/912

I have no experience with running custom MFTF tests myself, but does the comment from https://github.com/magento/magento2-functional-testing-framework/issues/912#issuecomment-1863379033 help @densen45? It sounds like maybe updating those yaml files might solve your issue?

KevinBKozan commented 1 month ago

@hostep Hello! I was not directly aware of this issue, but I'm going to create an internal ticket to take a look at this ASAP.

Surface level I understand that it's a namespacing issue, but given the complexity of the relationship between MFTF dependencies and M2 dependencies I want to understand the exact workflow that's leading to this. We run our MFTF suite when we create the M2 composer packages, and any issues like these would definitely be caught by the workflow of pulling composer magento and trying to run tests.

KevinBKozan commented 1 month ago

@hostep an update on the above:

We went through reproduction steps and yes, it seems like the only way to reproduce it is to:

Pull MFTF 2.4.x
Run build:project to generate the codeception.yml
Upgrade MFTF to MFTF 2.
Try to run any tests - namespacing issue is encountered

The intended workflow for this usecase is to just run mftf reset --hard (documentation) which will remove the old yml configuration files that are generated. Then you can build:project and the correct configuration files will be regenerated.

I also personally pulled Magento 2.4.7 (2.4.7-p2) and went through a normal workflow to install MFTF and I was able to run tests without any namespacing issues. Worth noting MFTF 4.7.2 is being pulled, so of course this would work.

@densen45 may I ask why you're locking MFTF to an older version as opposed to just using whichever comes with the composer package?

hostep commented 1 month ago

@KevinBKozan, thanks for the explanation, hopefully that will help @densen45 further.

But, any reason why you keep old classes around in the MFTF package that reference classes that your dependencies are not pulling in? In my opinion, you should remove the classes MagentoAllureAdapter and AddUniqueAttachmentEvent for example if you're expecting users to no longer use them.

KevinBKozan commented 1 month ago

@hostep Honestly no, I don't think there is a reason to have kept them around. If I remember correctly, the Codeception version bump had to keep being delayed due to M2 dependency issues; I'm positive the cleanup portion was just missed in lieu of wanting to be 100% sure version bump was thoroughly compatible.

I'm very motivated to perform that cleanup though - I'll create another internal task to take care of it. Good call out! 👍

densen45 commented 1 month ago

@hostep Thank you very much for your investigation, fast response and involving @KevinBKozan. Really well done!

@KevinBKozan I have executed mftf reset --hard and rebuilt the project afterwards (mftf build:project). And I can't believe it, but now I can finally carry out our MFTF tests again. Yippee :partying_face: oh, this is great! Many thanks!

That means, executing mftf build:project is not enough for a clean test preparation, if you have left overs form previous versions (see the end of this post)? Although it says

$ mftf build:project

mftf files removed from filesystem.
...

I'm not sure if I get you @KevinBKozan right here: "may I ask why you're locking MFTF to an older version as opposed to just using whichever comes with the composer package?" Initially I didn't locked the MFTF version. This was a workaround to make our MFTF tests work again in Magento 2.4.6 (see my initial report from June, 2023). We normally use the versions that come with the composer package, but at least since 2.4.6 we also use the directory/code base of the previous Magento version for our local projects' basis. That means, the old config files apparently were a left over from the workaround we applied in Magento 2.4.6.

magento / magento2