internetstandards / Internet.nl

Internet standards compliance test suite
https://internet.nl
166 stars 36 forks source link

Build fails on integration-test after a new language has been added #1338

Closed apio-sys closed 4 months ago

apio-sys commented 4 months ago

The integration-test after build fails as follows:

=================================== FAILURES ===================================
_ test_change_language[chromium-nl-en-Internet.nl is an initiative of the Internet community and the Dutch] _

page = <Page url='https://nl.internet.test/'>, app_domain = 'internet.test'
from_language = 'nl', to_language = 'en'
footer_text = 'Internet.nl is an initiative of the Internet community and the Dutch'

    @pytest.mark.parametrize(
        ("from_language", "to_language", "footer_text"),
        [("nl", "en", FOOTER_TEXT_EN), ("en", "nl", FOOTER_TEXT_NL)],
    )
    def test_change_language(page, app_domain, from_language, to_language, footer_text):
        """Test clicking the language change button."""

        page.goto(f"https://{from_language}.{app_domain}")
>       page.locator("#language-switch-header-container button:not(:disabled)").click()

integration_tests/integration/test_basic.py:152: 

I can imagine this is due to the fact that I added a 3rd language to the front (fr.). In such a case, should the test be fixed in my fork or ideally in the main project so that the test is more generic and would allow for additional languages?

mxsasha commented 4 months ago

It would be good to make it more generic, as long as that does not end up adding too much complexity. I think it will be a common issue.

apio-sys commented 4 months ago

Indeed but not an easy one. It will all depend on how many languages are kept in the end product, probably typically a local language + english. To try and tackle this at least partly, I have removed the sort order from internetnl/settings.py:

LANGUAGES = [
    ("en", "English"),
    ("nl", "Dutch"),
    ("fr", "French"),
]

That way, I'm sure my added language stays in 3rd position in the top menu and theoretically should interfere with the integration-test I thought. However, that one still fails with the same error as before:

>       result = next(iter(done)).result()
E       playwright._impl._api_types.Error: Error: strict mode violation: locator("#language-switch-header-container button:not(:disabled)") resolved to 2 elements:
E           1) <button>English</button> aka get_by_role("button", name="English")
E           2) <button>français</button> aka get_by_role("button", name="français")

as if the order of the buttons wasn't changed for the test instance. On the production I do see my 3rd language as 3rd menu entry (cf. https://conformite-internet.fr/) as opposed to 2nd entry when they were sorted alphabetically...

aequitas commented 4 months ago

@apio-sys I changed the test in https://github.com/internetstandards/Internet.nl/pull/1343, it should now work when other languages are added to the settings.

apio-sys commented 4 months ago

Many thanks, that works fine now!