CI Tests call smoke diff tests which are never equal due to different domain - Githubissues

adobe / helix-publish

Microservice for publishing Project Helix Fastly service configurations

Apache License 2.0

5 stars 6 forks source link

CI Tests call smoke diff tests which are never equal due to different domain #640

Closed tripodsan closed 3 years ago

tripodsan commented 3 years ago

Description https://app.circleci.com/pipelines/github/adobe/helix-pages/2108/workflows/2f56d05f-2ef8-49c4-85e4-c7eddcd4f079/jobs/8252/tests

$ diff ok ok2
6c6
<   <link rel="canonical" href="https://main--helix-purge--adobe.hlx.page/OK.html">
---
>   <link rel="canonical" href="https://main--helix-purge--adobe.hlx-2.page/OK.html">
10,12c10,12
<   <meta property="og:url" content="https://main--helix-purge--adobe.hlx.page/OK.html">
<   <meta property="og:image" content="https://main--helix-purge--adobe.hlx.page/default-meta-image.png">
<   <meta property="og:image:secure_url" content="https://main--helix-purge--adobe.hlx.page/default-meta-image.png">
---
>   <meta property="og:url" content="https://main--helix-purge--adobe.hlx-2.page/OK.html">
>   <meta property="og:image" content="https://main--helix-purge--adobe.hlx-2.page/default-meta-image.png">
>   <meta property="og:image:secure_url" content="https://main--helix-purge--adobe.hlx-2.page/default-meta-image.png">
20c20
<   <meta name="twitter:image" content="https://main--helix-purge--adobe.hlx.page/default-meta-image.png">
---
>   <meta name="twitter:image" content="https://main--helix-purge--adobe.hlx-2.page/default-meta-image.png">

tripodsan commented 3 years ago

@rofe did this change? did we always include the hostname in the meta properties?

rofe commented 3 years ago

Yes, since June 5: https://github.com/adobe/helix-pages/pull/323

trieloff commented 3 years ago

@kptdobe @MarquiseRosier so is this a problem of the diff tests (should exclude domains) or the smoke tests (should exclude diffs)?

kptdobe commented 3 years ago

humm... I have fixed that recently: https://github.com/adobe/helix-pages/blob/master/test/pages/pages.test.js#L36

kptdobe commented 3 years ago

The tests just passed... we have too many 504 (just had one when opening the https://main--helix-purge--adobe.hlx-2.page/OK.html page).

kptdobe commented 3 years ago

Could it be a concurrency access issue ? https://helix.coralogix.com/#/query/logs?id=2wuqvVbgIPo Corresponding tests fail. https://pages--adobe.hlx.page/ihearttech/en/header.plain.html https://pages--adobe.hlx-4.page/ihearttech/en/header.plain.html both report a 504. For the test purpose, they are both requested "at the time" and consume the same Word doc or GitHub file.

kptdobe commented 3 years ago

No concurrency issue but 2 different issues:

one is Google User Rate Limit Exceeded
second is Gateway timeout of 20000 milliseconds exceeded for https://adobeioruntime.net/api/v1/web/helix-pages/helix-services/content-proxy@v2?owner=adobe&repo=pages&path=%2Fihearttech%2Fen%2Fheader.md&ref=515fcb06cb2e0c583ab07a59bfc0f1364c6c23ce

kptdobe commented 3 years ago

Conclusion, I see 2 different cases (certainly related):

real DOM difference (apart from the domain name). This is what happened originally in this issue but we have no way to know the DOM difference. And the really strange thing is that it did not happen again when re-running the test. I just fixed and merged https://github.com/adobe/helix-pages/issues/525 which should help visualising the DOM difference of a failing test. Hopefully, the next time the CI should show us the diff and we can understand the reason (could be a timeout or a 504 in an ESI for example - the head(er) / footer should be then empty in one page). Which leads to second point...
I also opened https://github.com/adobe/helix-pages/issues/527 to track failing tests because of 504 which is currently way too frequent. If this happens, just re-run the test(s) but in the meantime we must reduce the number.

Closing this one for now.