Tests run against Hyvä demo site are failing

[peterjaap]

A number of tests are failing since the Hyvä demo site (https://demo.hyva.io/) is updated and some settings changed. For example; prices are now shown in Euro and the newsletter subscription box has been removed/deactivated.

Failing tests;

• Can subscribe to newsletter
• Can add a coupon to the cart
• Can delete an added coupon from the cart
• Cannot add an invalid coupon
• Can set the price to zero when every associated product qty is zero
• Can calculate the price based on selected options
• Can visit the category page and filters on color red
• Can see a price for the product
• Can add a product to the wishlist when customer is logged in
• Can see the correct product price and shipping costs
• Can see coupon discount in checkout
• Can find and order in the customer order history after having placed an order
• Can open default CMS pages
• Can find a single product
• Can see suggestions when entering search terms
• Can add an address automatically from saved address'
• Can remove an address
• Can edit the wishlist and remove item

  (Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  homepage.spec.js                         00:16        8        7        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  cart/cart.spec.js                        00:38        7        4        3        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  cart/minicart.spec.js                    00:13        7        7        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  catalog/bundle.spec.js                   00:16        5        3        2        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  catalog/category.spec.js                 00:11        6        5        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  catalog/product.spec.js                  00:23       13       11        2        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  checkout/checkout.spec.js                00:17        3        -        3        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  cms/cms-pages.spec.js                    00:05        2        1        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  search/product-searches.spec.js          00:13        4        2        2        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  user/account.spec.js                     01:57       17       12        5        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  9 of 10 failed (90%)                     04:33       72       52       20        -        -  

[peterjaap]

I'll change the default domain to test to our own Hyvä demo environment, and I'll remove any custom modules there like Elasticsuite, so it matches with the default.

[peterjaap]

Less failing tests when run on https://demo-hyva.elgentos.io

• Can add a coupon to the cart
• Can delete an added coupon from the cart
• Cannot add an invalid coupon
• Can add a product to the wishlist when customer is logged in
• Can see coupon discount in checkout
• Can find and order in the customer order history after having placed an order
• Can open default CMS pages
• Can remove an address
• Can edit the wishlist and remove item
• Can login from cart
• Can login from checkout

  (Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  homepage.spec.js                         00:18        8        8        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  cart/cart.spec.js                        00:45        7        4        3        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  cart/minicart.spec.js                    00:16        7        7        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  catalog/bundle.spec.js                   00:07        5        5        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  catalog/category.spec.js                 00:12        6        6        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  catalog/product.spec.js                  00:22       13       12        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  checkout/checkout.spec.js                00:34        3        1        2        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  cms/cms-pages.spec.js                    00:05        2        1        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  search/product-searches.spec.js          00:07        4        4        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  user/account.spec.js                     02:22       17       13        4        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  5 of 10 failed (50%)                     05:11       72       61       11        -        -  

[peterjaap]

Most of the issues are related to the flash messages. In Cypress, the first flash message keeps getting shown even though new ones are added. When using a regular browser in the conventionel way, this does not happen. Heisenbug.