codeceptjs / CodeceptJS

Supercharged End 2 End Testing Framework for NodeJS
http://codecept.io
MIT License
4.1k stars 721 forks source link

Scenario and step timeouts are not working as expected #4310

Open gfoal opened 4 months ago

gfoal commented 4 months ago

Problem

Hey guys, looks like there is a problem with timeouts. I notice two problems:

  1. Global step timeout doesn't always work (see example below where step1 doesn't have specific step timeout but it fails because of Scenario timeout)
  2. After Scenario times out, all the consequent Scenarios are failing because of 0.01ms timeout. It looks like Scenario timeout acts as a global timeout

Expected results

In my case, I'd expect step1 to fail because of global step timeout of 150sec and step2 fail because of custom timeout of 120sec which I set in config. All the Scenarios should execute and if they time out after 600sec (as set in config), it should affect only single Scenario. But it's not the case

Console output:

# initial step fails
[LOG] 04-17-24 03:35:58 error: Step 'step1' failed
[LOG] 04-17-24 03:35:58 error:  | TC-8652: Verify that *** failed:
Error: Action step1: *** was interrupted on step timeout 564676ms
# step from After() fails with the timeout of 38ms
[LOG] 04-17-24 03:36:03 error: Step 'step1' failed
[LOG] 04-17-24 03:36:03 error: "after each" hook: *** failed:
Error: Action step1: *** was interrupted on step timeout 38ms
# all the consequent tests fail with timeout of 0.01ms
[LOG] 04-17-24 03:36:08 error: Step 'step2' failed
[LOG] 04-17-24 03:36:08 error: "before all" hook: @@@ failed:
Error: Action step2: *** was interrupted on step timeout 0.01ms
# ...
[LOG] 04-17-24 03:36:09 error: Step 'step2' failed
[LOG] 04-17-24 03:36:09 error: "before all" hook: @@@ failed:
Error: Action step2: *** was interrupted on step timeout 0.01ms
# 400+ tests fail with the same log output

Details

Config

Here's timeouts config. Note that step1 which was the step where scenario initially failed doesn't have specific timeout

exports.config = {
  timeout: 600,
  plugins: {
    stepTimeout: {
      enabled: true,
      customTimeoutSteps: [
        ["step2", 120],
        ["step3", 60]
      ]
    }
  }
}
gfoal commented 4 months ago

I also noticed a note in the docs which says that previous implementation was disabled. I wonder if the issue still persists or returned somehow

image

github-actions[bot] commented 1 month ago

This issue is stale because it has been open for 90 days with no activity.