search

yahoo / protractor-retry

Use protractor features to automatically re-run failed tests with a specific configurable number of attempts.
MIT License
28 stars 32 forks source link

Chrome crashes are treated as success in 1.1.9 #40

Closed khitrenovich closed 6 years ago

khitrenovich commented 6 years ago

We run protractor-retry on sharded specs in sequence with the following configuration:

  capabilities: {
    'browserName': 'chrome',
    'shardTestFiles': true, // with shard, we can retry individual specs
    'maxInstances': 1,
    ...
  }

It appears that if Chrome crashes on one of the specs (see example in #25), protractor-retry treats that as success and marks all tests as passed. This is a regression since it worked fine in v1.1.8 and is broken now.

khitrenovich commented 6 years ago

Possible cause: https://github.com/yahoo/protractor-retry/pull/29/files#r185912791

dreuxl commented 6 years ago

mmh. I got what u mean but the 1.1.8 behavior was not completely ok. What is the return code you have when Chrome is crashing? the retry never intend to manage crash of browsers and treat it as a failure (as mentioned in https://github.com/yahoo/protractor-retry/issues/25), it s fortunate somehow that it worked for u up to 1.1.8.

I m open to suggestion to on how to handle that case.

khitrenovich commented 6 years ago

How can I see the return code? Is it logged somewhere?

dreuxl commented 6 years ago

it s available in when you run your Cmd in debug mode, ie

DEBUG=* yourprotractorcmd

khitrenovich commented 6 years ago

Thanks, I'll check how can I set this up in our CI environment. Unfortunately, this env goes through some major upgrade now, so next week is the earliest...

khitrenovich commented 6 years ago

I haven't seen Chrome crashes for long time, and I don't know how to force-crash it. Going to close the issue for now.