selecting multiple browsers doesn't work as expected

[kasperpeulen]

I select four different platforms: --platform=dartium,chrome,firefox,safari but only dartium and safari are tested:

Kaspers-MacBook-Pro:polymerjs_wrapper test$ grind
grinder running [test]

[test]
  pub run test:test --reporter=expanded --platform=dartium,chrome,firefox,safari test/polymer2.dart
  Compiling test/polymer2.dart...
  00:00 +0: [Dartium] WebComponentsReady
  00:00 +1: [Dartium] WebComponentsReady
  00:00 +2: [Dartium] WebComponentsReady
  00:00 +3: [Safari] WebComponentsReady
  00:00 +4: All tests passed!

finished in 22.923 seconds.

Not sure if I do something wrong here, I do:

• on the top of the file @TestOn('dartium||chrome||firefox||safari')
• on the command --platform=dartium,chrome,firefox,safari

[kasperpeulen]

If you want to reproduce, here is the test located: https://github.com/kasperpeulen/polymerjs_wrapper/blob/master/test/polymer.dart

the full test give this log:

[test]
  pub run test:test --reporter=expanded --platform=dartium,chrome,firefox,safari test/polymer2.dart
  Compiling test/polymer2.dart...
  00:00 +0: [Dartium] WebComponentsReady
  00:00 +1: [Dartium] WebComponentsReady
  00:00 +2: [Dartium] WebComponentsReady
  00:00 +3: [Safari] WebComponentsReady
  00:00 +4: [Safari] WebComponentsReady
  00:00 +5: [Safari] WebComponentsReady
  00:00 +6: [Safari] WebComponentsReady
  00:00 +7: [Safari] WebComponentsReady
  00:00 +8: [Safari] WebComponentsReady
  00:00 +9: [Safari] WebComponentsReady
  00:00 +10: [Safari] WebComponentsReady
  00:00 +11: [Chrome] IronMultiSelectableBehavior properties and methods
  00:00 +12: [Firefox] IronMultiSelectableBehavior properties and methods
  00:00 +13: [Firefox] IronMultiSelectableBehavior properties and methods
  00:00 +14: [Firefox] IronMultiSelectableBehavior properties and methods
  00:00 +15: [Firefox] IronMultiSelectableBehavior properties and methods
  00:00 +16: [Firefox] IronMultiSelectableBehavior properties and methods
  00:00 +17: [Firefox] IronMultiSelectableBehavior properties and methods
  00:00 +18: [Chrome] IronResizableBehavior listen
  00:00 +19: [Safari] IronMultiSelectableBehavior properties and methods
  00:00 +20: [Safari] IronMultiSelectableBehavior properties and methods
  00:00 +21: [Firefox] IronResizableBehavior listen
  00:00 +22: [Safari] IronMultiSelectableBehavior events
  00:00 +23: [Safari] IronResizableBehavior listen
  00:00 +24: All tests passed!

finished in 23.01 seconds.

[nex3]

I'm confused. What's the difference between the first run and the second? What's not working as you expect in the second run?

[a14n]

I think it's a dup of #86

[kasperpeulen]

The second one is just the first one with more tests running.

It is surely not as expected, I would think that every test should run on one browser, and only one time on that browser. If I change the order, it also gives different output:

  pub run test:test --reporter=expanded --platform=safari,firefox,chrome,dartium test/polymer.dart
  Compiling test/polymer.dart...
  00:00 +0: [Safari] WebComponentsReady
  00:00 +1: [Firefox] WebComponentsReady
  00:00 +2: [Firefox] WebComponentsReady
  00:00 +3: [Chrome] WebComponentsReady
  00:00 +4: [Dartium] WebComponentsReady
  00:00 +5: [Safari] IronSelectableBehavior methods
  00:00 +6: [Firefox] IronSelectableBehavior properties
  00:00 +7: [Firefox] IronSelectableBehavior properties
  00:00 +8: [Firefox] IronSelectableBehavior properties
  00:00 +9: [Safari] IronMultiSelectableBehavior properties and methods
  00:00 +10: [Safari] IronMultiSelectableBehavior properties and methods
  00:00 +11: [Safari] IronMultiSelectableBehavior properties and methods
  00:00 +12: [Firefox] IronSelectableBehavior methods
  00:00 +13: [Dartium] IronMultiSelectableBehavior properties and methods
  00:00 +14: [Chrome] IronMultiSelectableBehavior properties and methods
  00:00 +15: [Safari] IronMultiSelectableBehavior events
  00:00 +16: [Safari] IronMultiSelectableBehavior events
  00:00 +17: [Safari] IronMultiSelectableBehavior events
  00:00 +18: [Firefox] IronMultiSelectableBehavior properties and methods
  00:00 +19: [Chrome] IronMultiSelectableBehavior events
  00:00 +20: [Chrome] IronMultiSelectableBehavior events
  00:00 +21: [Firefox] IronMultiSelectableBehavior events
  00:00 +22: [Firefox] IronMultiSelectableBehavior events
  00:00 +23: [Firefox] IronResizableBehavior listen
  00:00 +24: All tests passed!

finished in 19.721 seconds.

This is the exact same test as the second one btw. But only the platforms listed in a different order.

[nex3]

@a14n is right: this is a duplicate of #86. See this comment for a long explanation, but the short version is that multiple tests are running simultaneously so the output has to make some compromises.

[kasperpeulen]

@nex3 Do I understand correctly that the problem is only with the output/presentation? I mean, do the test run correctly on the right platform ?

And one other thing I was wondering, is this true for everybody running multiple platorms ? Because I've never heard anyone about this issue. I wonder if there is some workaround. This makes testing on multiple platform allmost unusable. If a test fails, I've no idea on which platfrom it is actually failing, or which test is failing..

[nex3]

Do I understand correctly that the problem is only with the output/presentation? I mean, do the test run correctly on the right platform ?

Yes, the tests are running correctly.

And one other thing I was wondering, is this true for everybody running multiple platorms ? Because I've never heard anyone about this issue. I wonder if there is some workaround. This makes testing on multiple platform allmost unusable. If a test fails, I've no idea on which platfrom it is actually failing, or which test is failing..

If a test fails, the reporter will ensure that the status line immediately above the failure corresponds to the test that failed.

[kasperpeulen]

@nex3 ah okay, that seems like no big problem then, thanks !