search

ipfs-inactive / dev-team-enablement

[ARCHIVED] Dev Team Enablement Working Group
3 stars 1 forks source link

Should we have steps per OS on Jenkins #146

Closed hugomrdias closed 5 years ago

hugomrdias commented 5 years ago

Right now we have one stage for tests that runs all the OS's in parallel but when we need to restart it restarts all the test in all OS's.

Wouldn't be better one stage per OS so we can restart only the OS that fails the tests (normally windows ^^ or mac when it runs out of space) ?

@ipfs/javascript-team pls give your feedback about this. Thank you.

vasco-santos commented 5 years ago

Ideally, it would be better to run the tests in parallel, as they would run faster.

However, in our current state with all the flaky tests we have and taking into consideration that I mainly have test failures in a single OS, I agree that the best experience for me would be to have a different stage per OS.

achingbrain commented 5 years ago

I think one-step-per-os was how it was supposed to be at first https://github.com/ipfs/jenkins-libs/pull/35

hugomrdias commented 5 years ago

@achingbrain i understand that, but on your daily flow which one do you think its better for you ?

victorb commented 5 years ago

The way I see it for one-step-per-os:

hugomrdias commented 5 years ago

The last one is really a Con ? for me as a dev trying to make jenkins green its the same.

Can we make browser tests run only on linux ? we don't really need to run on mac or windows as we are not testing safari or edge.

victorb commented 5 years ago

@hugomrdias yeah, first one would be a con in the example of js-ipfs. Imagine we run tests in the order of linux > windows > macos and tests passes everywhere except in linux, then it still has to rerun the tests for windows and macos. Or maybe we can save results of tests somehow, hmm...

Can we make browser tests run only on linux?

We could, but I much rather make CI run tests with Safari and Edge now when we have AEGIR_BROWSERS for configurating which browsers to run.

hugomrdias commented 5 years ago

can't we make steps dependent from the previous step ? if linux fails nothing will run after it. Or save results somehow that would be perfect!

We could, but I much rather make CI run tests with Safari and Edge now when we have AEGIR_BROWSERS for configurating which browsers to run.

Agree! but i have a feeling that will be a rabbit hole really deep :)

victorb commented 5 years ago

can't we make steps dependent from the previous step ? if linux fails nothing will run after it. Or save results somehow that would be perfect!

That sounds worse. Even if a test in linux fails, we want to continue run tests on macOS and Windows, as there could be actual failures that only happens on macOS, so you want to know about it.

hugomrdias commented 5 years ago

And we would, after Linux passes and windows passes. Maybe I'm not explaining my self correctly, let me try again.

If npm install step fails the pipeline will not start the next step right? Because it would definitely fail. If Linux step fails because one test fails why would we start macos step and waste resources it will probably fail the same test. If the macos step fails because it ran out of space, we could start the windows step but if we wait the same way as the previous example it's the same thing because we need everything green. So another example, Linux passes, macos starts and fails, windows doesn't start because macos failed. Now we go to IRC call Victor saying macos has no space, Victor quickly fixes the problem, we restart macos it passes and windows starts and passes. This is usually how pipelines work on others CIs one step doesn't start if the previous has failed, and if it passes after a restart the next step automatically starts. Makes sense?

daviddias commented 5 years ago

@hugomrdias Can you consolidate this with the CI exploration thread?

hugomrdias commented 5 years ago

yes we can! this can be closed other CIs don't have this problem