search

Fantom-foundation / Norma

Integration project of Carmen and Tosca
GNU Lesser General Public License v3.0
2 stars 9 forks source link

[Testing] Test now includes running scenarios under scenario/test #272

Closed rpl-ffl closed 2 weeks ago

rpl-ffl commented 2 weeks ago

@kjezek Thank you for the feedback.

Let me repeat your points, see if I miss anything.

  1. During unit test, we should not run all scenarios because they are too expensive

We are not. We are only running a few selected test

  1. Test to use mocking of nodes/network rather than docker

Ideally we woulld need to mock the nodes, network, etc. i.e. we can test that the nodes, networks, etc. were started/stoped as expected, without actually excercising docker.

Do you mean that when we test scenarios, we should not be actually running the scenarios through docker, but to reconstruct the entire scenarios using mock? If so, as you mention this would be a non-trivial task that is outside the scope of this PR.

  1. Test before Refactoring

I agree with this, and this PR is not meant to be refactoring at all. The only action I can think of here is to revert anything under runner back to driver/norma/run.go, while keeping the functionality the same (e.g. that we execute the scenario/test/sanity-check.yml on each CI run).

I will prepare a fix now, but please do let me know if I miss anything.

kjezek commented 2 weeks ago

@kjezek Thank you for the feedback.

Let me repeat your points, see if I miss anything.

  1. During unit test, we should not run all scenarios because they are too expensive

We are not. We are only running a few selected test

  • scenario > all yml files are subjected to check
  • scenario/test > all files are subjected to both check and run. We currently only target the sanity check scenario as agreed here.
  1. Test to use mocking of nodes/network rather than docker

Ideally we woulld need to mock the nodes, network, etc. i.e. we can test that the nodes, networks, etc. were started/stoped as expected, without actually excercising docker.

Do you mean that when we test scenarios, we should not be actually running the scenarios through docker, but to reconstruct the entire scenarios using mock? If so, as you mention this would be a non-trivial task that is outside the scope of this PR.

  1. Test before Refactoring

I agree with this, and this PR is not meant to be refactoring at all. The only action I can think of here is to revert anything under runner back to driver/norma/run.go, while keeping the functionality the same (e.g. that we execute the scenario/test/sanity-check.yml on each CI run).

I will prepare a fix now, but please do let me know if I miss anything.

The discussion you point at says to run end-to-end tests as part of CI/CD tests, i.e. it should create a pipeline that runs a test scenario and makes sure it works. It does not have to be re-work or addition to current unit tests, i.e. I would not waste time on this at the moment. Once there is time, I would suggest to do the bigger task with mocking docker and running all scenarios to make sure they work. At the moment I would work on the pipeline and make sure it runs in Jenkins.

rpl-ffl commented 2 weeks ago

deprecated by https://github.com/Fantom-foundation/Norma/pull/279