search

marian-nmt / amun-regression-tests

Regression tests for Amun
0 stars 1 forks source link

Run all regression tests on Jenkins #12

Open snukky opened 6 years ago

snukky commented 6 years ago

I have lightened most tests by decreasing mini/maxi-batch sizes and reducing the length of test inputs wherever appropriate, but at the moment, only a few CPU and GPU tests are run on Jenkins.

However, some tests generate wrong textual output for me. @hieuhoang Does all tests work for you?

snukky commented 6 years ago

Mini/maxi-batches have been reduced because on Jenkins machine there is usually only 4GB GPU memory available.

hieuhoang commented 6 years ago

2 tests seems to be failing ensemble.return-nematus-alignment ensemble.return-soft-alignment

Hieu Hoang http://statmt.org/hieu

On Mon, 17 Sep 2018 at 22:12, Roman Grundkiewicz notifications@github.com wrote:

Mini/maxi-batches have been reduced because on Jenkins machine there is usually only 4GB GPU memory available.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marian-nmt/amun-regression-tests/issues/12#issuecomment-422173453, or mute the thread https://github.com/notifications/unsubscribe-auth/AAqOFDN_1U4MzJTIrBwMD7FWM6aNLJqiks5ucBBUgaJpZM4WsuiU .

hieuhoang commented 6 years ago

these 2 as well test_en_de_gpu.return-nematus-alignment/ test_en_de_gpu.return-soft-alignment There is textual difference

hieuhoang commented 6 years ago

Probably my fault.haven't checked reg tests in a while. Will take a look

snukky commented 6 years ago

OK. Jenkins jobs work now properly. @hieuhoang To add more tests to Jenkins (which we should do), just update TESTS_CPU and TESTS_GPU variables in the main Makefile.

hieuhoang commented 6 years ago

cheers. so only 3 GPU test pass on Jenkins? that's weird.

Is the GPU particularly old, with only 4GB? I don't have access/never used Jenkins.

Most tests pass on my GTX1080 (apart from 4 mentioned above), I'll make sure all tests pass when I have time. Then we will go from there

snukky commented 6 years ago

No, I added only 3 tests to be run on Jenkins to check if everything is configured correctly. I wasn't sure which tests are designed for GPU, and which for CPU as only some tests have gpu/cpu in their names. Maybe we should run all tests on both CUDA and CPU-only amun?

snukky commented 6 years ago

On the Jenkins machine these tests fail:

tests/ensemble.return-nematus-alignment/err:make[1]: [test] Error 1 (ignored)
tests/ensemble.return-soft-alignment/err:make[1]: [test] Error 1 (ignored)
tests/large-model/err:make[1]: [test] Error 1 (ignored)
tests/test_en_de_gpu.return-nematus-alignment/err:make[1]: [test] Error 1 (ignored)
tests/test_en_de_gpu.return-soft-alignment/err:make[1]: [test] Error 1 (ignored)
hieuhoang commented 6 years ago

Yep. They fail locally too. Will look at out next week