search

shaka-project / shaka-packager

A media packaging and development framework for VOD and Live DASH and HLS applications, supporting Common Encryption for Widevine and other DRM Systems.
https://shaka-project.github.io/shaka-packager/
Other
1.95k stars 505 forks source link

test: fix TTML integration test where ordering was changed by #1364 #1379

Closed cosmin closed 6 months ago

cosmin commented 6 months ago

We should turn on integration tests in CI now that they are stable

joeyparrish commented 6 months ago

We should turn on integration tests in CI now that they are stable

It should just be a matter of adding it to CMake, right?

cosmin commented 6 months ago

It's currently skipped in CMake by SKIP_INTEGRATION_TESTS which is ON by default. We could add a separate job just to run the integration tests, either by rebuilding with -DSKIP_INTEGRATION_TESTS=OFF or by directly invoking packager_test.py

cosmin commented 6 months ago

We could also turn them on by default although given the tests take a while to run I think it's convenient to get early feedback from running the unit tests without integration tests.

joeyparrish commented 6 months ago

I bet changing the default won't seriously increase the job time in CI. I'll do a local test to ballpark it.

joeyparrish commented 6 months ago

Clean, configure, build, and test w/o integration tests on my workstation at home: 4:49

Clean, configure, build, and test w/ integration tests on my workstation at home: 5:38

An increase of 49s

Clean, configure, build, and test w/o integration tests in fastest CI configuration: 5:47

Clean, configure, build, and test w/o integration tests in slowest CI configuration: 18:07

I would guess that turning it on by default wouldn't delay the last test result by more than 1-3 minutes.

cosmin commented 6 months ago

I can send a pull request to flip this on by default, although the test times on your workstation seem concerning. On my laptop running

ctest --test-dir build -j 8

runs in 18s

cosmin commented 6 months ago

Ah, this includes a full rebuild of everything including all the 3rd party dependencies.

Something still seems odd though, for me the integration tests take about ~5 minutes to run on their own, because the default unit test runner in Python won't parallelize the tests. I guess the difference could come down to single core performance.

cosmin commented 6 months ago

sent #1381 to turn on the integration tests and see how it looks in CI