Test Coverage of Ludwig

[ciphor]

I'm not seeing a lot of unit tests in this project, should this be a concern? How do we know it is still working as expected after making changes?

[w4nderlust]

It's a young project. version 0.1.0, it will be susceptible to a lot of changes in the next few next versions. Over committing to too many test at this early stage may hinder the speed at which we can improve the project. That said we definitely want to increase the number of test down the line, but if you look closely at what the tests do, at the moment more than 100 combinations of input and output features are covered. You are welcome to help in improving the test coverage, in which case we can use this issue to talk about how to do it and track progress.

[w4nderlust]

Right now Ludwig has more than 800 tests, with a test coverage of about 70%. We still want to improve the coverage, but the current suite, despite not covering all the corner cases, is quite robust. We welcome help from the community to increase the coverage and number of tests.

[TuanNguyen27]

@w4nderlust could you suggest a few places in the code base that could benefit from more unit tests ? Thank you ! :)

[w4nderlust]

@w4nderlust could you suggest a few places in the code base that could benefit from more unit tests ? Thank you ! :)

Thanks for being willing to help @TuanNguyen27 ! We'll release v0.3 shortly, and then I'll get back to you with some precise info. i want identify the areas more in need with a pytest run coverage and use that to decide together where you can help the msot ;)

[jimthompson5802]

@TuanNguyen27 I'm reaching out on behalf of the Ludwig project. I'm a contributor and will be looking to improve the project's unit tests. I'd love to have help with this and discuss possible ideas with you. Let me know if you're still able to help with this.

[TuanNguyen27]

@jimthompson5802 Yes i am! How can we proceed ?

[jimthompson5802]

@tuannguyen27 Here is my perspective of where we can improve the unit tests:

• With the exception of some recent tests, most of the tests, when writing out results, use the same output directory name. This is not necessarily an issue except if we try to run tests in parallel. For these tests, we'd like to put the output in a temporary directory. This way it will avoid conflicts.
• Many tests run multiple scenarios, i.e., combination of parameters. These multiple scenarios are hard coded in the the actual test itself. Where possible, we'd like to externalize the combination of parameters with the @pytest.mark.parameterize decorator.
• Right now the entire suite takes about 30 minutes to run. It would be great if we can reduce the time. Some thoughts on reducing the time:
  • Some tests, which is focused on results of training, often train their own model when they could share a single trained model to test the results, thus saving the time for training over and over.
  • A few tests take a shotgun approach to testing different scenarios. I'm thinking if we can take a more surgical approach we can offer the same coverage with less effort.

Hopefully, this provides you a view of where we can improve the tests. Let me think about how I can provide you details on the above. I'll be back in touch in a day or two.

If you have any thoughts on unit tests, I'm open for discussion.

[w4nderlust]

@TuanNguyen27 to give you context, @jimthompson5802 wrote many of the tests in v0.3 and he is definitely the authority on tests in Ludwig :) Also if you want we can add you to our chat so that alll of us in the core team can answer your questions as they emerge. Feel free to send me a private email and I can add you there.

[TuanNguyen27]

@jimthompson5802 i think all of your suggestions make a lot of sense and are doable for beginners to the code base. I'm most interested in the 3rd bullet point, refactoring some test to reuse previous trained model results. Could you give a few specific tests in https://github.com/uber/ludwig/tree/master/tests that I can read more?

A few tests take a shotgun approach to testing different scenarios. I'm thinking if we can take a more surgical approach we can offer the same coverage with less effort.

Wondering if you can also give an example of what you mean by "shotgun approach" ?

[jimthompson5802]

@TuanNguyen27 good to hear back from you. We can discuss the specifics in the chat platform that Piero(@w4nderlust) set up for the Ludwig project. Piero offered to add you to the chat platform if you send him a private email with your email address. This is where we are coordinating our work. You can get Piero's email address from his GitHub profile.