search

Cobular / distest

A library used to do integration testing on discord bots
MIT License
29 stars 8 forks source link

Add setup/teardown functions to the TestCollector #17

Open Cobular opened 5 years ago

Cobular commented 5 years ago

This is an important part of most testing libraries, so we should find a way to do it too! I want to plan how we will do this for a bit first, then we can make the PR and start working on it.

https://github.com/JakeCover/distest/blob/76a278b987ec4abb3ae023774f251362fbe0dff8/distest/bot.py#L31-L47

I'm pretty sure we just need to add a call to the up/down functions that should be run every test here (these could be used to reset databases or otherwise clean the slate). We would also need functions that can be run at the start and end of the overall program, which could probably be accomplished by adding some logic here, before and after the testing for loop runs.

https://github.com/JakeCover/distest/blob/76a278b987ec4abb3ae023774f251362fbe0dff8/distest/bot.py#L66-L74

Does that seem reasonable? This week is really bad for me, I have an SAT this weekend, but I can work on this next week for sure.

JosephFKnight commented 5 years ago

I guess I'm unclear on what exactly needs to be setup/tore down. What did you have in mind as the actual purpose? It's my understanding that setup/teardown is used for unit testing when you need to simulate an environment to test a function. However, our library is for application testing. It simulates a user calling commands to a bot, therefore it tests the application of the program and not the code itself. In fact, this library exists to solve the problem of a discord bot being inherently non-unit-testable. The nature of a discord bot is that it must be tightly coupled to the Discord API.

Now, an alternative solution to the problem our library attempts to solve would be designing a testing framework that simulates a discord environment and runs the bot in that, allowing for asynchronous unit testing. This would be a substantial undertaking, as we would need to simulate the entirety of the Discord API (correctly and realistically). In fact, I stumbled across DXsmiley's original lib while looking to see if anyone had done this already, as I am woefully ill-equipped for that task.

Good luck on your SATs. Study hard!

Cobular commented 5 years ago

Thanks! I mean to say that if the bot is being tested and it is stateful in some way (ie user accounts or server settings or something) you may want a place to run code to reset this before/after run or between tests.

For me, I want a way to send a fin message of some kind to have the example tester exit itself. This isn't an issue in the cloud, as the testing container is killed when the tests finish anyway, but on my laptop I have to go and explicitly kill the example_target.py background process every test or I run into issues. With function that runs when the tests are done, I could send one last message that would fire at the end of the tests, no matter how many tests are added, that the target could listen for and just run sys.exit() when it hears that. With my other bot, ReplyBot, the setup/teardown would make a connection to my database and clear that out between tests or restore it to a state that is known to be sure that the conditions are reproducible.

Does that make sense? I get that this is application testing, but the setup/teardown would mostly be to interact with external things (databases, the OS, discord API), not the target bot directly (would not call methods or set variables on the target).

I really don't want to emulate the API, I came to and refreshed the DXsmiley thing for the same reason as you.

Cobular commented 5 years ago

I'm moving this off 1.0, we may do this later since I just want to get this out soon.