Quality Assurance

[MarkoPaasila]

What can and should we do to ensure quality? Of primary importance is making sure keys are secure and that strategies do nothing stupid. This isn't any fun, but it really is very important. Of secondary importance is software stability.

Here's my idea for ensuring strategy sanity

1. A strategy with the sole purpose to do all kinds of stuff in the specified market. Stupid things, normal things, insane things, unexpected things. This would be used to create conditions that indirectly stress-test a strategy that's running in another bot. After all, unanticipated market actions are what most likely will trigger unexpected behavior in the production case. This strategy could include sending assets to another account - the account of the strategy to be tested. This, or even several instances of it, should be run in a separate instance of DEXBot to maximize "load". It could be limited to cli. Creators of strategies should think out how to manipulate their strategies to make a loss, and how to crash them; then add those procedures to this stupidity-strategy. Indirect testing like this will cover all future strategies without knowing exactly what the strategy to be tested is supposed to do.
2. A method which instantiates a strategy, then arbitrarily pauses it, resumes it, stops it, causes a connection error, restarts, etc, all pseudorandomly for as long as it is left on. This could be used separately or in conjunction with the crazy-insane-strategy.

This won't cover key security stuff, and I don't have any ideas for that.

[cloud-8]

QA is more than just thinking about technically how to make the bot do things, a proper QA process is required if DEXbot wants to have production ready software. To adopt best practices you need to look at it from an entire product perspective, there are plenty of blog posts on the internet about 'QA best practice process'. It includes much of the following steps:

• Identify current product scope- needs of users, how they interact with it, what are the workflows for user interaction, how specific functions of the product work with these interactions.
• Formulation of a test process document. Document every functionality check/test and assign severity of failures. Document should be a checklist for every time a development release is rolled out for which regression testing is performed.
• Investigate existing and past issues and attempt to replicate them on every platform as per the test document. If not enough information is required update the github template.
• Convey test cases appropriately back to the development team. This would likely take the form of Github tasks.

[PermieBTS]

Rest assured QA processes are being taken seriously..

A dedicated QA dev and a Shadow QA dev is being sought to ensure thorough QA. A test process document is being written to make the process transparent and to make sure it is repeated for every release of DEXBot

[MarkoPaasila]

Should the checks include separately testing each feature, installation using all supported methods and operating systems? I don't think the checklist should be so strict that it inhibits us from making releases, but it should make sure we never go backwards. Every release should be better than the previous.

So we should document (wiki) the QA procedure and prepare the checklist. Every new feature should be added to the checklist - with the result that the time testing takes will always increase.

Before each release testing should be done according to the procedure and any new bugs reported. This should be repeated after any change / bugfix, until all tests pass. Only then can a release be made.

[thehapax]

Should the checks include separately testing each feature, installation using all supported methods and operating systems?

Yes it should cover all new features and all platforms. I think we've gotten better but we need to do better to ensure that really egregious bugs don't show up. Right now for someone without technical skills the command line barrier is still too high for the software to be adopted.

If we have a dedicated QA working on a wiki and prepping the checklist as well as giving feedback to the team and listening to the public, that's already more than half of the solution.

I'm asking around in my communities if anyone is open to picking up this work, hopefully there will be some interest.

[PermieBTS]

I've found a potential dev who might be able to tackle some of this