Towards automated testing (long term)

[luis261]

This project could benefit from a suite of automated tests (at different levels, mainly system-level/integration tests but also unit tests for in-depth testing the behavior of the analyzer modules)

I have quite a bit of unit testing experience, specifically in Python with unittest/pytest. Also got some CI pipeline tinkering under my belt (mainly Azure DevOps/GitLab, but GH actions should be manageable too).

I think I might start with some very high level, simple integration testing via a bash script and then some unittest code coverage for the bearing constructs/fiddly analysis logic of some of the modules I'll be working on.

The difficult part will be the malware samples later I think, if and when I ever get to that point. Also getting the project installed on the CI runners properly might not be that simple considering its dependencies. Until then the tests will still be local, albeit (partially) automated.

[!NOTE] @CYB3RMX this might stay open a long time, I just wanted to have it documented as an issue so I/we don't lose track and can properly gather ideas over time. Realistically speaking, we're sadly talking months/year(s) until I'll reach a mature CI suite for this project in particular, depending on how much mental bandwith my other responsibilites will occupy, I hope that's ok.

[luis261]

I think I'll just start off by establishing a test directory which simply provides a simple comparison script (unixoids only for now) that checks out master, runs a sample, then checks out the target branch and checks the diff in terms of stdout and stderr, so stay tuned for that.

This simple, heuristic approach should prove quite effective at identifying the more obvious errors considering the way this program writes to the console.

[luis261]

more ideas:

• I see potential for employing fuzzing techniques to some extent
• would be nice to test against a curation of real-world samples
  • could be manually curated or fetched from some sort of feed?
  • we'll have to be careful not to trigger detections on the CI runners 😈

[luis261]

@CYB3RMX if you could go ahead and assign this to me (:

[luis261]

I think I'll just start off by establishing a test directory which simply provides a simple comparison script

btw, here's what such a script could look like (a preliminary draft) - "base_cmp_stdout.sh":

#!/bin/bash

# TODO add informational echo messages

# [ARRANGE]
# TODO parameterize the actual invocation that gets compared, either via config or cmd line args

# [ACT]
# TODO need to ignore the ASCII art header, maybe it might
# make sense to introduce a --no-header option for that purpose?
python3 ./qu1cksc0pe.py --file /usr/bin/ls --analyze > ./current.stdout.txt
git checkout master
python3 ./qu1cksc0pe.py --file /usr/bin/ls --analyze > ./baseline.stdout.txt

# [ASSERT]
diff --text --ignore-all-space ./baseline.stdout.txt ./current.stdout.txt
# TODO actually check result and exit accordingly instead of just emitting it
# TODO also diff any report files that might be present?

# [CLEANUP]
# TODO need to store the current branch above and check it out again here
# TODO cleanup files

[luis261]

@CYB3RMX if you'd provide me with a bit of feedback to simply me know whether or not you're open to the pursuit of automated testing that would help me a ton (: Ideally, it would include a merge (or rejection) of #65 so I can continue working on the integration testing frontier.

Again, this is all part of the plan initially laid out by me, enabling us both (and others) long term when it comes to improvements and new features of the actual analyzers (without impacting stability for existing users.

[CYB3RMX]

Hey @luis261! This idea is really great, but for now, we should focus on how to make the analysis/detection modules and techniques more effective and better, as malware continues to evolve day by day, so this part can wait for now :)

[luis261]

Hi @CYB3RMX, I appreciate your response! If you're completely set on this issue I'll drop it and just close this and #65 as well.

However, as previously explained, having tests will facilitate further development in terms of new features as well. Clearly, judging from past comments of yours, stability for end users is important to you, which automated tests would ensure.

To name a concrete example, testing made up the majority of the overhead we had to overcome to get #58 shipped (admittedly, that was only a refactoring change but it still illustrates the point).

I get that the idea(s) laid out here may sound a bit too ambitious but if you will support me in working on this I assure you over time we can get to a point where the tests greatly help us in developing this project. Just merging #65 would be a good first step towards using automated tests to help us in the development process (I already used that script for verifying past code changes btw and it does help in trying to reduce the effort of fully manual testing).

Also, this project has a "critical mass" of features anyway to the point where I do think that internal improvements (without adding new features all the time) can be worth it (based on my past experience of working on other projects of similar size), idk maybe you disagree.

If I don't hear something from you soon, I'll close this and abandon related efforts, just to make it easier for us both, no hard feelings (:

[luis261]

Alright, in the absence of a response I'll assume you don't care for this.

However, I do hope we can revisit this at some point. In my experience automated testing is really the only reliable way of assuring quality ...