Dynamic analysis

[mvondracek]

We need to use 2 tools for dynamic analysis. We should either find 2 tools for dynamic analysis of Python or implement C extension module and then use 2 tools for dynamic analysis of C for that code.

related #4

[lsolodkova]

It depends on what do you mean by the dynamic analysis. Could we just generate some input with radamsa and feed our script with it?

[mvondracek]

I think if we configure radamsa to generate inputs similar to patterns of valid ones (not just completely random), that would be one tool. :)

[lsolodkova]

Do we need performance profiling? Also if we have something to measure code coverage (and we have), it'll be enough.

[mvondracek]

I talked to doc. Švenda today and I described our current setup (CI, coverage, static analysers, ...). He confirmed, that including fuzzers (e. g. radamsa) would be good.

We also talked about fuzzing with @sobuch after today's lecture. It's not suitable to fuzz every commit on CI. We should have simple script, which configures fuzzer(s) and runs against our tool for long time. We could then run the script manually on our machines. Also, we could configure CI to run fuzzing on builds which are less frequent and are not affected by testing delays due to fuzzing -- release builds. This way, when code in dev branch is stable and working, we create release branch (according to git flow) and our CI will start long testing with fuzzers on this release. This approach does not delay development on dev branch. EDIT: Fuzzers on Travis CI could run for 50 minutes for each build. 😄 👍

@lsolodkova Can you please prepare shell script which would install radamsa for Ubuntu and another which would configure radamsa and run against our tool? In this case against mnemoniccli after installation of our tool.

[lsolodkova]

@sobuch May I ask you to look on that? I'm afraid I'm out of time this week.

[sobuch]

I will look into it

[mvondracek]

@sobuch, @lsolodkova, any updates?

[sobuch]

not yet, though according to https://gitlab.com/akihe/radamsa for installation should suffice following: 'sudo apt-get install gcc make git wget && git clone https://gitlab.com/akihe/radamsa.git && cd radamsa && make && sudo make install' is that enough for installation or do we need some other format perhaps?

[mvondracek]

I think that's fine for our development computers and also for Travis.

[sobuch]

Hi all! Please checkout https://github.com/mvondracek/PA193_mnemonic_Slytherin/tree/task-39-fuzzer/fuzz , I prepared simple fuzzer and would like some feedback, for example, what should count as failure? for now i decided that is if the return value is not 0(ok) or 65(invalid fails provided)

[sobuch]

added code 125 as valid for verify operation.

[sobuch]

This is first (about) hundred fails - https://pastebin.com/ABZLLwT2

[lsolodkova]

Awesome! And how to find the source of each problem?

[sobuch]

if you run the script yourselves, it also saves the input data, then you can reproduce it. As far as I can tell all these fails were due to invalid format of input data. For example, 'Odd-length string' is the output when the inputfile is in hexadecimal format and has odd length

[mvondracek]

Hi all! Please checkout https://github.com/mvondracek/PA193_mnemonic_Slytherin/tree/task-39-fuzzer/fuzz , I prepared simple fuzzer and would like some feedback, for example, what should count as failure? for now i decided that is if the return value is not 0(ok) or 65(invalid fails provided)

I would say that every exit code specified in mnemoniccli.ExitCode except for UNKNOWN_FAILURE = 1 is success for fuzzing. If our program gets error that it did not expect, it does not crash, but prints our error message and ends with 1. This is implemented for all built-in, non-system-exiting exceptions. It's not completely unhandled crash with traceback. But we should identify all errors that we would like to handle diferently. For example, when input mnemonic contains invalid UTF-8 sequences, it's better if we return EX_DATAERR = 65 instead of UNKNOWN_FAILURE = 1. This is just better UX, because we alredy display error message -- as far as we don't crash. :)

Except from radamsa, we will have to write tests for various edge cases like... how our program behaves when it has limited resources.

@sobuch radamsa setup is exactly what we needed, thanks.

• Please add some functionality which would write something (e. g. .) to script's stdout every few seconds (iterations). For integration to Travis, the script has to print something at leas every 10 minutes. 😄
• When radamsa detects error, could it print some info to stderr, too?
• For better detection of source of error, you can enable logging (-ll debug) and save log with test runs which had errors.

[mvondracek]

Added #51 and #52.

[mvondracek]

Related #53