search

Qucs / ADMS

ADMS is a code generator for the Verilog-AMS language
GNU General Public License v3.0
94 stars 32 forks source link

Fixup afl #16

Closed nplanel closed 8 years ago

nplanel commented 9 years ago

Fixup discovered with AFL "american fuzzy lop" tool

guitorri commented 9 years ago

Thank you for looking into this. I was not aware of AFL. I was following the discussion about qucsator. Would you mind commenting on how exactly should I configure and use AFL? One just instrument the binary and run it? How many times? Can a testsuite be used to improve the branch coverage? I will try to use it in the future as new things are added to the parsers. It would be helpful if I could reproduce what you just did.

nplanel commented 9 years ago

AFL is pretty easy to use but I done these steps, basically to optimize running time without changing the code.

1/ build the source with clang via afl-clang and afl-clang++ (overwrite the CC and CXX env) => that will help AFL a lot

For ADMS only 2/ create a quick dictonary/vobabulary based on va files (this step could be optimized)

mkdir va-dict ; cd va-dict ; find $qucssources -name ".va" | xargs cat | perl -pe 's|^//.||g' | perl -pe 's|^\s(.)|\1|g' | sort -u

3/ create an input directory (in) and copy all you .va files

4/ launch AFL, I used this command line for ADMS

afl-fuzz -t 2000 -i in -o out -- admsXml @@ -e ./analogfunction.xml

Some remarks : o you may need to remove some .va file as there are too long to process o 2000 ms (-t) is timeout value for the execution, it's quite long

For qucsator I just test the netlist checker (-c)

It will be quite difficulte to integrate AFL in your testsuite as the tests could we very/very/very long (in days, even more) But an automatic weekly build script and test, with some CPU/hosts dedicated to running this test, could be a good catcher.

For your testsuite, I advise you to compile you test suite with ASAN, compile and link with clang -O1 -g -fsanitize=address -fno-omit-frame-pointer By this way you will be aware of bad memory access ; ASAN is quite faster than valgrind and more accurate as it's during the instrumentation of the code.

You can mix Address Satinizer (ASAN) and AFL but it will slow done the execution.

To conclude, you may need to write a sample test to test only a portion (like only the parser) with AFL. Execution will be faster and you can validate the API, function per function

I don't known if you have some dedicated servers or VMs to do that at long term.

Contact me directly if you need more info / help Qucs is a quite good project

BTW: Do you have any roadmap, feature that you need coding help ?

in3otd commented 9 years ago

Thanks for the detailed explanation on how to use AFL, I was about to contact you directly about this, but surely it's better to keep the discussion public.

Regarding the help needed here, well, there are a lot of things that need to be done. For Qucs, there is a roadmap here. As shown there, at present the main focus is (should be) finishing porting the GUI to Qt4; Qt3Support is still used in several places and we need to get rid of that. Then the schematic editing should be ported to QGraphicsView, @guitorri surely knows this area better and can comment further. I would be great if you could help with this; if you prefer to work on other topics in the roadmap just let us know; as you will see, there are plenty of topics to choose from...

guitorri commented 9 years ago

There is plenty to do on Qucs. There is no clear roadmap for ADMS at the moment. There are some features I would like to diff/clean/test/merge from the upverter-features branch. We first need to resolve the issues around the .vams headers copyrighted by Accellera. They do not allow modification, which are incompatible with the license. This is actually blocking us to be back on Debian. I will open an issue to keep track of this.

guitorri commented 9 years ago

Hum, strndup is not available in MinGW (and MSVC). The build is failing on Windows.

in3otd commented 9 years ago

yeah, as noted here and here it seems strndup might not be in MinGW :confused: @guitorri , will we support MSVC? I saw you closed Qucs/qucs#239 ...

guitorri commented 9 years ago

@in3otd I thing it is important to know that the code is portable across compilers. You might have noticed that I added Appveyor support for ADMS. It will from now on build regularly on Windows/MinGW. I can try to add MSVC. I took down the PR because I wanted to rebase and rename the branch for Qucs. It is still in my fork. When I have more time I will give another try on supporting MSVC.