qjerome
commented
2 months ago Hi @be-a-panther,
Yes, version 1 is completely broken indeed ! You can find some details about it here: https://www.misp-project.org/2024/03/25/Poppy-a-new-bloom-filter-format-and-project.html/
Basically, version 1 must be used if and only if you need compatibility with format used by https://github.com/DCSO/bloom and related tools using the same format. It is there just for backward compatibility.
Are you experiencing any issue with version 2 ?
be-a-panther
I used this bash script to evalute IPv4 bruteforce for different FPR. The
dataset.txtare two different sets of IPv4 List. I changed it accordingly between the runs.selective information dump:
bruteforce ipv4 script
``` #!/usr/bin/env bash version=1 id=R for p in 0.000000000001 0.00000000001 0.0000000001 0.000000001 0.00000001 0.0000001 0.000001 0.00001 0.0001 0.001 0.01 0.1 1.0; do echo "${p}" #echo "creating filter" # creating a filter with a desired capacity `-c` and false positive probability `-p` #/home/build/.cargo/bin/poppy -j 0 create --version ${version} -p ${p} v${version}/filter_${p}.pop testdataRand.txt #cat testdata_rand.txt | /home/build/.cargo/bin/poppy insert filter_${p}.pop # showing information about the filter we just created /home/build/.cargo/bin/poppy show v${version}/filter${id}_${p}.pop > v${version}/result${id}_${p}.txt echo "bruteforce ipv4" ./ipv4 | /home/build/.cargo/bin/poppy check v${version}/filter${id}_${p}.pop 2>/dev/null | wc -l >> v${version}/result${id}_${p}.txt done ```But I got different results for using version 1 and version 2. I would say, the implementation for version 1 is somehow broken:
testdata Random
version 1: ``` tabs 30; for file in v1/resultR_*; do echo -n ${file}; echo -ne "\t"; tail -n1 ${file}; done; tabs -8 v1/resultR_0.000000000001.txt 2279827179 v1/resultR_0.00000000001.txt 2292496090 v1/resultR_0.0000000001.txt 2307695711 v1/resultR_0.000000001.txt 2279826672 v1/resultR_0.00000001.txt 2297276457 v1/resultR_0.0000001.txt 2319482144 v1/resultR_0.000001.txt 2279912424 v1/resultR_0.00001.txt 2307682256 v1/resultR_0.0001.txt 2348721215 v1/resultR_0.001.txt 2279909914 v1/resultR_0.01.txt 2348745809 v1/resultR_0.1.txt 2541550760 v1/resultR_1.0.txt 0 ``` version 2: ``` tabs 30; for file in v2/resultR_*; do echo -n ${file}; echo -ne "\t"; tail -n1 ${file}; done; tabs -8 v2/resultR_0.000000000001.txt 255104969 v2/resultR_0.00000000001.txt 255105149 v2/resultR_0.0000000001.txt 255105478 v2/resultR_0.000000001.txt 255105775 v2/resultR_0.00000001.txt 255106167 v2/resultR_0.0000001.txt 255106974 v2/resultR_0.000001.txt 255111509 v2/resultR_0.00001.txt 255150265 v2/resultR_0.0001.txt 255524983 v2/resultR_0.001.txt 259186885 v2/resultR_0.01.txt 295735913 v2/resultR_0.1.txt 669617261 v2/resultR_1.0.txt 4294967296 ```testdata 5 IPv4
version 1: ``` tabs 30; for file in v1/result_*; do echo -n ${file}; echo -ne "\t"; tail -n1 ${file}; done; tabs -8 v1/result_0.000000000001.txt 2169947840 v1/result_0.00000000001.txt 2090326774 v1/result_0.0000000001.txt 2300182147 v1/result_0.000000001.txt 2237356373 v1/result_0.00000001.txt 2226226152 v1/result_0.0000001.txt 2134588848 v1/result_0.000001.txt 2162538453 v1/result_0.00001.txt 2273858034 v1/result_0.0001.txt 2350928224 v1/result_0.001.txt 1996217247 v1/result_0.01.txt 2375923684 v1/result_0.1.txt 2614344677 v1/result_1.0.txt 0 ``` version 2: ``` tabs 30; for file in v2/result_*; do echo -n ${file}; echo -ne "\t"; tail -n1 ${file}; done; tabs -8 v2/result_0.000000000001.txt 30 v2/result_0.00000000001.txt 30 v2/result_0.0000000001.txt 30 v2/result_0.000000001.txt 30 v2/result_0.00000001.txt 30 v2/result_0.0000001.txt 30 v2/result_0.000001.txt 30 v2/result_0.00001.txt 30 v2/result_0.0001.txt 30 v2/result_0.001.txt 30 v2/result_0.01.txt 30 v2/result_0.1.txt 30 v2/result_1.0.txt 712969 ```filter size
``` stat -c "%s %n" v1/filter* 1833881816 v1/filterR_0.000000000001.pop 1681058336 v1/filterR_0.00000000001.pop 1528234856 v1/filterR_0.0000000001.pop 1375411376 v1/filterR_0.000000001.pop 1222587896 v1/filterR_0.00000001.pop 1069764416 v1/filterR_0.0000001.pop 916940936 v1/filterR_0.000001.pop 764117456 v1/filterR_0.00001.pop 611293976 v1/filterR_0.0001.pop 458470496 v1/filterR_0.001.pop 305647016 v1/filterR_0.01.pop 152823536 v1/filterR_0.1.pop 48 v1/filterR_1.0.pop 88 v1/filter_0.000000000001.pop 88 v1/filter_0.00000000001.pop 80 v1/filter_0.0000000001.pop 80 v1/filter_0.000000001.pop 72 v1/filter_0.00000001.pop 72 v1/filter_0.0000001.pop 72 v1/filter_0.000001.pop 64 v1/filter_0.00001.pop 64 v1/filter_0.0001.pop 64 v1/filter_0.001.pop 56 v1/filter_0.01.pop 56 v1/filter_0.1.pop 48 v1/filter_1.0.pop ``` I guess that the filter is correctly created, but I didn't check furthermore. ``` stat -c "%s %n" v2/filter* 1836384312 v2/filterR_0.000000000001.pop 1682612280 v2/filterR_0.00000000001.pop 1529872440 v2/filterR_0.0000000001.pop 1376682040 v2/filterR_0.000000001.pop 1223540792 v2/filterR_0.00000001.pop 1070596152 v2/filterR_0.0000001.pop 917385272 v2/filterR_0.000001.pop 764379192 v2/filterR_0.00001.pop 611414072 v2/filterR_0.0001.pop 458494008 v2/filterR_0.001.pop 305709112 v2/filterR_0.01.pop 152834104 v2/filterR_0.1.pop 4152 v2/filterR_1.0.pop 4152 v2/filter_0.000000000001.pop 4152 v2/filter_0.00000000001.pop 4152 v2/filter_0.0000000001.pop 4152 v2/filter_0.000000001.pop 4152 v2/filter_0.00000001.pop 4152 v2/filter_0.0000001.pop 4152 v2/filter_0.000001.pop 4152 v2/filter_0.00001.pop 4152 v2/filter_0.0001.pop 4152 v2/filter_0.001.pop 4152 v2/filter_0.01.pop 4152 v2/filter_0.1.pop 4152 v2/filter_1.0.pop ```