Incorrect match_type and scan_type

[taegeun-moon]

Current Behavior

When finding longest match, data types of match and scan becomes incorrect.

# case 1
Match : 0, Scan : 1
99-01-01</td></tr>
<tr class="odd"><td><input type='checkbox' name='check[]'
99-01-01</td></tr>
<tr class="odd"><td><input type='checkbox' name='check[]'

# case 2
Match : 0, Scan : 0
99-01-01
99-01-01

Expected Behavior

Above, Case 1 indicates a case where the type of the scan is 0 (TYPE_OTHERS), where it supposed to be 1 (TYPE_SECRET) since it is containing the secret 99-01-01. Case 2 shows a case where both scan and match has type of 0, where both should be 1.

[taegeun-moon]

Updating s->types is not working as expected.

[taegeun-moon]

There are still some secrets being matched.

# adminer/select/adminer.php0.nomarkers
99-01-01</td></tr>
<tr><td><input type='checkbox' name='check[]'
99-01-01</td></tr>
<tr><td><input type='checkbox' name='check[]'

99-01-01
99-01-01

99-01-01'
99-01-01'

99-01-01'> <
99-01-01'> <

99-01-01'
99-01-01'

99-01-01' class='edit'>edit</a>
99-01-01' class='edit'>edit</a>

[emp_no]' data-t
[emp_no]' data-t    // [emp_no] is a secret

0'>1003
0'>1003

[taegeun-moon]

Seems like this is because taint_brs is shifted by 8 bytes.

32507 <32627, 32747>
 value='where%5Bemp_no%5D=10050&amp;where%5Bdept_no%5D=d007&amp;where%5Bfrom_date%5D=1992-11-05&amp;where%5Bto_date%5D=99

Where the actual secret is

value='BPBPBPB{where%5Bemp_no%5D=10050&where%5Bdept_no%5D=d007&where%5Bfrom_date%5D=1992-11-05&where%5Bto_date%5D=9999-01-01BPBPBPB}'>

Here, taint_brs is left-shifted by 8 bytes.

This make sense because all the matched secrets' are length of 8.

[taegeun-moon]

The error is accumulatively increasing from 2 bytes to 8 bytes.

Below is a list of continuous secrets and corresponding range.

0 <7558, 7563>  // 2 bytes of error
 'emp_

0 <7766, 7869>  // 2 bytes of error
="/adminer/adminer.php?username=root&amp;db=employees&amp;select=current_dept_emp&amp;order%5B0%5D=dept_

0 <7872, 7907>  // 2 bytes of error
"><span title="char(4)">dept_no</spa
<span title="char(4)">dept_no</span>

0 <7949, 8070>  // 2 bytes of error
='/adminer/adminer.php?username=root&amp;db=employees&amp;select=current_dept_emp&amp;order%5B0%5D=dept_no&amp;desc%5B0%5D

0 <8282, 8288>  // 4 bytes of error
h, 'dep

0 <8491, 8596>  // 4 bytes of error
ef="/adminer/adminer.php?username=root&amp;db=employees&amp;select=current_dept_emp&amp;order%5B0%5D=from_

0 <8599, 8633>  // 4 bytes of error
te"><span title="date">from_date</s

0 <8675, 8798>  // 4 bytes of error
ef='/adminer/adminer.php?username=root&amp;db=employees&amp;select=current_dept_emp&amp;order%5B0%5D=from_date&amp;desc%5B0%

0 <9010, 9018>  // 6 bytes of error
rch, 'fro

0 <9221, 9324>  // 6 bytes of error
href="/adminer/adminer.php?username=root&amp;db=employees&amp;select=current_dept_emp&amp;order%5B0%5D=t

0 <9327, 9359>
date"><span title="date">to_date<

0 <9401, 9522>  // 6 bytes of error
href='/adminer/adminer.php?username=root&amp;db=employees&amp;select=current_dept_emp&amp;order%5B0%5D=to_date&amp;desc%5B

[taegeun-moon]

This problem was caused by UTF-8 encoding.

Left one is UTF-8 file interpreted by Python, which is used for generating brs. Right one is UTF-8 file interpreted by C, our zlib.

>>> b'\xe2\x86\x93'.decode('utf-8')
'↓'

Solution

We should treat multi-byte UTF-8 character as a multi-length character in Python.

[taegeun-moon]

markers_to_brs_dx.zip

FIx markers_to_brs_dx.py to use byte string instead of string.