Possible Text and Parser Improvements - Githubissues

MichaelAquilina / SpamFilter

Classification of emails using machine learning and natural language processing techniques in Java

5 stars 4 forks source link

Possible Text and Parser Improvements #21

Closed MichaelAquilina closed 10 years ago

MichaelAquilina commented 10 years ago

Note: all examples given are from actual data extracted

Parser Improvements

Remove base64 encoded images (Fixed) Examples

0xbmmhreayfp6zfcxggfiw4wo0loemcrk00gtjdyqb0vzbvf3uw2kx7zfjqtmc/tx8r23l+kuiow
9dvmg1tvgwxlnxtaidjagytmzjuu1tmbpdbyk9wr1o5vmwaakd7ojkc8po6s0rwtofwxdvbg

Improve detection of mailto items (Fixed) Examples

mailto:geoposti@uole.com?subject\u003ddelete-mort
mailto:one@aeopublishing.com
mailto:kiall@redpie.com

Separate href, name, src etc... from actual value (Fixed) Examples Note that \u003 represents '=' in the data below

href\u003http://www.internalmemos.com/\u003emor
href\u003dhttp://www.kaplanindex.com/memos/memodetails.php?memo_id\u003d1019\u003eexodu
src\u003d3d\"http://www.zdnet.com/include/ads/j
name\u003d\"title.gif
Text Processing Improvements

Detect formatted numbers (Fixed) Examples

500,000,000
610,470.00

Detect Date and Time Examples

06/29/01
3:36
3:00
12:31:56

Ignore file names / file paths Examples

css/style.css
usr/src/linux-2.4.18-3/linux/modversions.h
krb5-libs-\u003cv.e.r-no\u003e-i386.rpm

xhochy commented 10 years ago

I think we have most of this stuff integrated. No more or all feature selections graphs will be screwed.