Stemming returns values like `"___________________________________________________________________________________"`

[Hugo-ter-Doest]

Are you using the Porter stemmer for English? Can you give some examples?

[niftylettuce]

Yes, here's an example:

String passed:

 _   _ _____ _  __  | \ | |_   _| |/ / _ __   __ number_wquctgtkut - number_wquctgtkut - number_wquctgtkut _ o join! mail an empty message to |  \| | | | | ' / | '_ \ / _ \ \ /\ / / o  email_imkxepvqcp  | |\  | | | | . \ | | | | (_) \ v  v /  o website (+ archive) lives at: |_| \_| |_| |_|\_\|_| |_|\___/ \_/\_/   o      url_dzdepyvagh            "A case in point is web designer Matt Jones, the man           responsible for how BBC News Online looked when it launched.           Since then, he has invented 'warchalking', which he recently           described as a 'curse'..."                                url_dzdepyvagh                    - but you cannot turn against me! I... created you!                                   >> HARD NEWS <                                stiffening sinews           More hot summer days in the mailinglist alleyways,          dangerously empty of sane postings, strewn with the rotting          carcasses of broiling vacation messages. Hacktress and          Silicon Valley's chief rat-keeper LILE ELAM, excitedly posts          about a new open  number_wquctgtkut  network she's found. "I am here at          the police station waiting to see a judge and I thought I          would check to see if there is connectivity", she writes,          somewhat recklessly, to the Bay Area Wireless list. Exit          the rest of the Wifi community through the nearest window          and out into the streets... where, cooling tempers, the          Microsoft Palladium boys are on an endless summer tour,          reassuring the experts that while, hmm, they *suppose* Pd          could *theoretically* be used as a Hollywood DRM system,          they truly have no plans to do any such thing. Cypherpunk          and friend of freedom Lucky Green hears this; thinks up four          or five of the obvious Palladium DRM implementations; sends          them off to be patented in his name. Licensing funds, we          imagine, will go on cracking his own DRMs. And so the mail          loops on.           url_dzdepyvagh                                   - administrivia: HI MOM, I'M IN JAIL     http: email_imkxepvqcp                               - Green/Palladium, like Green Kryptonite           Could Lucky get himself arrested under the DMCA for          distributing a circumvention device? Worse: now we have the          EUCD incoming, could he here in the UK? Will Alan Cox go to          jail for posting detailed Changelogs? Will even the nicest          UK cryptographer (or curious garage tinkerer) find          themselves hauled up under our new and scarily DMCAish          copyright regime? Find out the facts at the free FAIR DEAL          FOR COPYRIGHT conference, organised by the irrepressible          FOUNDATION FOR INFORMATION POLICY RESEARCH for Wednesday           number_wquctgtkut - number_wquctgtkut - number_wquctgtkut  at the London School of Economics. All the usual          fun from the creators of the Scrambling for Safety crypto          cons: we confidently predict government spokesmen caught in          headlights, wanton Dave Bird heckling, some industry bigwig          fighting off the audience with a broken chairleg, and other          epiphenomena of the interzone between legal minds and hacker          ethics. Oh, and FIPR are still looking for a Programme          Director, so if you're interested, let them know. We          suggested a convention raffle (first prize: the director's          job, second prize: Ross Anderson as your personal slave for          a day). They say there's some rule that would break, though.           url_dzdepyvagh  - doesn't the Foundation use psychohistory for filling these positions?           For those of us who can't read the abbreviation EULA without           thinking of Martian fighting machines and their "deafening           howls... which roared like thunder", we're sorry to report           that this weekend's multimedia performance of Jeff Wayne's WAR           OF THE WORLDS has been postponed due to "health and safety           issues". The event was to feature computer graphics,           fireworks, " number_wquctgtkut ft-tall Martian fighting machines" wreaking           "havoc and destruction", and - most terrifyingly of all - the           possibility of a David Essex tribute singer performing with           Hawkwind, but UKP number_wquctgtkut  tickets for the Sat  number_wquctgtkut - number_wquctgtkut - number_wquctgtkut  show at           Manchester's Heaton Park will still be valid at a range of new           venues next summer. Ironically, the Martians' original           invasion plans were similarly thwarted by health and safety           issues, "slain after all man's devices had failed by the           humblest creatures that God, in his wisdom, has put upon this           earth: bacteria. Minute, invisible, bacteria. For, directly           the invaders arrived and drank and fed, our microscopic allies           attacked them..."           url_dzdepyvagh                             - "...From that moment, they were doomed."                                   >> ANTI-NEWS <                              berating the obvious           moving on from PUERILE GOOGLE MISSPELLINGS, weird search-and-          replace artefact:  url_dzdepyvagh  ,          Japanese fan sites for "plince", "steery dan", "def reppard"           et al, plus the  number_wquctgtkut  or more self-referential Usenet .sigs:            url_dzdepyvagh            ... url_dzdepyvagh  "Zerodowntime" ad leads           to:  url_dzdepyvagh  ... slightly           harsh alt text:  url_dzdepyvagh  ...           US military discovers the only "translator" those bastards           seem to understand:  url_dzdepyvagh            ... scary blue men herald return of the bizarre BBC hacking           pics:  url_dzdepyvagh  ... reporter RYAN           DILLEY  url_dzdepyvagh  pulls his            url_dzdepyvagh  face           ... banjo maestro GEORGE FORMBY still alive, cooking, black:        url_dzdepyvagh           ... thanks guys, that ought to do it:  url_dzdepyvagh  ...                                            >> EVENT QUEUE <                          goto's considered non-harmful           Controversially, we're all in favour of THE GUARDIAN GREAT           BRITISH BLOG COMPETITION (closing date next Fri  number_wquctgtkut - number_wquctgtkut - number_wquctgtkut            first prize UKP number_wquctgtkut  entry free), in that any initiative that           encourages this notoriously primadonna-ish "community" to try           and engage with real-world notions of editorial quality surely           has to be a good thing. Our only disappointment is that The           Guardian appears to be focussing on the "best" of the entries,           when everyone knows the real fun is to be had cruising the           truly terrible examples that the genre has to offer, mentally           allocating points for "Most Depressing Recycling Of Daypop Top           number_wquctgtkut  URLs", "Most Unsettling Revelations About Personal Life",           plus of course "Most Tedious Linking/Reciprocal Linking To Other           Bloggers In Absence Of Having Anything Interesting To Say".            url_dzdepyvagh                                - "A strange game, Professor Falken..."         url_dzdepyvagh                           - "...the only winning move is not to play."                                   >> TRACKING <                sufficiently advanced technology : the gathering            The respective trademark holders will hate this, but           Windows really *is* like the Sun. You have this big hulking           mass of concentrated power in the middle, with a few small           orbiting utilities - like WinZip, and PuTTY, and VNC.           Occasionally one will get a bit too close to the OS, and           Microsoft will suck it down and turn it into fuel for the           System. One such discrete satellite remains FILEZILLA, the           still-necessary ftp gui client for Windows. Those who know           it won't need the introduction, although they might           appreciate the note that it's getting close to v number_wquctgtkut  time.           For dogged WS_FTP users, though, it's got multiple           downloads, auto-restart of interrupted 'loads, queuing, and           sftp and Kerberos support. It's also GPL'd which makes it a           nice bit of source for anyone wanting to grok Win number_wquctgtkut            networking from something that works.            url_dzdepyvagh    - talking of trademarks, will the Godzilla people strike before MS?                                   >> MEMEPOOL <                 ceci n'est pas une  url_dzdepyvagh            (Not safe for work) next year's RED NOSE DAY looks more fun           than usual:  url_dzdepyvagh  ... Mirrored           Disaster Recovery Suite - to go with mirrored bathroom etc?:          url_dzdepyvagh           ... and then the kid can take you to court for mental cruelty:            url_dzdepyvagh  ...           "funny" prefixes in front of "chalking" #n+ number_wquctgtkut  - the actually           quite pragmatic:  url_dzdepyvagh  ...           no longer knowing - or caring - if these are prank AMAZON           reviews or not, for Potter's ever-popular "vibrating" broom:        url_dzdepyvagh           ... ditto "Use This Software At Your Own Risk" disclaimer for:            url_dzdepyvagh            ... DAFFY DUCK appears in dock - accused of "dethpicable"           behaviour?:  url_dzdepyvagh  ,            url_dzdepyvagh  ...                                   >> GEEK MEDIA <                                   get out less           TV>> celebrity cameo night tonight, with Brad Pitt in FRIENDS           ( number_wquctgtkut pm, Fri, C number_wquctgtkut ), Sydney Pollack in WILL AND GRACE ( number_wquctgtkut pm, Fri,           C number_wquctgtkut ), Dustin Hoffman in V GRAHAM NORTON ( number_wquctgtkut pm, Fri, C number_wquctgtkut ), and           a singing, dancing peanut in globalised trade documentary ALT-          TV ( number_wquctgtkut pm, Fri, C number_wquctgtkut )... the BBC have kept McEnroe and the           heart monitor, got rid of the live crocodiles in gimmicky           quizshow THE CHAIR ( number_wquctgtkut pm, Sat, BBC number_wquctgtkut )... and a month of           "September  number_wquctgtkut th" specials kicks off with AVENGING TERROR ( number_wquctgtkut pm,           Sat & Sun, C number_wquctgtkut ) - yet those responsible for BOWFINGER ( number_wquctgtkut pm,           Sat, C number_wquctgtkut ) and NOTTING HILL ( number_wquctgtkut pm, Sun, C number_wquctgtkut ) still remain           unpunished... John "The Last Seduction" Dahl's ROUNDERS ( number_wquctgtkut pm,           Sat, BBC number_wquctgtkut ) turns out to be about high-stakes poker, rather           than the girls' version of baseball... in the wake of DAVE           GORMAN'S IMPORTANT ASTROLOGY EXPERIMENT ( number_wquctgtkut pm, Sun, BBC number_wquctgtkut ),           how about a three-way challenge where he, Tony Hawks and Pete           McCarthy battle to come up with the most lucrative pointless           pretext for a book and TV show?... but we still have a soft           spot for Ron "Alien: Resurrection" Perlman liberal self-          flagellation THE LAST SUPPER ( number_wquctgtkut pm, Sun, C number_wquctgtkut )...  number_wquctgtkut / number_wquctgtkut  CLEAR           THE SKIES ( number_wquctgtkut pm, Sun, BBC number_wquctgtkut ) is a presumably uneventful account           of "how US air defence systems responded to the events of           September  number_wquctgtkut th"... inexplicably, the three finalists in THE           TARTIEST MEN IN BRITAIN ( number_wquctgtkut pm, Mon, ITV) all come from           Leeds... Larry Clark takes a somewhat indirect approach to           conveying his safe-sex message in New York filth-fest KIDS           ( number_wquctgtkut am, Tue, C number_wquctgtkut )... and the September  number_wquctgtkut th build-up continues           with HOW THE TWIN TOWERS COLLAPSED ( number_wquctgtkut pm, Mon, C number_wquctgtkut ), LET'S ROLL:           THE STORY OF FLIGHT  number_wquctgtkut  ( number_wquctgtkut pm, Wed, ITV), plus THE MEYSSAN           CONSPIRACY ( number_wquctgtkut pm, Tue, C number_wquctgtkut ) - ie the French guy behind:          url_dzdepyvagh           ... away from the polluted nightmare of modern living, a           family seek out a new way of life in Earth Summit tie-in A           LAND WORTH LOVING ( number_wquctgtkut pm, Wed, BBC number_wquctgtkut )... which coincidentally           also forms the plot of this week's second Heather "Bowfinger"           Graham turkey, LOST IN SPACE ( number_wquctgtkut pm, Wed, BBC number_wquctgtkut ) - not to be           confused with the return of those annoying posh women in           WORLD'S WORST DRESSED ( number_wquctgtkut pm, Wed, BBC number_wquctgtkut ), who have at least shut           up about their always-doomed hideously purple e-commerce           site:  url_dzdepyvagh  ...                    FILM>> the comic skills of Cameron Diaz, Christina Applegate           and Parker Posey combine in a cross between a teen smut comedy           and an episode of "Sex And The City", THE SWEETEST THING           (  url_dzdepyvagh  :          As [Diaz] and [Applegate] drive down the road still dressed in           just their bras and underwear, [Applegate] drops her bottle of           fingernail polish. [Diaz] then goes over to get it, with           her panty-covered butt in the air and her head down toward           [Applegate]'s legs and crotch; [Selma Blair] [has] her mouth           stuck around a man's privates after apparently performing oral           sex on him)... Robin Williams plays a surprisingly convincing           Hannibal Lecter in morally complicated Alaskan Al Pacino           murder madness INSOMNIA (  url_dzdepyvagh  : You can see           [Crystal Lowe's] tits in autopsy photos and again - along with           bush - when she's seen on a autopsy table. Nice boobs but           she's dead)... it's Eddie Murphy, Randy Quaid, Jay "Jerry           Maguire" Mohr, John Cleese and Pam Grier - together at last! -           in blaxploitation sci-fi spoof THE ADVENTURES OF PLUTO NASH    (  url_dzdepyvagh  :          the woman then causes the image of [Rosario "Kids" Dawson] to           suddenly have much larger breasts and an exaggeratedly large           rear end)... all of which, shockingly, are an improvement on           John Woo interspersing lame battle scenes with agonising anti-          racist philosophising in WW number_wquctgtkut  Navajo crypto clunker WINDTALKERS           (  url_dzdepyvagh  :           gambling; beheading; brief partial nudity of a Japanese           soldier; I have no doubt that such gore is present in war but           must it be regurgitated in and as entertainment?)...                                   >> SMALL PRINT <         Need to Know is a useful and interesting UK digest of things that          happened last week or might happen next week. You can read it        on Friday afternoon or print it out then take it home if you have      nothing better to do. It is compiled by NTK from stuff they get sent.                        Registered at the Post Office as                      "yeah, but bet we were banned first"                      url_dzdepyvagh                                     NEED TO KNOW             THEY STOLE OUR REVOLUTION. NOW WE'RE STEALING IT BACK.                          Archive -  url_dzdepyvagh                Unsubscribe? Mail  email_imkxepvqcp                  Subscribe? Mail  email_imkxepvqcp   NTK now is supported by  url_dzdepyvagh , and by you:  url_dzdepyvagh                             (K)  number_wquctgtkut  Special Projects.              Copying is fine, but include URL:  url_dzdepyvagh                       Tips, news and gossip to  email_imkxepvqcp               All communication is for publication, unless you beg.               Press releases from naive PR people to  email_imkxepvqcp       Remember: Your work email may be monitored if sending sensitive material.        Sending > number_wquctgtkut KB attachments is forbidden by the Geneva Convention.               Your country may be at risk if you fail to comply.

Tokens returned:

[
  '_____',             '__',                '__',
  '__',                'number_wquctgtkut', 'number_wquctgtkut',
  'number_wquctgtkut', 'join',              'mail',
  'empti',             'messag',            'email_imkxepvqcp',
  'websit',            'archiv',            'live',
  '___',               'url_dzdepyvagh',    'case',
  'point',             'web',               'design',
  'matt',              'jone',              'man',
  'respons',           'bbc',               'new',
  'onlin',             'look',              'launch',
  'invent',            'warchalk',          'recent',
  'describ',           'curs',              'url_dzdepyvagh',
  'turn',              'against',           'creat',
  'hard',              'new',               'stiffen',
  'sinew',             'hot',               'summer',
  'dai',               'mailinglist',       'alleywai',
  'danger',            'empti',             'sane',
  'post',              'strewn',            'rot',
  'carcass',           'broil',             'vacat',
  'messag',            'hacktress',         'silicon',
  'vallei',            'chief',             'rat',
  'keeper',            'lile',              'elam',
  'excitedli',         'post',              'new',
  'open',              'number_wquctgtkut', 'network',
  'she',               'found',             'polic',
  'station',           'wait',              'judg',
  'thought',           'check',             'connect',
  'she',               'write',             'somewhat',
  'recklessli',        'bai',               'area',
  'wireless',          'list',              'exit',
  'rest',              'wifi',              'commun',
  'nearest',           'window',            'street',
  'cool',              'temper',            'microsoft',
  'palladium',
  ... 1143 more items
]

[niftylettuce]

I believe the issue is if there's more than one underscore in sequence

[niftylettuce]

The published version of v2.1.2 is not accurate is the issue

[niftylettuce]

@Hugo-ter-Doest Please fix the version published by publishing a new version and ensuring this is accurate in the tarball:

λ ~/Projects/spamscanner/ master* cat node_modules/natural/package.json | grep "version"
  "version": "2.1.2",
λ ~/Projects/spamscanner/ master* cat node_modules/natural/lib/natural/tokenizers/aggressive_tokenizer.js
/*
Copyright (c) 2011, Chris Umbel

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/

var Tokenizer = require('./tokenizer'),
    util = require('util');

var AggressiveTokenizer = function() {
    Tokenizer.call(this);
};
util.inherits(AggressiveTokenizer, Tokenizer);

module.exports = AggressiveTokenizer;

AggressiveTokenizer.prototype.tokenize = function(text) {
    // break a string up into an array of tokens by anything non-word
    return this.trim(text.split(/\W+/));
};

As you can see, I'm on latest version v2.1.2 and the tokenize function is not matching up to your latest commit.

[niftylettuce]

To prevent this in future, I suggest using np@5.0, npm i -g np@5.0 and then use np command to publish and release.

[Hugo-ter-Doest]

I implemented a combination of sentence tokenizer and aggressive tokenizer: https://gist.github.com/Hugo-ter-Doest/0f2c158a1acd7e9a3fd71ee043e7e2c3

Output is here: https://gist.github.com/Hugo-ter-Doest/7d9dd2ae365aa1a9139f18ada7c6073b

I don't see any underscores anymore.

BUT: I think you should keep the underscores in some case like names like number_wquctgtkut

[niftylettuce]

I changed it so number_xyz is just numberxyz to keep things simple.

Closed per v2.1.3