Closed niftylettuce closed 4 years ago
Yes, here's an example:
String passed:
_ _ _____ _ __ | \ | |_ _| |/ / _ __ __ number_wquctgtkut - number_wquctgtkut - number_wquctgtkut _ o join! mail an empty message to | \| | | | | ' / | '_ \ / _ \ \ /\ / / o email_imkxepvqcp | |\ | | | | . \ | | | | (_) \ v v / o website (+ archive) lives at: |_| \_| |_| |_|\_\|_| |_|\___/ \_/\_/ o url_dzdepyvagh "A case in point is web designer Matt Jones, the man responsible for how BBC News Online looked when it launched. Since then, he has invented 'warchalking', which he recently described as a 'curse'..." url_dzdepyvagh - but you cannot turn against me! I... created you! >> HARD NEWS < stiffening sinews More hot summer days in the mailinglist alleyways, dangerously empty of sane postings, strewn with the rotting carcasses of broiling vacation messages. Hacktress and Silicon Valley's chief rat-keeper LILE ELAM, excitedly posts about a new open number_wquctgtkut network she's found. "I am here at the police station waiting to see a judge and I thought I would check to see if there is connectivity", she writes, somewhat recklessly, to the Bay Area Wireless list. Exit the rest of the Wifi community through the nearest window and out into the streets... where, cooling tempers, the Microsoft Palladium boys are on an endless summer tour, reassuring the experts that while, hmm, they *suppose* Pd could *theoretically* be used as a Hollywood DRM system, they truly have no plans to do any such thing. Cypherpunk and friend of freedom Lucky Green hears this; thinks up four or five of the obvious Palladium DRM implementations; sends them off to be patented in his name. Licensing funds, we imagine, will go on cracking his own DRMs. And so the mail loops on. url_dzdepyvagh - administrivia: HI MOM, I'M IN JAIL http: email_imkxepvqcp - Green/Palladium, like Green Kryptonite Could Lucky get himself arrested under the DMCA for distributing a circumvention device? Worse: now we have the EUCD incoming, could he here in the UK? Will Alan Cox go to jail for posting detailed Changelogs? Will even the nicest UK cryptographer (or curious garage tinkerer) find themselves hauled up under our new and scarily DMCAish copyright regime? Find out the facts at the free FAIR DEAL FOR COPYRIGHT conference, organised by the irrepressible FOUNDATION FOR INFORMATION POLICY RESEARCH for Wednesday number_wquctgtkut - number_wquctgtkut - number_wquctgtkut at the London School of Economics. All the usual fun from the creators of the Scrambling for Safety crypto cons: we confidently predict government spokesmen caught in headlights, wanton Dave Bird heckling, some industry bigwig fighting off the audience with a broken chairleg, and other epiphenomena of the interzone between legal minds and hacker ethics. Oh, and FIPR are still looking for a Programme Director, so if you're interested, let them know. We suggested a convention raffle (first prize: the director's job, second prize: Ross Anderson as your personal slave for a day). They say there's some rule that would break, though. url_dzdepyvagh - doesn't the Foundation use psychohistory for filling these positions? For those of us who can't read the abbreviation EULA without thinking of Martian fighting machines and their "deafening howls... which roared like thunder", we're sorry to report that this weekend's multimedia performance of Jeff Wayne's WAR OF THE WORLDS has been postponed due to "health and safety issues". The event was to feature computer graphics, fireworks, " number_wquctgtkut ft-tall Martian fighting machines" wreaking "havoc and destruction", and - most terrifyingly of all - the possibility of a David Essex tribute singer performing with Hawkwind, but UKP number_wquctgtkut tickets for the Sat number_wquctgtkut - number_wquctgtkut - number_wquctgtkut show at Manchester's Heaton Park will still be valid at a range of new venues next summer. Ironically, the Martians' original invasion plans were similarly thwarted by health and safety issues, "slain after all man's devices had failed by the humblest creatures that God, in his wisdom, has put upon this earth: bacteria. Minute, invisible, bacteria. For, directly the invaders arrived and drank and fed, our microscopic allies attacked them..." url_dzdepyvagh - "...From that moment, they were doomed." >> ANTI-NEWS < berating the obvious moving on from PUERILE GOOGLE MISSPELLINGS, weird search-and- replace artefact: url_dzdepyvagh , Japanese fan sites for "plince", "steery dan", "def reppard" et al, plus the number_wquctgtkut or more self-referential Usenet .sigs: url_dzdepyvagh ... url_dzdepyvagh "Zerodowntime" ad leads to: url_dzdepyvagh ... slightly harsh alt text: url_dzdepyvagh ... US military discovers the only "translator" those bastards seem to understand: url_dzdepyvagh ... scary blue men herald return of the bizarre BBC hacking pics: url_dzdepyvagh ... reporter RYAN DILLEY url_dzdepyvagh pulls his url_dzdepyvagh face ... banjo maestro GEORGE FORMBY still alive, cooking, black: url_dzdepyvagh ... thanks guys, that ought to do it: url_dzdepyvagh ... >> EVENT QUEUE < goto's considered non-harmful Controversially, we're all in favour of THE GUARDIAN GREAT BRITISH BLOG COMPETITION (closing date next Fri number_wquctgtkut - number_wquctgtkut - number_wquctgtkut first prize UKP number_wquctgtkut entry free), in that any initiative that encourages this notoriously primadonna-ish "community" to try and engage with real-world notions of editorial quality surely has to be a good thing. Our only disappointment is that The Guardian appears to be focussing on the "best" of the entries, when everyone knows the real fun is to be had cruising the truly terrible examples that the genre has to offer, mentally allocating points for "Most Depressing Recycling Of Daypop Top number_wquctgtkut URLs", "Most Unsettling Revelations About Personal Life", plus of course "Most Tedious Linking/Reciprocal Linking To Other Bloggers In Absence Of Having Anything Interesting To Say". url_dzdepyvagh - "A strange game, Professor Falken..." url_dzdepyvagh - "...the only winning move is not to play." >> TRACKING < sufficiently advanced technology : the gathering The respective trademark holders will hate this, but Windows really *is* like the Sun. You have this big hulking mass of concentrated power in the middle, with a few small orbiting utilities - like WinZip, and PuTTY, and VNC. Occasionally one will get a bit too close to the OS, and Microsoft will suck it down and turn it into fuel for the System. One such discrete satellite remains FILEZILLA, the still-necessary ftp gui client for Windows. Those who know it won't need the introduction, although they might appreciate the note that it's getting close to v number_wquctgtkut time. For dogged WS_FTP users, though, it's got multiple downloads, auto-restart of interrupted 'loads, queuing, and sftp and Kerberos support. It's also GPL'd which makes it a nice bit of source for anyone wanting to grok Win number_wquctgtkut networking from something that works. url_dzdepyvagh - talking of trademarks, will the Godzilla people strike before MS? >> MEMEPOOL < ceci n'est pas une url_dzdepyvagh (Not safe for work) next year's RED NOSE DAY looks more fun than usual: url_dzdepyvagh ... Mirrored Disaster Recovery Suite - to go with mirrored bathroom etc?: url_dzdepyvagh ... and then the kid can take you to court for mental cruelty: url_dzdepyvagh ... "funny" prefixes in front of "chalking" #n+ number_wquctgtkut - the actually quite pragmatic: url_dzdepyvagh ... no longer knowing - or caring - if these are prank AMAZON reviews or not, for Potter's ever-popular "vibrating" broom: url_dzdepyvagh ... ditto "Use This Software At Your Own Risk" disclaimer for: url_dzdepyvagh ... DAFFY DUCK appears in dock - accused of "dethpicable" behaviour?: url_dzdepyvagh , url_dzdepyvagh ... >> GEEK MEDIA < get out less TV>> celebrity cameo night tonight, with Brad Pitt in FRIENDS ( number_wquctgtkut pm, Fri, C number_wquctgtkut ), Sydney Pollack in WILL AND GRACE ( number_wquctgtkut pm, Fri, C number_wquctgtkut ), Dustin Hoffman in V GRAHAM NORTON ( number_wquctgtkut pm, Fri, C number_wquctgtkut ), and a singing, dancing peanut in globalised trade documentary ALT- TV ( number_wquctgtkut pm, Fri, C number_wquctgtkut )... the BBC have kept McEnroe and the heart monitor, got rid of the live crocodiles in gimmicky quizshow THE CHAIR ( number_wquctgtkut pm, Sat, BBC number_wquctgtkut )... and a month of "September number_wquctgtkut th" specials kicks off with AVENGING TERROR ( number_wquctgtkut pm, Sat & Sun, C number_wquctgtkut ) - yet those responsible for BOWFINGER ( number_wquctgtkut pm, Sat, C number_wquctgtkut ) and NOTTING HILL ( number_wquctgtkut pm, Sun, C number_wquctgtkut ) still remain unpunished... John "The Last Seduction" Dahl's ROUNDERS ( number_wquctgtkut pm, Sat, BBC number_wquctgtkut ) turns out to be about high-stakes poker, rather than the girls' version of baseball... in the wake of DAVE GORMAN'S IMPORTANT ASTROLOGY EXPERIMENT ( number_wquctgtkut pm, Sun, BBC number_wquctgtkut ), how about a three-way challenge where he, Tony Hawks and Pete McCarthy battle to come up with the most lucrative pointless pretext for a book and TV show?... but we still have a soft spot for Ron "Alien: Resurrection" Perlman liberal self- flagellation THE LAST SUPPER ( number_wquctgtkut pm, Sun, C number_wquctgtkut )... number_wquctgtkut / number_wquctgtkut CLEAR THE SKIES ( number_wquctgtkut pm, Sun, BBC number_wquctgtkut ) is a presumably uneventful account of "how US air defence systems responded to the events of September number_wquctgtkut th"... inexplicably, the three finalists in THE TARTIEST MEN IN BRITAIN ( number_wquctgtkut pm, Mon, ITV) all come from Leeds... Larry Clark takes a somewhat indirect approach to conveying his safe-sex message in New York filth-fest KIDS ( number_wquctgtkut am, Tue, C number_wquctgtkut )... and the September number_wquctgtkut th build-up continues with HOW THE TWIN TOWERS COLLAPSED ( number_wquctgtkut pm, Mon, C number_wquctgtkut ), LET'S ROLL: THE STORY OF FLIGHT number_wquctgtkut ( number_wquctgtkut pm, Wed, ITV), plus THE MEYSSAN CONSPIRACY ( number_wquctgtkut pm, Tue, C number_wquctgtkut ) - ie the French guy behind: url_dzdepyvagh ... away from the polluted nightmare of modern living, a family seek out a new way of life in Earth Summit tie-in A LAND WORTH LOVING ( number_wquctgtkut pm, Wed, BBC number_wquctgtkut )... which coincidentally also forms the plot of this week's second Heather "Bowfinger" Graham turkey, LOST IN SPACE ( number_wquctgtkut pm, Wed, BBC number_wquctgtkut ) - not to be confused with the return of those annoying posh women in WORLD'S WORST DRESSED ( number_wquctgtkut pm, Wed, BBC number_wquctgtkut ), who have at least shut up about their always-doomed hideously purple e-commerce site: url_dzdepyvagh ... FILM>> the comic skills of Cameron Diaz, Christina Applegate and Parker Posey combine in a cross between a teen smut comedy and an episode of "Sex And The City", THE SWEETEST THING ( url_dzdepyvagh : As [Diaz] and [Applegate] drive down the road still dressed in just their bras and underwear, [Applegate] drops her bottle of fingernail polish. [Diaz] then goes over to get it, with her panty-covered butt in the air and her head down toward [Applegate]'s legs and crotch; [Selma Blair] [has] her mouth stuck around a man's privates after apparently performing oral sex on him)... Robin Williams plays a surprisingly convincing Hannibal Lecter in morally complicated Alaskan Al Pacino murder madness INSOMNIA ( url_dzdepyvagh : You can see [Crystal Lowe's] tits in autopsy photos and again - along with bush - when she's seen on a autopsy table. Nice boobs but she's dead)... it's Eddie Murphy, Randy Quaid, Jay "Jerry Maguire" Mohr, John Cleese and Pam Grier - together at last! - in blaxploitation sci-fi spoof THE ADVENTURES OF PLUTO NASH ( url_dzdepyvagh : the woman then causes the image of [Rosario "Kids" Dawson] to suddenly have much larger breasts and an exaggeratedly large rear end)... all of which, shockingly, are an improvement on John Woo interspersing lame battle scenes with agonising anti- racist philosophising in WW number_wquctgtkut Navajo crypto clunker WINDTALKERS ( url_dzdepyvagh : gambling; beheading; brief partial nudity of a Japanese soldier; I have no doubt that such gore is present in war but must it be regurgitated in and as entertainment?)... >> SMALL PRINT < Need to Know is a useful and interesting UK digest of things that happened last week or might happen next week. You can read it on Friday afternoon or print it out then take it home if you have nothing better to do. It is compiled by NTK from stuff they get sent. Registered at the Post Office as "yeah, but bet we were banned first" url_dzdepyvagh NEED TO KNOW THEY STOLE OUR REVOLUTION. NOW WE'RE STEALING IT BACK. Archive - url_dzdepyvagh Unsubscribe? Mail email_imkxepvqcp Subscribe? Mail email_imkxepvqcp NTK now is supported by url_dzdepyvagh , and by you: url_dzdepyvagh (K) number_wquctgtkut Special Projects. Copying is fine, but include URL: url_dzdepyvagh Tips, news and gossip to email_imkxepvqcp All communication is for publication, unless you beg. Press releases from naive PR people to email_imkxepvqcp Remember: Your work email may be monitored if sending sensitive material. Sending > number_wquctgtkut KB attachments is forbidden by the Geneva Convention. Your country may be at risk if you fail to comply.
Tokens returned:
[
'_____', '__', '__',
'__', 'number_wquctgtkut', 'number_wquctgtkut',
'number_wquctgtkut', 'join', 'mail',
'empti', 'messag', 'email_imkxepvqcp',
'websit', 'archiv', 'live',
'___', 'url_dzdepyvagh', 'case',
'point', 'web', 'design',
'matt', 'jone', 'man',
'respons', 'bbc', 'new',
'onlin', 'look', 'launch',
'invent', 'warchalk', 'recent',
'describ', 'curs', 'url_dzdepyvagh',
'turn', 'against', 'creat',
'hard', 'new', 'stiffen',
'sinew', 'hot', 'summer',
'dai', 'mailinglist', 'alleywai',
'danger', 'empti', 'sane',
'post', 'strewn', 'rot',
'carcass', 'broil', 'vacat',
'messag', 'hacktress', 'silicon',
'vallei', 'chief', 'rat',
'keeper', 'lile', 'elam',
'excitedli', 'post', 'new',
'open', 'number_wquctgtkut', 'network',
'she', 'found', 'polic',
'station', 'wait', 'judg',
'thought', 'check', 'connect',
'she', 'write', 'somewhat',
'recklessli', 'bai', 'area',
'wireless', 'list', 'exit',
'rest', 'wifi', 'commun',
'nearest', 'window', 'street',
'cool', 'temper', 'microsoft',
'palladium',
... 1143 more items
]
I believe the issue is if there's more than one underscore in sequence
The published version of v2.1.2 is not accurate is the issue
@Hugo-ter-Doest Please fix the version published by publishing a new version and ensuring this is accurate in the tarball:
λ ~/Projects/spamscanner/ master* cat node_modules/natural/package.json | grep "version"
"version": "2.1.2",
λ ~/Projects/spamscanner/ master* cat node_modules/natural/lib/natural/tokenizers/aggressive_tokenizer.js
/*
Copyright (c) 2011, Chris Umbel
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
*/
var Tokenizer = require('./tokenizer'),
util = require('util');
var AggressiveTokenizer = function() {
Tokenizer.call(this);
};
util.inherits(AggressiveTokenizer, Tokenizer);
module.exports = AggressiveTokenizer;
AggressiveTokenizer.prototype.tokenize = function(text) {
// break a string up into an array of tokens by anything non-word
return this.trim(text.split(/\W+/));
};
As you can see, I'm on latest version v2.1.2 and the tokenize function is not matching up to your latest commit.
To prevent this in future, I suggest using np@5.0
, npm i -g np@5.0
and then use np
command to publish and release.
I implemented a combination of sentence tokenizer and aggressive tokenizer: https://gist.github.com/Hugo-ter-Doest/0f2c158a1acd7e9a3fd71ee043e7e2c3
Output is here: https://gist.github.com/Hugo-ter-Doest/7d9dd2ae365aa1a9139f18ada7c6073b
I don't see any underscores anymore.
BUT: I think you should keep the underscores in some case like names like number_wquctgtkut
I changed it so number_xyz
is just numberxyz
to keep things simple.
Closed per v2.1.3
Are you using the Porter stemmer for English? Can you give some examples?