search

ageitgey / node-unfluff

Automatically extract body content (and other cool stuff) from an html document
Apache License 2.0
2.15k stars 223 forks source link

Removes embedded Instagram posts from articles #72

Open peterwilli opened 7 years ago

peterwilli commented 7 years ago

Removes embedded Instagram posts from articles (which are often in a different language) (test article: http://www.huffingtonpost.com/entry/asian-models-victorias-secret-fashion-show_us_58420596e4b09e21702ec3e9)

Before fork, the text-variable held:

When xiaowen was on the stage and we were dancing with her at backstage 💃🏻💃🏻💃🏻 @liuwenlw @mingxi11 #vsfs2016 #victoriasecrets #vsbackstage\n\nA photo posted by 何穗 sui he (@hesui923) on

But after this change, it's:

Compare that with the two Asian models, out of a total of 44 women, who were in the show last year.\n\nThe four Chinese models shared their excitement earlier this week by releasing a group photo of them posing, with peace signs raised, in front of the pink Victoria’s Secret banner in Paris.\n\nVictoria’s Secret had been criticized for not featuring a more racially diverse lineup of models over the course of the fashion show’s 21-year history. In fact, no Asian model had walked the VS runway until 2009, according to Yahoo Style.\n\nIn recent years, however, the lingerie giant has ramped up its efforts to be inclusive.\n\nLast year, Angolan model Maria Borges was the first black Victoria’s Secret model to wear natural hair down the runway. And this year, you can expect to see several models of color rocking a variety of natural tresses ― Afros, loose curls and all.\n\nThe lingerie brand’s decision to double down on Chinese models for the 2016 show may have been a strategic move as well, Yahoo Style reports.\n\nVictoria’s Secret slipped into the Chinese market in 2015 with beauty and accessory shops and, after seeing success in the market, announced plans to open its signature bra and panty stores in the country.\n\nAnd, as Reuters reported in July, the market for high-end lingerie in China has skyrocketed in five years, amounting to an $18 billion industry.\n\nVictoria’s Secret echoed the Chinese models’ excitement by posting a photo of three of them on their own account, greeting them with a traditional “Ni hao” on Instagram.\n\nCheck out a preview of Victoria’s Secret’s event in the photo gallery below, and tune in to CBS at 5 p.m. ET on Dec. 5 for the full runway show.

I had to create a new filter function (see cleanSelector), as I couldn't find a existing function to do this. Feedback is welcome :+1:

ageitgey commented 7 years ago

This looks pretty reasonable to me. Can you add a test case for it with one of the affected articles?