Scrape content hidden behind js print function

[mweinelt]

Wowhead.com hides their content behind a javascript print function, is there any way to write a scraper rule for that? Does goquery evaluate javascript? After execution the content lands in the empty div.news-post-5b81a133ac83a.

https://www.wowhead.com/news=286563/battle-for-azeroth-darkmoon-deck-community-opinions

    <div class="news-post-content">
        <div id="news-post-5b81a133ac83a"></div>
        <script>//<![CDATA[
            WH.markup.printHtml("Welcome to the first Community Opinions of Battle for Azeroth! In this post, we'll be taking a look at what each class and spec thinks of the Darkmoon Deck. See if you should buy one for your spec!\r\n\r\nWe reached out to a representative from each spec's community to give their current thoughts on the Darkmoon Decks [b]before yesterday's[url=https:\/\/www.wowhead.com\/news=286686\/battle-for-azeroth-hotfixes-for-august-24th-azerite-powers-darkmoon-deck-squalls] Squalls hotfix[\/url][\/b].\r\n\r\nWant to know what other trinkets are good? Wowhead's [url=https:\/\/www.wowhead.com\/class-guides]Class Guides[\/url] include pre-raid gear recommendations, updated for Battle for Azeroth.\r\n\r\n[center][table class=\"grid\" width=750px]\r\n[tr]\r\n[td width=250px][url=https:\/\/www.wowhead.com\/blood-death-knight-gear-guide][color=c6]Blood Death Knight[\/color][\/url][\/td] \r\n[td][url=https:\/\/www.wowhead.com\/arcane-mage-gear-guide][color=c8]Arcane Mage[\/color][\/url][\/td] \r\n[td][url=https:\/\/www.wowhead.com\/assassination-rogue-gear-guide][color=c4]Assassination Rogue[\/color][\/url][\/td] \r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/frost-death-knight-gear-guide][color=c6]Frost Death Knight[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/fire-mage-gear-guide][color=c8]Fire Mage [\/color][\/url][\/td] \r\n[td][url=https:\/\/www.wowhead.com\/outlaw-rogue-gear-guide][color=c4]Outlaw Rogue[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/unholy-death-knight-gear-guide][color=c6]Unholy Death Knight[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/frost-mage-gear-guide][color=c8]Frost Mage[\/color][\/url][\/td] \r\n[td][url=https:\/\/www.wowhead.com\/subtlety-rogue-gear-guide][color=c4]Subtlety Rogue[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/havoc-demon-hunter-gear-guide][color=c12]Havoc Demon Hunter[\/color][\/url][\/td]\r\n[td width=250px][url=https:\/\/www.wowhead.com\/brewmaster-monk-gear-guide][color=c10]Brewmaster Monk[\/color][\/url][\/td]\r\n[td width=250px] [url=https:\/\/www.wowhead.com\/elemental-shaman-gear-guide][color=c7]Elemental Shaman[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/vengeance-demon-hunter-gear-guide][color=c12]Vengeance Demon Hunter[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/mistweaver-monk-gear-guide][color=c10]Mistweaver Monk[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/enhancement-shaman-gear-guide][color=c7]Enhancement Shaman[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/balance-druid-gear-guide][color=c11]Balance Druid[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/windwalker-monk-gear-guide][color=c10]Windwalker Monk [\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/restoration-shaman-gear-guide][color=c7]Restoration Shaman[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/feral-druid-gear-guide][color=c11]Feral Druid[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/holy-paladin-gear-guide][color=c2]Holy Paladin[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/affliction-warlock-gear-guide][color=c9]Affliction Warlock[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/guardian-druid-gear-guide][color=c11]Guardian Druid [\/color][\/url][\/td]\r\n[td width=250px][url=https:\/\/www.wowhead.com\/protection-paladin-gear-guide][color=c2]Protection Paladin[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/demonology-warlock-gear-guide][color=c9]Demonology Warlock[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/restoration-druid-gear-guide][color=c11]Restoration Druid[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/retribution-paladin-gear-guide][color=c2]Retribution Paladin[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/destruction-warlock-gear-guide][color=c9]Destruction Warlock[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/beast-mastery-hunter-gear-guide][color=c3]Beast Mastery Hunter[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/discipline-priest-gear-guide][color=c5]Discipline Priest[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/arms-warrior-gear-guide][color=c1]Arms Warrior[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/marksmanship-hunter-gear-guide][color=c3]Marksmanship Hunter [\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/holy-priest-gear-guide][color=c5]Holy Priest[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/fury-warrior-gear-guide][color=c1]Fury Warrior[\/color][\/url][\/td]\r\n[\/tr]\r\n[tr]\r\n[td][url=https:\/\/www.wowhead.com\/survival-hunter-gear-guide][color=c3]Survival Hunter [\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/shadow-priest-gear-guide][color=c5]Shadow Priest[\/color][\/url][\/td]\r\n[td][url=https:\/\/www.wowhead.com\/protection-warrior-gear-guide][color=c1]Protection Warrior[\/color][\/url][\/td]\r\n[\/tr]\r\n[\/table][\/center]\r\n\r\n[toc]\r\n\r\n[h2][item=159125] - Melee \/ Hunter[\/h2]\r\n\r\n[toggler name=\"Hunters by Moofz\" size=3 closed=false][div class=patch-diff]\r\n[item=159125] has been hotfixed recently and now works properly for both Marksmanship and Beast Mastery Hunters. It is now the best in slot trinket (pre-raid) in single target situations and fairly decent in AoE situations for all 3 Hunter specs.\r\n[\/div][\/toggler][toggler name=\"Feral Druid by Guiltyas\" size=3 closed=false][div class=patch-diff]\r\nAs most people who have been following gearing this expansion should be aware, BOE\u2019s appear to be A) dropping at a high rate and B) are extremely well statted for ferals (with a lot of haste present). Of the BOE\u2019s available however there is a clear stand out. [item=159125]. If you\u2019re going to choose to spend gold on anything in preparation for raid then there is nothing I would recommend more. You may have concerns due to the trinket having received a 9% nerf recently? Don\u2019t worry, prior to the nerf Fathoms was a superior choice to 390 raid trinkets and likely would not have been replaced until next tier (longer if ilvl upgrades became available as in Legion), after the nerf you might look to replace the trinket at 390 raid trinkets but with personal loot this is no guarantee. It\u2019s likely that the trinket will continue to be used into Uldir farm and with prices seeming to have settled out I would strongly recommend an Investment. As always however use tools like [url=https:\/\/www.raidbots.com\/simbot]Raidbots[\/url] to evaluate your gearing choices yourself.\r\n[\/div][\/toggler][toggler name=\"Death Knights DPS by Bicepspump\" size=3 closed=false][div class=patch-diff]\r\nBicepspump, Frost Unholy - [item=159125] is the best obtainable trinket for the raid release. The recent nerfs weren't enough to bring it down from the top spot.\r\n[\/div][\/toggler][toggler name=\"Havoc Demon Hunter by Kib\" size=3 closed=false][div class=patch-diff]\r\nIt's very strong right now and basically mandatory. You can use other trinkets but nothing will compare to [item=159125] until you ready the 385 ilvl trinkets.[\/div][\/toggler][toggler name=\"Windwalker Monk by Babylonius\" size=3 closed=false][div class=patch-diff]\r\nThe [item=159125] is a very strong trinket for Windwalkers. The large chunk of Agility is very strong, as is the damaging proc. Current sims show it as the hands down best trinket even through much of Uldir\u2019s higher difficulties. However, even though it is strong, current sims appear to show that Windwalkers get slightly less of a damage boost from this trinket when compared to other Fathoms users. Because it\u2019s a BoE crafted item, this isn\u2019t super important, but something worth noting.\r\n[\/div][\/toggler][toggler name=\"Rogues by Aethys & Ravenholdt\" size=3 closed=false][div class=patch-diff]\r\nCurrently a must have for Rogues, primary stat + the effect can be pretty good and is hasted. Will probably stay Best-in-Slot until Mythic raid \/ TF items.\r\n[\/div][\/toggler][toggler name=\"Warriors DPS by Archimtiros\" size=3 closed=false][div class=patch-diff]\r\n[item=159125] is currently BiS for virtually all specs, and is competitive well into raid content.\r\n[\/div][\/toggler][toggler name=\"Retribution Paladin by Skeletor\" size=3 closed=false][div class=patch-diff]\r\n[item=159125] is a very strong ST trinket considering what we have available at this time. In AoE it is not as powerful as it only drops an Anchor on a single target. Once Uldir is released and M+ is available [item=159125] will start to drop off as you acquire the Raid trinkets and higher level dungeon trinkets.[\/div][\/toggler][toggler name=\"Enhancement by Wordup\" size=3 closed=false][div class=patch-diff]\r\nFor Enhancement Darkmoon Deck: Fathoms serves as a solid chunk of Agility this early on, and comes with a reasonably strong direct damage proc. Whilst it's fairly average for the content currently available (i.e. dungeons) due to it being single target in nature, for Uldir it's looking like a very strong choice, in part due to its item level advantage, but mostly due to its consistent output via a high RPPM.\r\n\r\nDo take note this is purely due to the trinket having powerful raw values, as Enhancement has no unique interactions or scaling with the effect.[\/div][\/toggler]\r\n\r\n[h2][item=159126] - Caster[\/h2]\r\n\r\n[toggler name=\"Demonology Warlock by Not\" size=3 closed=false][div class=patch-diff]\r\n[item=159126] is about equal to a 350 WQ trinket. It's value is entirely tied to single target and has no value on multi-target fights. It also has a somewhat limited value in current content since if it procs on a mob that is about to die; there goes your RPPM trinket proc on a dead mob. It might *sim* well but it's not the most practical best trinket.[\/div][\/toggler][toggler name=\"Affliction Warlock by Terryn\" size=3 closed=false][div class=patch-diff]\r\nDeck was really bad before the hotfix, and now the deck is still bad, but less so.  Its still not worth the cash in most cases.[\/div][\/toggler][toggler name=\"Destruction Warlock by Gahddo\" size=3 closed=false][div class=patch-diff]\r\nIt's slightly better than most of the other pre-raid trinkets for ST, but I'd still recommend not buying one. It has no scaling for multi-target and will quickly be outscaled by Mythic+ gear in a few weeks.[\/div][\/toggler][toggler name=\"Boomkin by Tettles\" size=3 closed=false][div class=patch-diff]\r\n[item=159126] is currently a mediocre trinket when compared to some of the other options at a normalized item level. If you are taking it at any significant amount of item level increase will make it worth using on single target, while being pretty mediocre as a whole on multi-target.[\/div][\/toggler][toggler name=\"Elemental by Slander\" size=3 closed=false][div class=patch-diff]\r\nThe recent buffs to [item=159126] brought its damage up to parity with  most item level 340 dungeon trinkets, making it an excellent alternative for gearing before Uldir and Mythic+ dungeons open[\/div][\/toggler][toggler name=\"Mages by Frostedmages\" size=3 closed=false][div class=patch-diff]\r\nAfter the [item=159126] buff, the trinket is considerably better for all three Mage specs. While not BiS, and not able to compete high Warforged or Titanforged trinkets, it does offer a convenient way to obtain a trinket that is about as good as the base mythic dungeon trinkets.[\/div][\/toggler][toggler name=\"Shadow Priest by Anshlun\" size=3 closed=false][div class=patch-diff]\r\nEven after the recent buffs that the deck received, [item=159126] is still pretty mediocre for shadow priests, when compared with dungeon ones at the same item level. It is a viable option, however, if you do not have a trinket around ilvl 340 or higher.[\/div][\/toggler]\r\n\r\n[h2][item=159127] - Healer[\/h2]\r\n\r\n[toggler name=\"Mistweaver Monk by Garg\" size=3 closed=false][div class=patch-diff][item=159127] is better the more targets there are, as healing can be lost on the higher-numbered card buffs because of the bounces. Because of this, it may seem weak in dungeons or other content with very few targets, whereas in raids, the raw intellect and increased number of targets really bring it up above Mythic 0 trinkets. The mana it provides every 20 seconds is a nice little cherry on top of this trinket. This trinket is definitely worth it for the start of Uldir, but not so much dungeons if you already have trinkets with item level close enough to [item=159127] already.[\/div][\/toggler][toggler name=\"Resto Shaman by Maevey\" size=3 closed=false][div class=patch-diff]\r\n[item=159127] is a solid pickup for early stages of gearing due to its high item level, but falls off quickly once m+ and raids open and we have better options with higher item level trinkets.  At an equivalent item level to other trinkets that provide healing (on use healing trinkets and other passive ones) it is significantly stronger, but is outshined by trinkets that give raw stats.  The mana restoration is a small bonus on top of the healing it provides since it only restores mana on shuffles but can proc multiple heals per shuffle.  It also helps fulfill our Chain Heal fantasy since the way the healing works is exactly like Chain Heal with the top most card being the number of bounces.[\/div][\/toggler][toggler name=\"Holy Priest by Shelanne & Mend\" size=3 closed=false][div class=patch-diff]\r\nCompetitive ranking compared to others of similar item level. You will replace it quickly whenever the raids and mythic+ come out. We are mostly using it for the heal component, as the mana returns are very negligible. One upside is that heal component does proc our mastery, Echo of Light.[\/div][\/toggler][toggler name=\"Disc Priest by Shelanne & Mend\" size=3 closed=false][div class=patch-diff]\r\nCompetitive ranking compared to other trinkets of a similar item level, however there is no current way to upgrade it. You will replace it quickly whenever the raids and mythic+ come out. Pretty underwhelming in general.[\/div][\/toggler][toggler name=\"Resto Druid by Voulk\" size=3 closed=false][div class=patch-diff]\r\n[item=159127] is a great starter raid trinket and you\u2019ll use it until you pick up heroic raid or world boss trinkets. It\u2019s weaker in dungeons since the cards can bounce to fewer targets and the mana regeneration is near useless in an environment where you can frequently drink (it still blows lower ilvl alternatives out of the water). You should expect the trinket to contribute 2-3% of your healing on top of the mana effect.[\/div][\/toggler][toggler name=\"Holy Paladin by Adamselene\" size=3 closed=false][div class=patch-diff]\r\n[item=159127] is a below average trinket for Holy Paladins for its item level.  It can still be a solid trinket if you are lacking any higher item level dungeon  or world quest trinkets, but will be quickly replaced once raids and Mythic+ have opened.[\/div][\/toggler]\r\n\r\n[h2][item=159128] - Tank[\/h2]\r\n\r\n[toggler name=\"Prot Paladin by Liminara\" size=3 closed=false][div class=patch-diff]Albeit it is still not incredible as a trinket, it is now usable unlike previous version. It's high item level means that it has a high strength value, and now decent stamina value. While it won't make or break your ability to survive any fights, it can be helpful on fights with high magic damage, due to bigger EHP.[\/div][\/toggler][toggler name=\"Brewmaster Monk by BrewingScribe\" size=3 closed=false][div class=patch-diff]\r\nRandom stamina gains are just too unreliable for tanks in general to look at it favorably, especially for Brewmasters as our effective health is so high due to Stagger. Stamina has little benefit for us, and in fact is detrimental to what self-healing we cam generate via Gift of the Ox.  The healing from the deck is too low to be competitive. You're better off getting Darkmoon Deck: Fathoms  for the baseline agility and extra damage dealt.[\/div][\/toggler][toggler name=\"Guardian Druid by Faide\" size=3 closed=false][div class=patch-diff]\r\nBlockades is a strong option for survivability now. The trinket's effect - which grants Stamina and periodically heals for amounts based on the shuffle of the deck - was recently buffed to grant a significantly larger amount of Stamina and heal for a moderate amount on every reshuffle, making it very strong for increasing effective health and therefore increasing survivability. The flat primary stat is a nice bonus as well, since primary stats provide both an offensive and defensive boost.\r\n\r\nFor Guardian Druids, Blockades benefits additionally from our Mastery, specifically the portion which increases our maximum health. \r\n\r\nCurrently Blockades is one of the strongest trinkets for survivability available. Its naturally high item level makes it very competitive and it will likely remain that way until Uldir is released and the item level cap is raised.[\/div][\/toggler][toggler name=\"Vengeance DH by Munkky\" size=3 closed=false][div class=patch-diff]\r\nThis is the tank trinket for Battle for Azeroth that provides a base amount of main stat (strength\/agility) and its \"deck effect\" is to give stamina and heal upon revealing a new card. \r\nIn general, stamina trinkets aren't great for tanks, as damage reduction is far stronger than more health, due to more health requiring healing to recover that \"extra survivability\". Unless other trinkets are terrible or the stamina trinket provides a large sum of stamina, it will rarely be taken. With this in mind however, the [item=159128] trinket was buffed shortly after the launch of the expansion, which has made it potentially the best Pre Raid trinket for every tank. It provides main stat, which every tank desires heavily, and Stamina, which works well for anyone. \r\nSo the answer is, if you want one of the best options for survival from pre raid, then use this trinket.[\/div][\/toggler][toggler name=\"Blood DK by Melekus\" size=3 closed=false][div class=patch-diff]\r\nIt's a relatively good defensive trinket for blood: its high ilevel and strength budget, combined with a moderate stamina increase, make it one of the best defensive trinkets available at the moment.\r\nRealistically, this trinket will not make or break your ability to survive encounters, and if you don't want to invest in [item=159128], there are still a lot of good stat oriented, or even dps oriented, trinkets out there.[\/div][\/toggler][toggler name=\"Prot Warrior by Marok\" size=3 closed=false][div class=patch-diff]\r\nIn addition to the base Strength, [item=159128] has two equip effects. A small heal (when the deck is shuffled), and a Stamina gain based on the shuffle. The heal portion of this trinket is a bit weak, but the average Stamina increase is significant. Not only that, since the trinket has a high item level compared to most of the available gear right now, the Strength on the trinket provides a (relatively) decent survivability boost via Vanguard. In terms of pure survivability, Blockades is the best trinket pre-Uldir, and will only be beat during Uldir by high item level dungeon trinkets, such as Jes' Howler.[\/div][\/toggler]", "news-post-5b81a133ac83a", {"uid":286563,"inNewsType":2,"allow":"WH.markup.CLASS.ADMIN"});
        //]]></script>

<noscript></noscript>
    </div>

[fguillot]

The content scraper doesn't evaluate Javascript and the tags <script/> and <noscript/> are stripped by the content sanitizer for security reasons.

I guess this website has been designed to mess with crawlers. That also means that it doesn't work without Javascript.

[AdamVig]

The Kinja family of blogs (Lifehacker, Gizmodo, Jalopnik, etc.) also cause problems for the scraper because they use JavaScript to render their content.

The scraper could be replaced with one that simulates the browser more closely, but that would obviously be quite a bit of added complexity, so it may not be worth it.