matthewmueller / x-ray

The next web scraper. See through the <html> noise.
MIT License
5.87k stars 349 forks source link

Null error, but result returns empty string when the classname does indeed exist #264

Closed austinyearlykim closed 7 years ago

austinyearlykim commented 7 years ago

Subject of the issue

x-ray works for the " 'http://www.google.com', 'title' " example, but xray doesnt detect this classname even though I can see it in the developer console

const xray = require('x-ray')(); xray('http://shop.tcgplayer.com/magic/product/show?ProductName=Jace%2c+the+Mind+Sculptor', '.product__market-price-value')((err, result) => { console.log(err); console.log(result); })

Describe your issue here. err is null and result misses and returns an empty string

Your environment

Steps to reproduce

Tell us how to reproduce this issue. Copy and paste above code in node and run.

Expected behaviour

Tell us what should happen. err should be null, results should equal a string dollar amount

Actual behaviour

err IS null, but results returns an empty string Tell us what happens instead.

DannyCork commented 7 years ago

I'm able to reproduce the issue but only on the site you have detailed. I've tried other tags and get Request unsuccessful. Incapsula incident ID: xxxxxx-xxxxxxx back.

I've tried multiple other sites and they work just fine. I'm guessing that the Incapsula cdn or something unique to your site is filtering and rejecting these requests. Try creating a very basic page with a couple of lines of html including the title tag and see does it happen here.

austinyearlykim commented 7 years ago

Hi @DannyCork. Thank you for the response. I've boiled this problem down to the site's Incapsula integration and not a problem with X-Ray. I'm going to close this thread.