dharmafly / noodle

A node server and module which allows for cross-domain page scraping on web documents with JSONP or POST.
https://noodle.dharmafly.com/
745 stars 69 forks source link

POST Request not working on particular page #98

Closed grssnbchr closed 9 years ago

grssnbchr commented 9 years ago

I don't know if this is a bug or if I am doing something wrong, but I crosschecked the issue with an artificial POST Request both using the Firefox developer console and a Firefox Addon called "Poster". With both methods, I got the expected answer from the website I am querying. However, it does not work with Noodle.js as a module. Here's my code:

var query = {
    url: 'http://www.wgzimmer.ch/wgzimmer/search/mate.html?',
    type: 'html',
    selector: 'ul.list li',
    post: {
        'priceMin': '800',
        'priceMax': '1500',
        'state': '"zurich-stadt"',
        'student': '"false"',
        'country': '"ch"',
        'orderBy': '"MetaData/@mgnl:lastmodified"',
        'orderDir': '"descending"',
        'startSearchMate': '"true"',
        'wgStartSearch': '"true"',
        'query': ''
    },
    'cache': false
};
var noodle = require('noodlejs');
noodle.query(query).then(function(results){
    console.log(results.results);
}).fail(function(error){
    console.log(error.message);
})

The error I get is: error: 'Could not match with that selector or extract value' This is because the result doesn't even contain ul.list li elements, as only the search form is returned but not the actual results.

What am I doing wrong? You can use the exact same query parameters and a tool such as Poster to verify that this should actually work.

premasagar commented 9 years ago

Hi @wnstnsmth, I had a quick look. If you remove the selector parameter, you'll get the full HTML returned. There are no elements in the HTML that match ul.list li. Noodle is faithfully giving you the server's response. I'm not sure you get different results by using a different client. Perhaps the headers sent to the server are different in those two cases?