search

ariya / phantomjs

Scriptable Headless Browser
http://phantomjs.org
BSD 3-Clause "New" or "Revised" License
29.46k stars 5.75k forks source link

document.referrer is empty (windows xp with phantomjs 1.9.7) #11944

Closed tenth closed 4 years ago

tenth commented 10 years ago 
var page = require('webpage').create();
page.customHeaders = {
    "Referer":"http://www.google.com/ncr"
}
page.open('http://www.whatismyreferer.com/',function(status){
        if(status == 'success'){
            var curr_ref = page.evaluate(function(){
                //return window.document.title;
                return window.document.referrer;
            });
            console.log('referrer is:' + curr_ref);
            setTimeout(function(){ phantom.exit();},3000);
        }else{
            console.log('error');
            phantom.exit();
        }
});

request

document.referrer is empty,but custom header seems has sent by request. os: windows xp useragent: Mozilla/5.0 (Window NT 5.1) AppletWebKit/534.34 (HTML, like Gecko)

i tried lower version phantomjs get same result

2014-02-03T12:21:18 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=<filename>' to enable persisten cookie storage) 
2014-02-03T12:21:18 [DEBUG] Phantom - execute: Configuration 
2014-02-03T12:21:18 [DEBUG]      0 objectName : "" 
2014-02-03T12:21:18 [DEBUG]      1 cookiesFile : "" 
2014-02-03T12:21:18 [DEBUG]      2 diskCacheEnabled : "false" 
2014-02-03T12:21:18 [DEBUG]      3 maxDiskCacheSize : "-1" 
2014-02-03T12:21:18 [DEBUG]      4 ignoreSslErrors : "false" 
2014-02-03T12:21:18 [DEBUG]      5 localToRemoteUrlAccessEnabled : "false" 
2014-02-03T12:21:18 [DEBUG]      6 outputEncoding : "UTF-8" 
2014-02-03T12:21:18 [DEBUG]      7 proxyType : "http" 
2014-02-03T12:21:18 [DEBUG]      8 proxy : ":1080" 
2014-02-03T12:21:18 [DEBUG]      9 proxyAuth : ":" 
2014-02-03T12:21:18 [DEBUG]      10 scriptEncoding : "UTF-8" 
2014-02-03T12:21:18 [DEBUG]      11 webSecurityEnabled : "true" 
2014-02-03T12:21:18 [DEBUG]      12 offlineStoragePath : "" 
2014-02-03T12:21:18 [DEBUG]      13 offlineStorageDefaultQuota : "-1" 
2014-02-03T12:21:18 [DEBUG]      14 printDebugMessages : "true" 
2014-02-03T12:21:18 [DEBUG]      15 javascriptCanOpenWindows : "true" 
2014-02-03T12:21:18 [DEBUG]      16 javascriptCanCloseWindows : "true" 
2014-02-03T12:21:18 [DEBUG]      17 sslProtocol : "sslv3" 
2014-02-03T12:21:18 [DEBUG]      18 sslCertificatesPath : "" 
2014-02-03T12:21:18 [DEBUG]      19 webdriver : ":" 
2014-02-03T12:21:18 [DEBUG]      20 webdriverLogFile : "" 
2014-02-03T12:21:18 [DEBUG]      21 webdriverLogLevel : "INFO" 
2014-02-03T12:21:18 [DEBUG]      22 webdriverSeleniumGridHub : "" 
2014-02-03T12:21:18 [DEBUG] Phantom - execute: Script & Arguments 
2014-02-03T12:21:18 [DEBUG]      script: "o.js" 
2014-02-03T12:21:18 [DEBUG] Phantom - execute: Starting normal mode 
2014-02-03T12:21:18 [DEBUG] WebPage - setupFrame "" 
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/_coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/package.json" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./parser.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./nodes.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././scope.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-03T12:21:18 [DEBUG] WebPage - updateLoadingProgress: 10 
2014-02-03T12:21:19 [DEBUG] WebPage - updateLoadingProgress: 30 
2014-02-03T12:21:19 [DEBUG] WebPage - updateLoadingProgress: 38 
2014-02-03T12:21:19 [DEBUG] WebPage - updateLoadingProgress: 100 
2014-02-03T12:21:19 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {

                //return window.document.title;

                return window.document.referrer;

            })(); })()" 
2014-02-03T12:21:19 [DEBUG] WebPage - setupFrame "" 
2014-02-03T12:21:19 [DEBUG] WebPage - evaluateJavaScript result QVariant(QString, "") 
2014-02-03T12:21:19 [DEBUG] WebPage - setupFrame "" 
referrer is:
tenth commented 10 years ago

With phantomjs-1.9.7-linux-i686.tar.bz2 on centos 6.5 got the same problem

vokkim commented 10 years ago

Seems that the bug is in document.referrer. This small script helps to compare what the server receives and how the document.referrer remains empty.

var page = require('webpage').create()
page.customHeaders = {'Referer': 'http://www.google.com/'}
page.onLoadFinished = function(){
  var jsRef = page.evaluate(function() {
    return document.referrer
  })
  var pageRef = page.evaluate(function() {
    return document.getElementsByTagName("p")[0].textContent.trim()
  })
  console.log('JavaScript document.referrer is: ' + jsRef)
  console.log('http://www.whatismyreferer.com/ found this referer: ' + pageRef)
  phantom.exit()
}
page.open('http://www.whatismyreferer.com/')

With OS X and Homebrew installed PhantomJS 1.9.7 it produces:

$ phantomjs --debug=true fakeref.js
2014-02-11T14:25:36 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=<filename>' to enable persisten cookie storage)
2014-02-11T14:25:36 [DEBUG] Phantom - execute: Configuration
2014-02-11T14:25:36 [DEBUG]      0 objectName : ""
2014-02-11T14:25:36 [DEBUG]      1 cookiesFile : ""
2014-02-11T14:25:36 [DEBUG]      2 diskCacheEnabled : "false"
2014-02-11T14:25:36 [DEBUG]      3 maxDiskCacheSize : "-1"
2014-02-11T14:25:36 [DEBUG]      4 ignoreSslErrors : "false"
2014-02-11T14:25:36 [DEBUG]      5 localToRemoteUrlAccessEnabled : "false"
2014-02-11T14:25:36 [DEBUG]      6 outputEncoding : "UTF-8"
2014-02-11T14:25:36 [DEBUG]      7 proxyType : "http"
2014-02-11T14:25:36 [DEBUG]      8 proxy : ":1080"
2014-02-11T14:25:36 [DEBUG]      9 proxyAuth : ":"
2014-02-11T14:25:36 [DEBUG]      10 scriptEncoding : "UTF-8"
2014-02-11T14:25:36 [DEBUG]      11 webSecurityEnabled : "true"
2014-02-11T14:25:36 [DEBUG]      12 offlineStoragePath : ""
2014-02-11T14:25:36 [DEBUG]      13 offlineStorageDefaultQuota : "-1"
2014-02-11T14:25:36 [DEBUG]      14 printDebugMessages : "true"
2014-02-11T14:25:36 [DEBUG]      15 javascriptCanOpenWindows : "true"
2014-02-11T14:25:36 [DEBUG]      16 javascriptCanCloseWindows : "true"
2014-02-11T14:25:36 [DEBUG]      17 sslProtocol : "sslv3"
2014-02-11T14:25:36 [DEBUG]      18 sslCertificatesPath : ""
2014-02-11T14:25:36 [DEBUG]      19 webdriver : ":"
2014-02-11T14:25:36 [DEBUG]      20 webdriverLogFile : ""
2014-02-11T14:25:36 [DEBUG]      21 webdriverLogLevel : "INFO"
2014-02-11T14:25:36 [DEBUG]      22 webdriverSeleniumGridHub : ""
2014-02-11T14:25:36 [DEBUG] Phantom - execute: Script & Arguments
2014-02-11T14:25:36 [DEBUG]      script: "fakeref.js"
2014-02-11T14:25:36 [DEBUG] Phantom - execute: Starting normal mode
2014-02-11T14:25:36 [DEBUG] WebPage - setupFrame ""
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/_coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/package.json" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./lexer.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././helpers.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./parser.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./helpers.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./nodes.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././scope.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././helpers.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././lexer.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r") ) )
2014-02-11T14:25:36 [DEBUG] WebPage - updateLoadingProgress: 10
2014-02-11T14:25:36 [DEBUG] WebPage - updateLoadingProgress: 30
2014-02-11T14:25:36 [DEBUG] WebPage - updateLoadingProgress: 100
2014-02-11T14:25:36 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {
    return document.referrer;
  })(); })()"
2014-02-11T14:25:36 [DEBUG] WebPage - setupFrame ""
2014-02-11T14:25:36 [DEBUG] WebPage - evaluateJavaScript result QVariant(QString, "")
2014-02-11T14:25:36 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {
    return document.getElementsByTagName("p")[0].textContent.trim();
  })(); })()"
2014-02-11T14:25:36 [DEBUG] WebPage - evaluateJavaScript result QVariant(QString, "http://www.google.com/")
JavaScript document.referrer is:
http://www.whatismyreferer.com/ found this referer: http://www.google.com/
JamesMGreene commented 10 years ago

Is it supported in Phantom at all? Try:

var isSupported = page.evaluate(function() { return "referrer" in document; });

Also note some key text on the MDN page for document.referrer:

The value is an empty string if the user navigated to the page directly (not through a link, but, for example, via a bookmark).

tenth commented 10 years ago 
var page = require('webpage').create();
page.customHeaders = {
    "Referer":"http://www.google.com/ncr"
}
page.open('http://www.whatismyreferer.com/',function(status){
        if(status == 'success'){
            var curr_ref = page.evaluate(function(){
                //return window.document.title;
                return window.document.referrer;
            });
            var isSupported = page.evaluate(function() { return "referrer" in document; });
            console.log('referrer is: ' + curr_ref);
            console.log('is supported: ' + isSupported);
            setTimeout(function(){ phantom.exit();},3000);
        }else{
            console.log('error');
            phantom.exit();
        }
});
2014-02-12T13:45:38 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=<filename>' to enable persisten cookie storage) 
2014-02-12T13:45:38 [DEBUG] Phantom - execute: Configuration 
2014-02-12T13:45:38 [DEBUG]      0 objectName : "" 
2014-02-12T13:45:38 [DEBUG]      1 cookiesFile : "" 
2014-02-12T13:45:38 [DEBUG]      2 diskCacheEnabled : "false" 
2014-02-12T13:45:38 [DEBUG]      3 maxDiskCacheSize : "-1" 
2014-02-12T13:45:38 [DEBUG]      4 ignoreSslErrors : "false" 
2014-02-12T13:45:38 [DEBUG]      5 localToRemoteUrlAccessEnabled : "false" 
2014-02-12T13:45:38 [DEBUG]      6 outputEncoding : "UTF-8" 
2014-02-12T13:45:38 [DEBUG]      7 proxyType : "http" 
2014-02-12T13:45:38 [DEBUG]      8 proxy : ":1080" 
2014-02-12T13:45:38 [DEBUG]      9 proxyAuth : ":" 
2014-02-12T13:45:38 [DEBUG]      10 scriptEncoding : "UTF-8" 
2014-02-12T13:45:38 [DEBUG]      11 webSecurityEnabled : "true" 
2014-02-12T13:45:38 [DEBUG]      12 offlineStoragePath : "" 
2014-02-12T13:45:38 [DEBUG]      13 offlineStorageDefaultQuota : "-1" 
2014-02-12T13:45:38 [DEBUG]      14 printDebugMessages : "true" 
2014-02-12T13:45:38 [DEBUG]      15 javascriptCanOpenWindows : "true" 
2014-02-12T13:45:38 [DEBUG]      16 javascriptCanCloseWindows : "true" 
2014-02-12T13:45:38 [DEBUG]      17 sslProtocol : "sslv3" 
2014-02-12T13:45:38 [DEBUG]      18 sslCertificatesPath : "" 
2014-02-12T13:45:38 [DEBUG]      19 webdriver : ":" 
2014-02-12T13:45:38 [DEBUG]      20 webdriverLogFile : "" 
2014-02-12T13:45:38 [DEBUG]      21 webdriverLogLevel : "INFO" 
2014-02-12T13:45:38 [DEBUG]      22 webdriverSeleniumGridHub : "" 
2014-02-12T13:45:38 [DEBUG] Phantom - execute: Script & Arguments 
2014-02-12T13:45:38 [DEBUG]      script: "test.js" 
2014-02-12T13:45:38 [DEBUG] Phantom - execute: Starting normal mode 
2014-02-12T13:45:38 [DEBUG] WebPage - setupFrame "" 
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/_coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/package.json" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./parser.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./nodes.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././scope.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r") ) )  
2014-02-12T13:45:38 [DEBUG] WebPage - updateLoadingProgress: 10 
2014-02-12T13:45:39 [DEBUG] WebPage - updateLoadingProgress: 30 
2014-02-12T13:45:39 [DEBUG] WebPage - updateLoadingProgress: 38 
2014-02-12T13:45:39 [DEBUG] WebPage - updateLoadingProgress: 100 
2014-02-12T13:45:39 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {

                //return window.document.title;

                return window.document.referrer;

            })(); })()" 
2014-02-12T13:45:39 [DEBUG] WebPage - setupFrame "" 
2014-02-12T13:45:39 [DEBUG] WebPage - evaluateJavaScript result QVariant(QString, "") 
2014-02-12T13:45:39 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () { return "referrer" in document; })(); })()" 
2014-02-12T13:45:39 [DEBUG] WebPage - evaluateJavaScript result QVariant(bool, true) 
2014-02-12T13:45:39 [DEBUG] WebPage - setupFrame "" 
referrer is: 
is supported: true

"referrer" in document it's just return true

vokkim commented 10 years ago

@JamesMGreene It seems to work in this little gist I found: https://gist.github.com/papoms/3453137 However, the referrer is faked here by first navigating to the referrer URL, creating a link to the target page and then clicking the link. Thus the document.referrer works as intended in that case, but not when we just define it as an additional HTTP param.

Here's a bit more specific HTML 5 spec about document.referrer: http://www.w3.org/html/wg/drafts/html/master/dom.html#dom-document-referrer

In the case of HTTP, the referrer IDL attribute will match the Referer (sic) header that was sent when fetching the current page.

stale[bot] commented 4 years ago

Due to our very limited maintenance capacity, we need to prioritize our development focus on other tasks. Therefore, this issue will be automatically closed (see #15395 for more details). In the future, if we see the need to attend to this issue again, then it will be reopened. Thank you for your contribution!