Open michaelwechner opened 11 years ago
Spiders/Crawlers are often accessing websites directly without referer. Yanel is then setting the referer tag to null, e.g.
52781864 2012-11-15 01:01:00,891 -0500 [http-bio-9390-exec-21] INFO Access - url:http%3A%2F%2Fwww.yanel.org%2Fen%2Fabout.html%3Fyanel.resource.viewid%3Ddefault d:a9ee9c0b655cbcbd c:YA-5ce13105-d222-4765-b425-bc3570b6f1b4 ref:null ua:Mozilla%2F5.0+%28compatible%3B+Baiduspider%2F2.0%3B+%2Bhttp%3A%2F%2Fwww.baidu.com%2Fsearch%2Fspider.html%29 t:overview%2CAbout+Yanel a-lang:en-US sid:11AF83624716736B020B4C22CEAADFFA http-status:200 ip:180.76.6.225
The we should probably rather not set the tag/field at all if there is no referer.
Spiders/Crawlers are often accessing websites directly without referer. Yanel is then setting the referer tag to null, e.g.
52781864 2012-11-15 01:01:00,891 -0500 [http-bio-9390-exec-21] INFO Access - url:http%3A%2F%2Fwww.yanel.org%2Fen%2Fabout.html%3Fyanel.resource.viewid%3Ddefault d:a9ee9c0b655cbcbd c:YA-5ce13105-d222-4765-b425-bc3570b6f1b4 ref:null ua:Mozilla%2F5.0+%28compatible%3B+Baiduspider%2F2.0%3B+%2Bhttp%3A%2F%2Fwww.baidu.com%2Fsearch%2Fspider.html%29 t:overview%2CAbout+Yanel a-lang:en-US sid:11AF83624716736B020B4C22CEAADFFA http-status:200 ip:180.76.6.225
The we should probably rather not set the tag/field at all if there is no referer.