Open Luka-Loncar opened 4 weeks ago
I just ran a web scraping query and the following is a perfect example of what can happen. The miner that is in Germany (.de) is getting a score of 0.98, whereas US based miners are getting a score of 1, b/c that's where the validator / source of truth logic is residing. It's good to see Germany not get TOO punished, but we should consider if we need to address this or not.
[
{
"response": {
"pages": [
"https://www.google.com/imghp?hl=en&tab=wi",
"https://maps.google.com/maps?hl=en&tab=wl",
"https://play.google.com/?hl=en&tab=w8",
"https://www.youtube.com/?tab=w1",
"https://news.google.com/?tab=wn",
"https://mail.google.com/mail/?tab=wm",
"https://drive.google.com/?tab=wo",
"https://www.google.com/intl/en/about/products?tab=wh",
"http://www.google.com/history/optout?hl=en",
"https://www.google.com/preferences?hl=en",
"https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=GAZAAQ",
"https://www.google.com/advanced_search?hl=en&authuser=0",
"https://www.google.com/intl/en/ads/",
"https://www.google.com/services/",
"https://www.google.com/intl/en/about.html",
"https://www.google.com/intl/en/policies/privacy/",
"https://www.google.com/intl/en/policies/terms/"
],
"sections": null
},
"uid": 1,
"score": 1,
"latency": 1.45477294921875
},
{
"response": {
"pages": [
"https://www.google.com/imghp?hl=en&tab=wi",
"https://maps.google.com/maps?hl=en&tab=wl",
"https://play.google.com/?hl=en&tab=w8",
"https://www.youtube.com/?tab=w1",
"https://news.google.com/?tab=wn",
"https://mail.google.com/mail/?tab=wm",
"https://drive.google.com/?tab=wo",
"https://www.google.com/intl/en/about/products?tab=wh",
"http://www.google.com/history/optout?hl=en",
"https://www.google.com/preferences?hl=en",
"https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=GAZAAQ",
"https://www.google.com/advanced_search?hl=en&authuser=0",
"https://www.google.com/intl/en/ads/",
"https://www.google.com/services/",
"https://www.google.com/intl/en/about.html",
"https://www.google.com/intl/en/policies/privacy/",
"https://www.google.com/intl/en/policies/terms/"
],
"sections": null
},
"uid": 245,
"score": 1,
"latency": 8.330689907073975
},
{
"response": {
"pages": [
"https://www.google.com/imghp?hl=de&tab=wi",
"https://maps.google.de/maps?hl=de&tab=wl",
"https://play.google.com/?hl=de&tab=w8",
"https://www.youtube.com/?tab=w1",
"https://news.google.com/?tab=wn",
"https://mail.google.com/mail/?tab=wm",
"https://drive.google.com/?tab=wo",
"https://www.google.de/intl/de/about/products?tab=wh",
"http://www.google.de/history/optout?hl=de",
"https://www.google.com/preferences?hl=de",
"https://accounts.google.com/ServiceLogin?hl=de&passive=true&continue=https://www.google.com/&ec=GAZAAQ",
"https://www.google.com/advanced_search?hl=de&authuser=0",
"https://www.google.com/intl/de/ads/",
"https://www.google.com/services/",
"https://www.google.com/intl/de/about.html",
"https://www.google.com/setprefdomain?prefdom=DE&prev=https://www.google.de/&sig=K_TebNuDUJtfz2gGyHy7LckE6atX4%3D",
"https://www.google.com/intl/de/policies/privacy/",
"https://www.google.com/intl/de/policies/terms/"
],
"sections": null
},
"uid": 53,
"score": 0.9889886975288391,
"latency": 14.976975202560425
}
]
Interesting point here @grantdfoster and good research. I think knowing that this is an minor edge case is sufficient for this now.
Moving to backlog until further notice since web scraping isn't the main focus for us right now.
how do we accommodate different responses for different parts of the world (i.e. my website might look different than yours because of our locations, even though its the same URL)