Spike: web scrape scoring

[Luka-Loncar]

how do we accommodate different responses for different parts of the world (i.e. my website might look different than yours because of our locations, even though its the same URL)

[grantdfoster]

I just ran a web scraping query and the following is a perfect example of what can happen. The miner that is in Germany (.de) is getting a score of 0.98, whereas US based miners are getting a score of 1, b/c that's where the validator / source of truth logic is residing. It's good to see Germany not get TOO punished, but we should consider if we need to address this or not.

[
  {
    "response": {
      "pages": [
        "https://www.google.com/imghp?hl=en&tab=wi",
        "https://maps.google.com/maps?hl=en&tab=wl",
        "https://play.google.com/?hl=en&tab=w8",
        "https://www.youtube.com/?tab=w1",
        "https://news.google.com/?tab=wn",
        "https://mail.google.com/mail/?tab=wm",
        "https://drive.google.com/?tab=wo",
        "https://www.google.com/intl/en/about/products?tab=wh",
        "http://www.google.com/history/optout?hl=en",
        "https://www.google.com/preferences?hl=en",
        "https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=GAZAAQ",
        "https://www.google.com/advanced_search?hl=en&authuser=0",
        "https://www.google.com/intl/en/ads/",
        "https://www.google.com/services/",
        "https://www.google.com/intl/en/about.html",
        "https://www.google.com/intl/en/policies/privacy/",
        "https://www.google.com/intl/en/policies/terms/"
      ],
      "sections": null
    },
    "uid": 1,
    "score": 1,
    "latency": 1.45477294921875
  },
  {
    "response": {
      "pages": [
        "https://www.google.com/imghp?hl=en&tab=wi",
        "https://maps.google.com/maps?hl=en&tab=wl",
        "https://play.google.com/?hl=en&tab=w8",
        "https://www.youtube.com/?tab=w1",
        "https://news.google.com/?tab=wn",
        "https://mail.google.com/mail/?tab=wm",
        "https://drive.google.com/?tab=wo",
        "https://www.google.com/intl/en/about/products?tab=wh",
        "http://www.google.com/history/optout?hl=en",
        "https://www.google.com/preferences?hl=en",
        "https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/&ec=GAZAAQ",
        "https://www.google.com/advanced_search?hl=en&authuser=0",
        "https://www.google.com/intl/en/ads/",
        "https://www.google.com/services/",
        "https://www.google.com/intl/en/about.html",
        "https://www.google.com/intl/en/policies/privacy/",
        "https://www.google.com/intl/en/policies/terms/"
      ],
      "sections": null
    },
    "uid": 245,
    "score": 1,
    "latency": 8.330689907073975
  },
  {
    "response": {
      "pages": [
        "https://www.google.com/imghp?hl=de&tab=wi",
        "https://maps.google.de/maps?hl=de&tab=wl",
        "https://play.google.com/?hl=de&tab=w8",
        "https://www.youtube.com/?tab=w1",
        "https://news.google.com/?tab=wn",
        "https://mail.google.com/mail/?tab=wm",
        "https://drive.google.com/?tab=wo",
        "https://www.google.de/intl/de/about/products?tab=wh",
        "http://www.google.de/history/optout?hl=de",
        "https://www.google.com/preferences?hl=de",
        "https://accounts.google.com/ServiceLogin?hl=de&passive=true&continue=https://www.google.com/&ec=GAZAAQ",
        "https://www.google.com/advanced_search?hl=de&authuser=0",
        "https://www.google.com/intl/de/ads/",
        "https://www.google.com/services/",
        "https://www.google.com/intl/de/about.html",
        "https://www.google.com/setprefdomain?prefdom=DE&prev=https://www.google.de/&sig=K_TebNuDUJtfz2gGyHy7LckE6atX4%3D",
        "https://www.google.com/intl/de/policies/privacy/",
        "https://www.google.com/intl/de/policies/terms/"
      ],
      "sections": null
    },
    "uid": 53,
    "score": 0.9889886975288391,
    "latency": 14.976975202560425
  }
]

[teslashibe]

Interesting point here @grantdfoster and good research. I think knowing that this is an minor edge case is sufficient for this now.

[Luka-Loncar]

Moving to backlog until further notice since web scraping isn't the main focus for us right now.