When running runJobs we see HTTP429 errors from wikidata

physikerwelt commented 8 months ago

Describe the issue I see the following error in the log

Error endpoint: https://query.wikidata.org/sparql
Error http_response_code: 429
Error message: <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 429 Too Many Requests - Please retry in 3 seconds.</title>
</head>
<body><h2>HTTP ERROR 429</h2>
<p>Problem accessing /bigdata/namespace/wdq/sparql. Reason:
<pre>    Too Many Requests - Please retry in 3 seconds.</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.12.v20180830</a><hr/>

</body>
</html>

Error number: 0
Error message: 

Error data: Array
(
    [query] => PREFIX target: <http://www.wikidata.org/entity/Q56755528>

SELECT ?node ?nodeLabel ?childNode ?childNodeLabel 
WITH {
  SELECT DISTINCT ?node ?childNode WHERE {
      BIND(target: AS ?node)
      ?node ?p ?i.
      ?childNode ?x ?p.
      ?childNode rdf:type wikibase:Property.
      FILTER(STRSTARTS(STR(?i), "http://www.wikidata.org/entity/Q"))
      FILTER(STRSTARTS(STR(?childNode), "http://www.wikidata.org/entity/P"))
    }
  LIMIT 5000
} AS %nodes
WITH {
  SELECT DISTINCT ?childNode ?node ?rgb WHERE {
      BIND("EFFBD8" AS ?rgb)
      target: ?p ?childNode.
      ?node ?x ?p.
      ?node rdf:type wikibase:Property.
      FILTER(STRSTARTS(STR(?childNode), "http://www.wikidata.org/entity/Q"))
    }
  LIMIT 5000
} AS %childNodes
WHERE {
  {
    INCLUDE %nodes
  }
  UNION
  {
    INCLUDE %childNodes
  }
  FILTER (?node != target:)
  SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }        
}
ORDER BY ASC(?nodeLabel) ASC(?childNodeLabel)
    [output] => xml
    [Accept] => application/sparql-results+xml
)

I wonder where this comes from. Are any pages trying to access wikidata.org?

eloiferrer commented 7 months ago

Just in case this is still relevant, all the Profile pages for authors send requests to Wikidata. This is where the query comes from: https://portal.mardi4nfdi.de/wiki/Template:Scholia-author_context

physikerwelt commented 6 months ago

The goal is not to query data from wikidata on the fly, but to import relevant data from wikidata.

MaRDI4NFDI / portal-compose

When running runJobs we see HTTP429 errors from wikidata #416