Re-Enable federated queries

[Daniel-Mietchen]

Describe the bug

Federated queries currently give errors like Unknown error: Service URI https://query.wikidata.org/sparql is not allowed

Expected behavior

Federated query runs for endpoints that are included in some kind of whitelist/ allowlist that should contain Wikidata.

To Reproduce Steps to reproduce the behavior:

1. Go to https://portal.mardi4nfdi.de/w/index.php?title=User:Daniel/Queries/Federated&oldid=5810775
2. Try any of the queries in the "MaRDI and" sections, e.g. this one
3. See error

Screenshots

Additional context

For Wikidata, the procedure is described at https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service#Updating_federation_allowlist .

Checklist for this issue: (Some checks for making sure this issue is completely formulated)

• [x] Assignee has been set for this issue
• [X] All fields of the issue have been filled
• [X] the main MaRDI project has been assigned as project

[physikerwelt]

@eloiferrer, can you check if any custom config might have been overwritten during the container migration process #451 ?+

[physikerwelt]

from @eloiferrer :

I don't think we have ever explicitly enabled or disabled this (I did not at least). That this is not working now might just be a consequence of having updated the WDQS versions, which maybe now needs this to be explicitly enabled, but I am just guessing. On the other hand, having to many Wikidata requests from our server was also resulting in 'Too many requests' errors from Wikdata, see https://github.com/MaRDI4NFDI/portal-compose/issues/416

[physikerwelt]

Investigating this NOW:

I understand that this whitelist should be used https://github.com/MaRDI4NFDI/wikibase-release-pipeline/blob/main/build/WDQS/whitelist.txt Let's see via portainer

[physikerwelt]

The list is in portainer... en passant I complained about the term https://phabricator.wikimedia.org/T355893

[physikerwelt]

I can't understand what's wrong, this needs deeper investigation. I suggest we schedule this work in next weeks meeting?

[physikerwelt]

@Daniel-Mietchen can you bring that up on next monday?

[eloiferrer]

I think the change from whitelist to allowlist has already taken place (https://phabricator.wikimedia.org/T343856). And this might be why it is not working for us, since we seem to still have the whitelist.txt. So far, I have not found further details on how this is configured now.

[eloiferrer]

I see now that whitelist.txt is indeed still used in the wdqs image we are pulling: https://github.com/MaRDI4NFDI/wikibase-release-pipeline/tree/main/build/WDQS

[physikerwelt]

I will try to restart the service, maybe this helps https://github.com/blazegraph/database/issues/102#issuecomment-855805011

[eloiferrer]

Otherwise we can try copying whitelist.txt as allowlist.txt

[physikerwelt]

The query still does not work as it runs into wmde rate limits. But at least it is a different error

Caused by: java.io.IOException: Broken pipe

    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)

    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)

    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)

    at sun.nio.ch.IOUtil.write(IOUtil.java:51)

    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)

    at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:264)

    ... 37 common frames omitted

19:16:41.164 [qtp861842890-37] INFO  o.w.q.r.b.t.ThrottlingFilter - A request is being throttled.