How to test queries from remote

[kleegrewec]

Hi Frank,

When I create my sparql-ms deployment I am able to test the available services from localhost according to your documentation (https://github.com/frmichel/sparql-micro-service/blob/master/doc/04-install.md#test-sparql-querying) but when I try to query from a distant host I get a 404 URL not found as a response. To test from remote I simply replaced the "localhost" in the example test by the target host.

Looking into kgram.log file I find the following for the remote query:

10.0.7.10 - - [28/Jul/2022:15:33:38 +0000] "GET /service/siguard-pdp/getAllChannels?query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D HTTP/1.1" 404 456 "-" "curl/7.58.0"

And for the localhost based query:

[Thu Jul 28 15:34:27.796260 2022] [php7:notice] [pid 17] [client 127.0.0.1:47892] PHP Notice:  Undefined index: custom_parameter in /var/www/html/sparql-ms/src/common/Configuration.php on line 81
[Thu Jul 28 15:34:27.796320 2022] [php7:warn] [pid 17] [client 127.0.0.1:47892] PHP Warning:  sizeof(): Parameter must be an array or an object that implements Countable in /var/www/html/sparql-ms/src/common/Configuration.php on line 81
[Thu Jul 28 15:34:27.814082 2022] [php7:warn] [pid 17] [client 127.0.0.1:47892] PHP Warning:  Invalid argument supplied for foreach() in /var/www/html/sparql-ms/src/common/Utils.php on line 315
127.0.0.1 - - [28/Jul/2022:15:34:27 +0000] "GET /sparql-ms/src/sparqlms/service.php?root_url=http://localhost/service&querymode=sparql&service=siguard-pdp/getAllChannels&query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D HTTP/1.1" 200 9795 "-" "curl/7.58.0"
172.19.0.1 - - [28/Jul/2022:15:34:27 +0000] "GET /service/siguard-pdp/getAllChannels?query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D HTTP/1.1" 200 9783 "-" "curl/7.58.0"

Do I need to send my query to a different URL to get things working?

Any help will be appreciated,

thanks, Christian

[kleegrewec]

The remote query look like this:

SELECT='select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D'

curl -vvvv --header "Accept: application/sparql-results+json" \
  "http://dev-sag-sms.ensure2.siemens.cloud/service/siguard-pdp/getAllChannels?query=${SELECT}"

[frmichel]

Hi @kleegrewec, that should work exactly the same, so there must be a mismatch somewhere in the network config of the server or of Docker.

Please try this: run the query (with hostname dev-sag-sms.ensure2.siemens.cloud) from the server where you run the SPARQL µservice.

Also, please can you issue the following queries and show me the result:

cat /etc/hosts
ifconfig -a (or equivalent on your system)
nslookup dev-sag-sms.ensure2.siemens.cloud

[kleegrewec]

vmExchange.zip

Hi @frmichel thanks for the quick reply. Since I can see log file entries for the missed queries I suggest there is no issue in the network connectivity. Here is the resquested information:

cat /etc/hosts on the distant server

adminuser@dev-blazegraph:~$ cat /etc/hosts
127.0.0.1 localhost dev-blazegraph

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
194.138.20.208 qagpublic.ipinsplus-qa.siemens.com

ifconfig -a on the distant server

adminuser@dev-blazegraph:~$ ifconfig -a
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:ebff:fee0:51d  prefixlen 64  scopeid 0x20<link>
        ether 02:42:eb:e0:05:1d  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 746 (746.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.7.10  netmask 255.255.255.0  broadcast 10.0.7.255
        inet6 fe80::20d:3aff:fedd:7ee1  prefixlen 64  scopeid 0x20<link>
        ether 00:0d:3a:dd:7e:e1  txqueuelen 1000  (Ethernet)
        RX packets 304105  bytes 168335132 (168.3 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 320904  bytes 106821099 (106.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 49612  bytes 7571536 (7.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 49612  bytes 7571536 (7.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth65787a8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::4848:b2ff:fe29:b711  prefixlen 64  scopeid 0x20<link>
        ether 4a:48:b2:29:b7:11  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 47  bytes 3466 (3.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

nslookup from the distant server:

adminuser@dev-blazegraph:~$ netstat -an | grep 8080
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN
tcp6       0      0 :::8080                 :::*                    LISTEN
adminuser@dev-blazegraph:~$ nslookup dev-sag-sms.ensure2.siemens.cloud
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   dev-sag-sms.ensure2.siemens.cloud
Address: 10.0.7.81

Port 80 is also accessible:

adminuser@dev-blazegraph:~$ nmap -Pn -p 80 dev-sag-sms.ensure2.siemens.cloud

Starting Nmap 7.60 ( https://nmap.org ) at 2022-07-29 09:45 CEST
Nmap scan report for dev-sag-sms.ensure2.siemens.cloud (10.0.7.81)
Host is up (0.0013s latency).
rDNS record for 10.0.7.81: dev-sag-sparql-ms.internal.cloudapp.net

PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.07 seconds

The query using the hostname / ip-address does not work from the server where the micro services are installed either:

adminuser@dev-sag-sparql-ms:~/bin$ ip-all-channels.sh
*   Trying 10.0.7.81...
* TCP_NODELAY set
* Connected to dev-sag-sms.ensure2.siemens.cloud (10.0.7.81) port 80 (#0)
> GET /service/siguard-pdp/getAllChannels?query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D HTTP/1.1
> Host: dev-sag-sms.ensure2.siemens.cloud
> User-Agent: curl/7.58.0
> Accept: application/sparql-results+json
>
< HTTP/1.1 404 Not Found
< Date: Fri, 29 Jul 2022 07:46:48 GMT
< Server: Apache/2.4.52 (Debian)
< Content-Length: 295
< Content-Type: text/html; charset=iso-8859-1
<
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.52 (Debian) Server at dev-sag-sms.ensure2.siemens.cloud Port 80</address>
</body></html>
* Connection #0 to host dev-sag-sms.ensure2.siemens.cloud left intact
adminuser@dev-sag-sparql-ms:~/bin$ nslookup dev-sag-sms.ensure2.siemens.cloud
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   dev-sag-sms.ensure2.siemens.cloud
Address: 10.0.7.81

adminuser@dev-sag-sparql-ms:~/bin$

I attached the configuration file for the example service and the sparql-microservice.ini file that I use to the ticket. Thanks again for your help,

Christian

[frmichel]

So, just to make sure I get it, when you are on the machine that hosts the services, the following works:

curl ...  "http://localhost/service/siguard-pdp/getAllChannels?query=${SELECT}"

But the following does not work:

curl ... "http://dev-sag-sms.ensure2.siemens.cloud/service/siguard-pdp/getAllChannels?query=${SELECT}"

Is that right?

I can see in the nslookup trace that dev-sag-sms.ensure2.siemens.cloud resolves to 10.0.7.81, bit the ifconfig says your machine has address 10.0.7.10. So it looks like the name you are using is not the one of the server you're working on. Or am I missing something?

Can you do the opposite: nslookup 10.0.7.10

For later, in the log directory, you have the kgram_server.log, but you should also have a log file specifically from the SPARQL µservices: sms-.log. Can you see it? If not, that may mean the docker container does not have full right access. In that case do a chmod 777 log. I want to see whether the query really gets to the SPARQL µs container, probably not I would say.

[kleegrewec]

Hi @frmichel

Your first assuption regarding curl -.. is correct. The nslookup was executed from the distant host, not from the server that hosts the micro services.

Access to the log directory is working and I can see a sparql-micro-services logfile too but for the failing queries this file does not contain any entry. Nevertheless I get a 404 error and curl connects to port 80 of the server what can be seen in the curl output. So I suppose the query cannot be processed by the sparql-ms container but it somehow gets there.

I found that the apache configuration file for sparql microservices contains a ServerName directive that is

ServerName localhost

I will add a server alias here

ServerAlias dev-sag-sms.ensure2.siemens.cloud

and check whether this helps.

[kleegrewec]

Hi @frmichel

the last one did the tirck. Sorry for bothering you with such apache newbie problems but disuccion with you at least helped me to dig into some details here. Now I need to edit the docker compose file so that an adopted 000-default file containing the ServerAlias directive will be loaded to the container on start.

Thanks a lot for your help and that looked into this,

Christian

[frmichel]

No pb, I'm happy that it worked eventually. ;) Just for my information, unless this is confidential Siemens stuff, can you tell me more about your use case and what you are using SPARQL µs for? (private email may be better). You can find my email address on https://w3id.org/people/franckmichel.