Closed hanzy1110 closed 1 year ago
Could you share the full error message returned by the Security Server when you try to query the API? The error message can be found in the /var/log/xroad/proxy.log
file.
Have you tried to establish a connection from the Security Server to the Private Server using telnet:
telnet <PRIVATE_SERVER_IP> 60000
You could also check the service configuration on the Security Server and make sure that the service URL is correct and the verify TLS certificate checkbox is unchecked:
Thanks for the reply! The telnet connection does go through but I can't make requests. Here's the complete proxy.log file after sending a couple requests. It doesn't show any new connections being made and no new errors except some error relating to the OCSP service being unavailable. proxy.log (Edit: The API wasn't running on the private server)
The telnet connection going through means that the firewall configuration is OK. Therefore, the problem is probably related to the way how your doing the request or how the service is configured on the provider side Security Server.
Thanks for sharing the proxy log. According to the log, you haven't sent any requests through the Security Server. Could you share the request details and describe how did you send the request? The request must be sent using the Security Server's REST interface running on port 80
/ 443
(Ubuntu) or 8080
/ 8443
(RHEL).
For example, here's a sample request using the public X-Road Playground environment. You can try out the request by yourself.
curl -X GET -H 'X-Road-Client: PLAYGROUND/COM/1234567-8/TestClient' -i 'http://testcomss01.playground.x-road.systems/r1/PLAYGROUND/GOV/8765432-1/TestService/XRoadStatistics/instances'
Here's some details about the request:
http://testcomss01.playground.x-road.systems
PLAYGROUND/COM/1234567-8/TestClient
PLAYGROUND/GOV/8765432-1/TestService/XRoadStatistics/instances
Hi! This is the curl command I used to make the request and it still fails:
curl -vvv -X GET -H "X-Road-Client:OPTIC/EDU/EDU00001/GP-UNCOMA" -i 'http://rtonqn.ddns.net/r1/OPTIC/EDU/EDU00001/WS-TEST-API'
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying <private server IP>:80...
* connect to <private server IP> port 80 failed: Connection refused
* Failed to connect to rtonqn.ddns.net port 80 after 189 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to rtonqn.ddns.net port 80 after 189 ms: Connection refused
Because of some complications related to server placement I'm using a dynamic dns service using a NoIP service. While troubleshooting with my peers they were able to access the API using another xroad dns. Could my choice of dns be an issue?
Hi! The curl
request seems to be OK. Are you able to send the request if you use the Security Server IP address instead of the DNS name?
curl -vvv -X GET -H "X-Road-Client: OPTIC/EDU/EDU00001/GP-UNCOMA" -i 'http://<SECURITY_SERVER_IP>/r1/OPTIC/EDU/EDU00001/WS-TEST-API'
If the request succeeds with the IP address, then the DNS configuration is causing the issue.
Hi! Still troubleshooting this. The curl command fails even with the IP of the Security server so we can rule out the DNS. The router or ISP also don't seem to be the issue (The private and security server are behind the same router with assigned static IPs. When only fowarding port 80 of the private server everything seems to be OK, problem arises when fowarding port 80 of the security server). When wunning nmap on my dns this is the output:
Starting Nmap 7.80 ( https://nmap.org ) at 2023-05-01 22:11 -03
Nmap scan report for rtonqn.ddns.net (152.169.102.130)
Host is up (0.047s latency).
rDNS record for 152.169.102.130: 130-102-169-152.fibertel.com.ar
Not shown: 991 filtered ports
PORT STATE SERVICE
80/tcp closed http
113/tcp closed ident
443/tcp closed https
2222/tcp open EtherNetIP-1
3000/tcp open ppp
3306/tcp open mysql
4001/tcp closed newoak
5500/tcp open hotline
8080/tcp open http-proxy
Nmap done: 1 IP address (1 host up) scanned in 6.86 seconds
Which states quite clearly the issue, port 80 and 443 are closed and that's why my requests are being rejected. Question is why XROAD is closing those ports since I confirmed there are only xroad services listening on ports 80 and 443.
The output of sudo lsof -i :80
:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 42369 xroad 52u IPv6 797293 0t0 TCP localhost:http (LISTEN)
Which seems to indicate that xroad is listening only on IPv6 addresses which may be the problem?
I've also checked the possibility of multiple firewalls installed and running in the system but this isn't happening.
For reference, the output of iptables --list
:
Chain INPUT (policy DROP)
target prot opt source destination
ufw-before-logging-input all -- anywhere anywhere
ufw-before-input all -- anywhere anywhere
ufw-after-input all -- anywhere anywhere
ufw-after-logging-input all -- anywhere anywhere
ufw-reject-input all -- anywhere anywhere
ufw-track-input all -- anywhere anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ufw-before-logging-forward all -- anywhere anywhere
ufw-before-forward all -- anywhere anywhere
ufw-after-forward all -- anywhere anywhere
ufw-after-logging-forward all -- anywhere anywhere
ufw-reject-forward all -- anywhere anywhere
ufw-track-forward all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ufw-before-logging-output all -- anywhere anywhere
ufw-before-output all -- anywhere anywhere
ufw-after-output all -- anywhere anywhere
ufw-after-logging-output all -- anywhere anywhere
ufw-reject-output all -- anywhere anywhere
ufw-track-output all -- anywhere anywhere
Chain DOCKER (2 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain ufw-after-forward (1 references)
target prot opt source destination
Chain ufw-after-input (1 references)
target prot opt source destination
ufw-skip-to-policy-input udp -- anywhere anywhere udp dpt:netbios-ns
ufw-skip-to-policy-input udp -- anywhere anywhere udp dpt:netbios-dgm
ufw-skip-to-policy-input tcp -- anywhere anywhere tcp dpt:netbios-ssn
ufw-skip-to-policy-input tcp -- anywhere anywhere tcp dpt:microsoft-ds
ufw-skip-to-policy-input udp -- anywhere anywhere udp dpt:bootps
ufw-skip-to-policy-input udp -- anywhere anywhere udp dpt:bootpc
ufw-skip-to-policy-input all -- anywhere anywhere ADDRTYPE match dst-type BROADCAST
Chain ufw-after-logging-forward (1 references)
target prot opt source destination
Chain ufw-after-logging-input (1 references)
target prot opt source destination
LOG all -- anywhere anywhere limit: avg 3/min burst 10 LOG level warning prefix "[UFW BLOCK] "
Chain ufw-after-logging-output (1 references)
target prot opt source destination
Chain ufw-after-output (1 references)
target prot opt source destination
Chain ufw-before-forward (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT icmp -- anywhere anywhere icmp destination-unreachable
ACCEPT icmp -- anywhere anywhere icmp time-exceeded
ACCEPT icmp -- anywhere anywhere icmp parameter-problem
ACCEPT icmp -- anywhere anywhere icmp echo-request
ufw-user-forward all -- anywhere anywhere
Chain ufw-before-input (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ufw-logging-deny all -- anywhere anywhere ctstate INVALID
DROP all -- anywhere anywhere ctstate INVALID
ACCEPT icmp -- anywhere anywhere icmp destination-unreachable
ACCEPT icmp -- anywhere anywhere icmp time-exceeded
ACCEPT icmp -- anywhere anywhere icmp parameter-problem
ACCEPT icmp -- anywhere anywhere icmp echo-request
ACCEPT udp -- anywhere anywhere udp spt:bootps dpt:bootpc
ufw-not-local all -- anywhere anywhere
ACCEPT udp -- anywhere 224.0.0.251 udp dpt:mdns
ACCEPT udp -- anywhere 239.255.255.250 udp dpt:1900
ufw-user-input all -- anywhere anywhere
Chain ufw-before-logging-forward (1 references)
target prot opt source destination
Chain ufw-before-logging-input (1 references)
target prot opt source destination
Chain ufw-before-logging-output (1 references)
target prot opt source destination
Chain ufw-before-output (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ufw-user-output all -- anywhere anywhere
Chain ufw-logging-allow (0 references)
target prot opt source destination
LOG all -- anywhere anywhere limit: avg 3/min burst 10 LOG level warning prefix "[UFW ALLOW] "
Chain ufw-logging-deny (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere ctstate INVALID limit: avg 3/min burst 10
LOG all -- anywhere anywhere limit: avg 3/min burst 10 LOG level warning prefix "[UFW BLOCK] "
Chain ufw-not-local (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
RETURN all -- anywhere anywhere ADDRTYPE match dst-type MULTICAST
RETURN all -- anywhere anywhere ADDRTYPE match dst-type BROADCAST
ufw-logging-deny all -- anywhere anywhere limit: avg 3/min burst 10
DROP all -- anywhere anywhere
Chain ufw-reject-forward (1 references)
target prot opt source destination
Chain ufw-reject-input (1 references)
target prot opt source destination
Chain ufw-reject-output (1 references)
target prot opt source destination
Chain ufw-skip-to-policy-forward (0 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain ufw-skip-to-policy-input (7 references)
target prot opt source destination
DROP all -- anywhere anywhere
Chain ufw-skip-to-policy-output (0 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain ufw-track-forward (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere ctstate NEW
ACCEPT udp -- anywhere anywhere ctstate NEW
Chain ufw-track-input (1 references)
target prot opt source destination
Chain ufw-track-output (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere ctstate NEW
ACCEPT udp -- anywhere anywhere ctstate NEW
Chain ufw-user-forward (1 references)
target prot opt source destination
Chain ufw-user-input (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT udp -- anywhere anywhere udp dpt:22
ACCEPT tcp -- anywhere anywhere tcp dpt:2222
ACCEPT udp -- anywhere anywhere udp dpt:2222
ACCEPT tcp -- anywhere anywhere tcp dpt:4000
ACCEPT udp -- anywhere anywhere udp dpt:4000
ACCEPT tcp -- anywhere anywhere tcp dpt:5577
ACCEPT udp -- anywhere anywhere udp dpt:5577
ACCEPT tcp -- anywhere anywhere tcp dpt:4001
ACCEPT udp -- anywhere anywhere udp dpt:4001
ACCEPT tcp -- anywhere anywhere tcp dpt:5500
ACCEPT udp -- anywhere anywhere udp dpt:5500
ACCEPT tcp -- anywhere anywhere tcp dpt:8000
ACCEPT udp -- anywhere anywhere udp dpt:8000
ACCEPT udp -- anywhere anywhere udp dpt:51825
ACCEPT all -- anywhere anywhere
ACCEPT tcp -- anywhere anywhere tcp dpt:http
ACCEPT udp -- anywhere anywhere udp dpt:80
ACCEPT tcp -- anywhere anywhere tcp dpt:https
Chain ufw-user-limit (0 references)
target prot opt source destination
LOG all -- anywhere anywhere limit: avg 3/min burst 5 LOG level warning prefix "[UFW LIMIT BLOCK] "
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain ufw-user-limit-accept (0 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
Chain ufw-user-logging-forward (0 references)
target prot opt source destination
Chain ufw-user-logging-input (0 references)
target prot opt source destination
Chain ufw-user-logging-output (0 references)
target prot opt source destination
Chain ufw-user-output (1 references)
target prot opt source destination
Could you share the error message returned by the curl command?
On what host you're running the Security Server - Ubuntu, Red Hat or Docker? The default ports vary between different hosts:
80
/ 443
8080
/ 8443
-p <consumer information system port>:8080
-p <consumer information system port>:8443
Also, have you installed the Security Server using the Estonian meta package (xroad-securityserver-ee
)? With that package, client connections to the Security Server are limited to localhost only by default. The proxy.connector-host
host property defines the IP address on which the service client's Security Server listens for connections from client applications. The default value is 0.0.0.0
and it allows listening on all IPv4 interfaces. In the Estonian meta package, the value 127.0.0.1
and it allows listening on localhost only. If you have installed the Estonian meta package, you have to update the property value manually.
Here's the output of the curl command I was using.
curl -vvv -X GET -H "X-Road-Client: OPTIC/EDU/EDU00001/GP-UNCOMA" -i 'http://152.169.102.130/r1/OPTIC/EDU/EDU00001/WS-TEST-API'
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying 152.169.102.130:80...
* connect to 152.169.102.130 port 80 failed: Connection refused
* Failed to connect to 152.169.102.130 port 80 after 175 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 152.169.102.130 port 80 after 175 ms: Connection refused
I'm not using the Estonian meta package it seems since the proxy.connector-host property is set to 0.0.0.0 and the ports are 80 and 443 (I'm running xroad on ubuntu 20.04). I was also tinkering with the ports could changing the default port help? That seems to only produce a timeout in the server. Here's the proxy.ini file proxy.ini.txt
It worked! [https://docs.x-road.global/Manuals/ug-syspar_x-road_v6_system_parameters.html#37-message-log-add-on-parameters-message-log](xroad system parameters) I've edited local.ini adding: [proxy] connector-host=0.0.0.0 server-listen-address=0.0.0.0
Then restarted the proxy and confclient. The proxy process was no longer bound to localhost and I was able to query my test api succesfully. You might want to check why the default config didn't work in the first place!
This is my first time using X-ROAD so this may be a just config error I'm trying to set up an X-ROAD security server to form part of a larger network which I don't manage. I gone through the installation steps that appear in the installation guide, registered a REST endpoint and gave permissions to access it to a registered client. The set up is thus:
When I try to query this test API internally and through the VPN I get the responses from the private server. Problem starts when I try to query through X-ROAD which just returns connection refused. I've already troubleshooted the firewall in both the security and private servers (Output of
sudo ufw status verbose
)Security server:
Private Server:
(Output of
sudo netstat -nlpt
) Security Server:Private Server:
When using the iptables log on port 443 for example I get a log entry for the connection but the firewall doesn't register it and the xroad/proxy.log file doesn't update either.
How can I troubleshoot this?