search

SeleniumHQ / selenium

A browser automation framework and ecosystem.
https://selenium.dev
Apache License 2.0
30.65k stars 8.18k forks source link

[🐛 Bug]: Grid crashes with Apache as reverse Proxy #11031

Closed erik-rnbt closed 1 year ago

erik-rnbt commented 2 years ago

What happened?

I need to connect to our selenium grid over a secure connection (https). I couldnt figure out a way to do it with the functions that selenium delivers so i decided to use Apaches httpd as an reverse proxy.

After a couple of days (maybe a week or less) it becomes unreachable. Even without running test on that grid. Image if i connect with http://[server]:4444/ui image

Image if i connect with https://[server] image

The default log from grid and nodes says nothing. Next i am going to put the log-level to debug.

How can we reproduce the issue?

Grid setup:
podman run -d -p 4442-4444:4442-4444 \
-e "TZ=Europe/Berlin" \
--name selenium-hub [hub-image]

podman run -d -p 5555:5555 \
--shm-size 4g -e "TZ=Europe/Berlin" \
-e SE_EVENT_BUS_HOST=[server] \
-e SE_EVENT_BUS_PUBLISH_PORT=4442 -e SE_EVENT_BUS_SUBSCRIBE_PORT=4443 \
-e SE_NODE_HOST=[server] -e SE_NODE_PORT=5555 \
-e SE_NODE_SESSION_TIMEOUT="300" -e SE_NODE_MAX_SESSIONS=1 \
--name selenium-node1 [node-chrome-image]

For Apache:
{
install httpd, mod_ssl, openssl
create ssl-cert with openssl
}
Run Apache:
systemctl start httpd
nano /etc/httpd/conf.d/default-site.conf
content:{
<VirtualHost *:443>
ServerName [servername]
SSLEngine On
SSLCertificateFile [pathToFile].crt
SSLCertificateKeyFile [pathToFile].key
ProxyRequests Off
ProxyPreserveHost On
ProxyPass / http://[servername]:4444/
ProxyPassReverse / http://[servername]:4444/
</VirtualHost>

<VirtualHost *:80>
    ProxyPreserveHost On
    ProxyPass / http://[servername]:4444/
    ProxyPassReverse / http://[servername]:4444/
</VirtualHost>

}

systemctl restart httpd

Relevant log output

2022-09-09 14:11:15,466 INFO Included extra file "/etc/supervisor/conf.d/selenium-grid-hub.conf" during parsing
2022-09-09 14:11:15,469 INFO RPC interface 'supervisor' initialized
2022-09-09 14:11:15,470 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-09-09 14:11:15,470 INFO supervisord started with pid 8
2022-09-09 14:11:16,473 INFO spawned: 'selenium-grid-hub' with pid 10
14:11:16.809 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
14:11:16.813 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
2022-09-09 14:11:16,814 INFO success: selenium-grid-hub entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
14:11:16.904 INFO [BoundZmqEventBus.<init>] - XPUB binding to [binding to tcp://*:4442, advertising as tcp:// [ip]:4442], XSUB binding to [binding to tcp://*:4443, advertising as tcp://[ip]:4443]
14:11:16.955 INFO [UnboundZmqEventBus.<init>] - Connecting to tcp:// [ip]:4442 and tcp:// [ip]:4443
14:11:16.981 INFO [UnboundZmqEventBus.<init>] - Sockets created
14:11:17.983 INFO [UnboundZmqEventBus.<init>] - Event bus ready
14:11:18.641 INFO [Hub.execute] - Started Selenium Hub 4.4.0 (revision e5c75ed026a): http:// [ip]:4444
14:11:19.401 INFO [Node.<init>] - Binding additional locator mechanisms: name, relative, id
14:11:20.069 INFO [GridModel.setAvailability] - Switching Node 527e5556-6102-4f76-818b-dcd27454df18 (uri: http:// [servername]:5555) from DOWN to UP
14:11:20.069 INFO [LocalDistributor.add] - Added node 527e5556-6102-4f76-818b-dcd27454df18 at http:// [servername]:5555. Health check every 120s
14:11:20.073 INFO [Node.<init>] - Binding additional locator mechanisms: relative, id, name
14:11:20.207 INFO [GridModel.setAvailability] - Switching Node b73d743e-8c3a-4ea5-baad-926a89bc7a4b (uri: http:// [servername]:5556) from DOWN to UP
14:11:20.208 INFO [LocalDistributor.add] - Added node b73d743e-8c3a-4ea5-baad-926a89bc7a4b at http:// [servername]:5556. Health check every 120s

Operating System

RHEL 8

Selenium version

4.4.0

What are the browser(s) and version(s) where you see this issue?

Chrome

What are the browser driver(s) and version(s) where you see this issue?

105

Are you using Selenium Grid?

4.4.0

github-actions[bot] commented 2 years ago

@ehesognp, thank you for creating this issue. We will troubleshoot it as soon as we can.

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

erik-rnbt commented 2 years ago

This is what Log-Level finest says when you enter the ui and it shows "Loading"

10:04:50.068 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui HTTP/1.1
Host: [servername]
Cache-Control: max-age=0
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
DNT: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.068 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui HTTP/1.1
Host: [servername]
Cache-Control: max-age=0
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
DNT: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.068 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
10:04:50.068 DEBUG [RequestConverter.channelRead0] - Closing input pipe.
10:04:50.086 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/static/js/main.009ba7cb.js HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Accept: */*
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: script
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.086 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/static/js/main.009ba7cb.js HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Accept: */*
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: script
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.086 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
10:04:50.086 DEBUG [RequestConverter.channelRead0] - Closing input pipe.
10:04:50.099 DEBUG [LoggingHandler.channelRead] - [id: 0x4bcbf4cc, L:/0:0:0:0:0:0:0:0:4444] READ: [id: 0x7891d9c5, L:/[ip]:4444 - R:/[ip]:37334]
10:04:50.099 DEBUG [LoggingHandler.channelReadComplete] - [id: 0x4bcbf4cc, L:/0:0:0:0:0:0:0:0:4444] READ COMPLETE
10:04:50.099 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/static/css/main.ce2bf04a.css HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Accept: text/css,*/*;q=0.1
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: style
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.099 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/static/css/main.ce2bf04a.css HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Accept: text/css,*/*;q=0.1
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: style
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.099 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
10:04:50.099 DEBUG [RequestConverter.channelRead0] - Closing input pipe.
10:04:50.179 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/static/media/selenium-grid-logo.86cdb73d38521c0e49691666db832e49.svg HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Accept: image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: image
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.179 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/static/media/selenium-grid-logo.86cdb73d38521c0e49691666db832e49.svg HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Accept: image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: image
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.179 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
10:04:50.179 DEBUG [RequestConverter.channelRead0] - Closing input pipe.
10:04:50.204 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/favicon.svg HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Accept: image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: image
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.204 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/favicon.svg HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
DNT: 1
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Accept: image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: no-cors
Sec-Fetch-Dest: image
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.204 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
10:04:50.204 DEBUG [RequestConverter.channelRead0] - Closing input pipe.
10:04:50.214 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /graphql HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
accept: */*
DNT: 1
content-type: application/json
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Origin: https://[servername]
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
content-length: 361
10:04:50.214 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /graphql HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
accept: */*
DNT: 1
content-type: application/json
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Origin: https://[servername]
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
content-length: 361
10:04:50.214 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 281, cap: 281/281, unwrapped: PooledUnsafeDirectByteBuf(ridx: 1024, widx: 1024, cap: 1024)), decoderResult: success)
10:04:50.214 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 80, cap: 80/80, unwrapped: PooledUnsafeDirectByteBuf(ridx: 80, widx: 80, cap: 16384)), decoderResult: success)
10:04:50.214 DEBUG [RequestConverter.channelRead0] - Closing input pipe.
10:04:50.215 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /graphql HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
accept: */*
DNT: 1
content-type: application/json
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Origin: https://[servername]
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
content-length: 211
10:04:50.215 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /graphql HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
accept: */*
DNT: 1
content-type: application/json
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Origin: https://[servername]
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
content-length: 211
10:04:50.216 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 211, cap: 211/211, unwrapped: PooledUnsafeDirectByteBuf(ridx: 954, widx: 954, cap: 1024)), decoderResult: success)
10:04:50.216 DEBUG [RequestConverter.channelRead0] - Closing input pipe.
10:04:50.235 DEBUG [LoggingHandler.channelRead] - [id: 0x4bcbf4cc, L:/0:0:0:0:0:0:0:0:4444] READ: [id: 0xc5394679, L:/[ip]:4444 - R:/[ip]:37344]
10:04:50.236 DEBUG [LoggingHandler.channelReadComplete] - [id: 0x4bcbf4cc, L:/0:0:0:0:0:0:0:0:4444] READ COMPLETE
10:04:50.236 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/static/media/encode-sans-v7-latin-regular.0665411e45d6cb8e7c5d.woff2 HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
Origin: https://[servername]
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
DNT: 1
sec-ch-ua-platform: "Windows"
Accept: */*
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: font
Referer: https://[servername]/ui/static/css/main.ce2bf04a.css
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.236 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
GET /ui/static/media/encode-sans-v7-latin-regular.0665411e45d6cb8e7c5d.woff2 HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
Origin: https://[servername]
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
DNT: 1
sec-ch-ua-platform: "Windows"
Accept: */*
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: font
Referer: https://[servername]/ui/static/css/main.ce2bf04a.css
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
10:04:50.236 DEBUG [RequestConverter.channelRead0] - Incoming message: EmptyLastHttpContent
10:04:50.236 DEBUG [RequestConverter.channelRead0] - Closing input pipe.

This is what Log-Level finest shows when it switches to "error while loading" after a couple of seconds

10:10:47.546 DEBUG [RequestConverter.channelInactive] - Closing input pipe, channel became inactive.
10:10:47.547 DEBUG [RequestConverter.channelInactive] - Closing input pipe, channel became inactive.
10:10:47.547 DEBUG [SeleniumSpanExporter$1.lambda$export$4] - SpanData{spanContext=ImmutableSpanContext{traceId=e301f0c3a9d3d31eafb7c52e245c947e, spanId=3778184c957e4b1e, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.9.0, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.16.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=default, version=null, schemaUrl=null}, name=grid.status, kind=INTERNAL, startEpochNanos=1663920587503000000, endEpochNanos=1663920647546514233, attributes={}, totalAttributeCount=0, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}
10:10:47.547 DEBUG [SeleniumSpanExporter$1.lambda$export$4] - SpanData{spanContext=ImmutableSpanContext{traceId=621b92602b7d7b5f209bd39d8bbe10ff, spanId=efc40abcd3f37ca2, traceFlags=01, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=true}, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.9.0, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.16.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=default, version=null, schemaUrl=null}, name=grid.status, kind=INTERNAL, startEpochNanos=1663920587505000000, endEpochNanos=1663920647546849341, attributes={}, totalAttributeCount=0, events=[], totalRecordedEvents=0, links=[], totalRecordedLinks=0, status=ImmutableStatusData{statusCode=UNSET, description=}, hasEnded=true}
10:10:52.561 DEBUG [LoggingHandler.channelRead] - [id: 0x4bcbf4cc, L:/0:0:0:0:0:0:0:0:4444] READ: [id: 0x45cef7dc, L:/[ip]:4444 - R:/[ip]:56556]
10:10:52.562 DEBUG [LoggingHandler.channelReadComplete] - [id: 0x4bcbf4cc, L:/0:0:0:0:0:0:0:0:4444] READ COMPLETE
10:10:52.562 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /graphql HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
accept: */*
DNT: 1
content-type: application/json
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Origin: https://[servername]
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
content-length: 361
10:10:52.562 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /graphql HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
accept: */*
DNT: 1
content-type: application/json
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Origin: https://[servername]
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
content-length: 361
10:10:52.562 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 361, cap: 361/361, unwrapped: PooledUnsafeDirectByteBuf(ridx: 1104, widx: 1104, cap: 2048)), decoderResult: success)
10:10:52.562 DEBUG [RequestConverter.channelRead0] - Closing input pipe.
10:10:52.563 DEBUG [LoggingHandler.channelRead] - [id: 0x4bcbf4cc, L:/0:0:0:0:0:0:0:0:4444] READ: [id: 0x45e1ec87, L:/[ip]:4444 - R:/[ip]:56572]
10:10:52.563 DEBUG [LoggingHandler.channelReadComplete] - [id: 0x4bcbf4cc, L:/0:0:0:0:0:0:0:0:4444] READ COMPLETE
10:10:52.563 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /graphql HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
accept: */*
DNT: 1
content-type: application/json
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Origin: https://[servername]
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
content-length: 211
10:10:52.563 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /graphql HTTP/1.1
Host: [servername]
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Microsoft Edge";v="104"
accept: */*
DNT: 1
content-type: application/json
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.81 Safari/537.36 Edg/104.0.1293.54
sec-ch-ua-platform: "Windows"
Origin: https://[servername]
Sec-Fetch-Site: same-origin
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://[servername]/ui
Accept-Encoding: gzip, deflate, br
Accept-Language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
X-Forwarded-For: [ip]
X-Forwarded-Host: [servername]
X-Forwarded-Server: [servername]
Connection: Keep-Alive
content-length: 211
10:10:52.563 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 211, cap: 211/211, unwrapped: PooledUnsafeDirectByteBuf(ridx: 954, widx: 954, cap: 2048)), decoderResult: success)
10:10:52.563 DEBUG [RequestConverter.channelRead0] - Closing input pipe.

The whole time the status of the grid is ready. But you cant connect with a node.

pujagani commented 2 years ago

Thank you for sharing the details. "I need to connect to our selenium grid over a secure connection (https). I couldnt figure out a way to do it with the functions that selenium delivers so i decided to use Apaches httpd as an reverse proxy." - Have you tried running the commands https://www.selenium.dev/documentation/grid/configuration/help/#security and using the instructions to set up a secure grid? The instructions are also available here https://www.selenium.dev/documentation/grid/configuration/help/#security. It might help get rid of the dependency on Apache for reverse proxy.
After that, if the issue still persists, we can help triage it. We will need a way to reproduce the issue though.

erik-rnbt commented 2 years ago

I tried setting up a secure grid to get rid of the dependency on Apache, but I'm getting an error:

2022-11-02 10:36:49,129 INFO Included extra file "/etc/supervisor/conf.d/selenium-grid-hub.conf" during parsing
2022-11-02 10:36:49,133 INFO RPC interface 'supervisor' initialized
2022-11-02 10:36:49,133 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-11-02 10:36:49,133 INFO supervisord started with pid 8
2022-11-02 10:36:50,136 INFO spawned: 'selenium-grid-hub' with pid 10
Appending Selenium options: --https-private-key /root/key.pem --https-certificate /root/cert.cer
2022-11-02 10:36:50,143 INFO success: selenium-grid-hub entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
10:36:50.497 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
10:36:50.502 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.openqa.selenium.grid.Bootstrap.runMain(Bootstrap.java:77)
        at org.openqa.selenium.grid.Bootstrap.main(Bootstrap.java:70)
Caused by: org.openqa.selenium.grid.config.ConfigException: java.lang.reflect.InvocationTargetException
        at org.openqa.selenium.grid.config.MemoizedConfig.getClass(MemoizedConfig.java:115)
        at org.openqa.selenium.grid.server.EventBusOptions.createBus(EventBusOptions.java:51)
        at org.openqa.selenium.grid.server.EventBusOptions.getEventBus(EventBusOptions.java:41)
        at org.openqa.selenium.grid.commands.Hub.createHandlers(Hub.java:123)
        at org.openqa.selenium.grid.TemplateGridServerCommand.asServer(TemplateGridServerCommand.java:41)
        at org.openqa.selenium.grid.commands.Hub.execute(Hub.java:220)
        at org.openqa.selenium.grid.TemplateGridCommand.lambda$configure$4(TemplateGridCommand.java:129)
        at org.openqa.selenium.grid.Main.launch(Main.java:83)
        at org.openqa.selenium.grid.Main.go(Main.java:57)
        at org.openqa.selenium.grid.Main.main(Main.java:42)
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.openqa.selenium.grid.config.ClassCreation.callCreateMethod(ClassCreation.java:50)
        at org.openqa.selenium.grid.config.MemoizedConfig.lambda$getClass$4(MemoizedConfig.java:100)
        at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
        at org.openqa.selenium.grid.config.MemoizedConfig.getClass(MemoizedConfig.java:95)
        ... 15 more
Caused by: org.openqa.selenium.grid.config.ConfigException: Cannot read the certificate file: /root/cert.cer
        at org.openqa.selenium.grid.security.SecretOptions.getRegistrationSecret(SecretOptions.java:53)
        at org.openqa.selenium.events.zeromq.ZeroMqEventBus.create(ZeroMqEventBus.java:79)
        ... 23 more
2022-11-02 10:36:50,628 INFO exited: selenium-grid-hub (exit status 1; not expected)

For running the Hub i use:

docker run -p 4442-4444:4442-4444 \
-e SE_OPTS="--https-certificate /root/cert.pem --https-private-key /root/key.pem" \
--name selenium-hub selenium/hub:latest

I tried different selfsigned certificates (.pem, .crt, .cer) which i created with openssl openssl req -x509 -newkey rsa:4096 -nodes -keyout key.pem -out cert.pem -sha256 -days 365

diemol commented 2 years ago

You need to mount the file to the container, otherwise it won't be able to read it.

erik-rnbt commented 2 years ago

Thanks for the info. I tried mounting it to the container with:

docker run -p 4442-4444:4442-4444 \
-v /root/cert.cer:/cert.cer \
-v /root/key.pem:/key.pem \
-e SE_OPTS="--https-certificate cert.cer --https-private-key key.pem" \
--name selenium-hub selenium/hub:latest

I checked that the files are in the container. But I still get the same error.

Do you need a specific cert/key file?

diemol commented 2 years ago

You need to put the whole path here --https-certificate cert.cer --https-private-key key.pem

erik-rnbt commented 2 years ago

You mean change this line -e SE_OPTS="--https-certificate cert.cer --https-private-key key.pem" \ to -e SE_OPTS="--https-certificate /cert.cer --https-private-key /key.pem" \ ?

This also wont work. But thats where the files are, arent they?

diemol commented 2 years ago

What is the error? Have you checked inside the container if the files are actually there?

erik-rnbt commented 2 years ago

This is the error

java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.openqa.selenium.grid.Bootstrap.runMain(Bootstrap.java:77)
        at org.openqa.selenium.grid.Bootstrap.main(Bootstrap.java:70)
Caused by: org.openqa.selenium.grid.config.ConfigException: java.lang.reflect.InvocationTargetException
        at org.openqa.selenium.grid.config.MemoizedConfig.getClass(MemoizedConfig.java:115)
        at org.openqa.selenium.grid.server.EventBusOptions.createBus(EventBusOptions.java:51)
        at org.openqa.selenium.grid.server.EventBusOptions.getEventBus(EventBusOptions.java:41)
        at org.openqa.selenium.grid.commands.Hub.createHandlers(Hub.java:123)
        at org.openqa.selenium.grid.TemplateGridServerCommand.asServer(TemplateGridServerCommand.java:41)
        at org.openqa.selenium.grid.commands.Hub.execute(Hub.java:220)
        at org.openqa.selenium.grid.TemplateGridCommand.lambda$configure$4(TemplateGridCommand.java:129)
        at org.openqa.selenium.grid.Main.launch(Main.java:83)
        at org.openqa.selenium.grid.Main.go(Main.java:57)
        at org.openqa.selenium.grid.Main.main(Main.java:42)
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.openqa.selenium.grid.config.ClassCreation.callCreateMethod(ClassCreation.java:50)
        at org.openqa.selenium.grid.config.MemoizedConfig.lambda$getClass$4(MemoizedConfig.java:100)
        at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
        at org.openqa.selenium.grid.config.MemoizedConfig.getClass(MemoizedConfig.java:95)
        ... 15 more
Caused by: org.openqa.selenium.grid.config.ConfigException: Cannot read the certificate file: /cert.cer
        at org.openqa.selenium.grid.security.SecretOptions.getRegistrationSecret(SecretOptions.java:53)
        at org.openqa.selenium.events.zeromq.ZeroMqEventBus.create(ZeroMqEventBus.java:79)
        ... 23 more

The files are there. I looked into the container with: docker exec fd1b89292c5f ls

diemol commented 2 years ago

Maybe the files are corrupted or do not work. Why don't you try them locally first and then in the image?

erik-rnbt commented 1 year ago

The files were currupted. I created a new cert and key file with:

openssl req -newkey rsa:2048 -nodes -keyout key.key -out cert.csr
openssl x509 -signkey key.key -in cert.csr -req -days 365 -out cert.crt

I tested both and they work now. But i still get the same error if i mount these files to the container. I even tried --mount instead of -v.

pujagani commented 1 year ago

Thank you @ehesognp for trying it out locally. If you were successful in setting up locally, does the Grid still crash? Did you get a chance to see if something seems off with the Grid? Regarding using it with docker, @diemol is the best person to help :)

erik-rnbt commented 1 year ago

The hub still shows me the error when i try to start it with the cert and key. I just made sure i have a valid certificate and key. They both work, but i cant use them for the grid.

I mounted them to the container with -v and --mount. With both ways they are inside the container but the Hub shows me Cannot read the certificate file: /cert.cer

Im sure @diemol can help, like he did in the past with some other problems

krmahadevan commented 1 year ago

@diemol @pujagani

The instructions generated when running 

java -jar selenium-server-<version>.jar info security

doesn't seem to help when running in docker mode.

Here's what I have done so far.

  1. Installed minica by running brew install minica
  2. Ran minica --domains selenium-hub
  3. Ran the key conversion using openssl pkcs8 -in selenium-hub/key.pem -topk8 -out selenium-hub/key.pkcs8 -nocrypt
  4. Copied $JAVA_HOME/lib/security/cacerts to .
  5. Imported the minica pem file into trust store using keytool --import -file minica.pem -alias minica -keystore ./cacerts -storepass changeit
  6. Created a docker-compose file that looks like below
version: "3.8"
services:
  chrome:
    image: selenium/node-chrome:4.6.0-20221104
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_OPTS=--registration-secret cheese --https-certificate /home/seluser/cert.pem --https-private-key /home/seluser/key.pkcs8
      - SE_JAVA_OPTS=-Djavax.net.debug=ssl:handshake -Djavax.net.ssl.trustStore=/home/seluser/cacerts -Djavax.net.ssl.trustStorePassword=changeit
    volumes:
      - ./selenium-hub/cert.pem:/home/seluser/cert.pem:ro
      - ./selenium-hub/key.pkcs8:/home/seluser/key.pkcs8:ro
      - ./cacerts:/home/seluser/cacerts:ro
  selenium-hub:
    image: selenium/hub:4.6.0-20221104
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    environment:
      - SE_OPTS=--registration-secret cheese --https-certificate /home/seluser/cert.pem --https-private-key /home/seluser/key.pkcs8
      - SE_JAVA_OPTS=-Djavax.net.debug=ssl:handshake -Djavax.net.ssl.trustStore=/home/seluser/cacerts -Djavax.net.ssl.trustStorePassword=changeit
    volumes:
      - ./selenium-hub/cert.pem:/home/seluser/cert.pem:ro
      - ./selenium-hub/key.pkcs8:/home/seluser/key.pkcs8:ro
      - ./cacerts:/home/seluser/cacerts:ro
selenium-hub   | javax.net.ssl|ERROR|1C|HttpClient-1-Worker-0|2022-11-13 08:39:42.468 UTC|TransportContext.java:345|Fatal (CERTIFICATE_UNKNOWN): No subject alternative names matching IP address 192.168.16.3 found (
selenium-hub   | "throwable" : {
selenium-hub   |   java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.16.3 found
selenium-hub   |        at java.base/sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165)
selenium-hub   |        at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:101)
selenium-hub   |        at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
selenium-hub   |        at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:429)
selenium-hub   |        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283)
selenium-hub   |        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
selenium-hub   |        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
selenium-hub   |        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
selenium-hub   |        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
selenium-hub   |        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
selenium-hub   |        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
selenium-hub   |        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
selenium-hub   |        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
selenium-hub   |        at java.base/java.security.AccessController.doPrivileged(Native Method)
selenium-hub   |        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
selenium-hub   |        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)

This causes the new node to fail and the registration keeps running in a loop. Any pointers around this ?

The issue definitely seems to be around how the certificate is generated and added to the Node from within the docker environment.

krmahadevan commented 1 year ago

@ehesognp

I mounted them to the container with -v and --mount. With both ways they are inside the container but the Hub shows me Cannot read the certificate file: /cert.cer

This is most probably due to permission issues. Please try adding more lenient permissions (777 worked for me) so that the seluser account which runs the JVM inside the docker container is able to access the cert and key available in the physical host via the mount.

diemol commented 1 year ago

Thanks for troubleshooting, @krmahadevan!

erik-rnbt commented 1 year ago

Thanks @krmahadevan

Changing the permission and mounting them with :z to the container worked.

The hub is running. But when trying to add a node it shows this:

javax.net.ssl|ERROR|21|nioEventLoopGroup-3-5|2022-11-21 10:19:29.974 CET|TransportContext.java:345|Fatal (INTERNAL_ERROR): closing inbound before receiving peer's close_notify (
"throwable" : {
  javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:340)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:296)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
        at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
        at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1868)
        at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1068)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)}

Do I need to add the certificate to a truststore?

diemol commented 1 year ago

Yes, the easiest is to build your own docker image with the certs in it.

Thanks for troubleshooting, @krmahadevan!

Closing this as a solution was found.

erik-rnbt commented 1 year ago

@krmahadevan

  • Now when running docker-compose up I noticed the below exception at the grid side
selenium-hub   | javax.net.ssl|ERROR|1C|HttpClient-1-Worker-0|2022-11-13 08:39:42.468 UTC|TransportContext.java:345|Fatal (CERTIFICATE_UNKNOWN): No subject alternative names matching IP address 192.168.16.3 found (
selenium-hub   | "throwable" : {
selenium-hub   |   java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.16.3 found
selenium-hub   |        at java.base/sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165)
selenium-hub   |        at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:101)
selenium-hub   |        at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
selenium-hub   |        at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:429)
selenium-hub   |        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283)
selenium-hub   |        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
selenium-hub   |        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
selenium-hub   |        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
selenium-hub   |        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
selenium-hub   |        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
selenium-hub   |        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
selenium-hub   |        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
selenium-hub   |        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
selenium-hub   |        at java.base/java.security.AccessController.doPrivileged(Native Method)
selenium-hub   |        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
selenium-hub   |        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)

This causes the new node to fail and the registration keeps running in a loop. Any pointers around this ?

The issue definitely seems to be around how the certificate is generated and added to the Node from within the docker environment.

I had them same issue when using a self signed certificate. After switching to a cert signed by my ca it worked.

krmahadevan commented 1 year ago

I had them same issue when using a self signed certificate. After switching to a cert signed by my ca it worked.

@ehesognp - Would it be possible for you to please list the steps on how to get the cert signed along with all the steps to generate a cert. Maybe we could add it to the user facing documentation so that, others can benefit easily. I can help document it. But I dont know what the steps are. If you can share them, I can try to document it.

erik-rnbt commented 1 year ago

At first you have to add a SAN to your csr and cert by using your own openssl.cnf. For example openssl.cnf:

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
...

[v3_req]
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
IP.1 = [your server ip (if you dont have a DNS)]
DNS.1 = [your server name (if you have one)]

And create your csr and cert with -config openssl.cnf -extensions v3_req

If you use a DNS in the config make sure to put -e SE_NODE_HOST=[servername] when setting up a node, else it will throw your exception again

github-actions[bot] commented 1 year ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.