tobijk / caius

A functional testing framework in object-oriented Tcl
MIT License
26 stars 5 forks source link

WebDriver failure with latest Selenium Server #20

Open asifhg opened 1 year ago

asifhg commented 1 year ago

I might be missing something in the execution. Any help would be greatly appreciated!

Install: Caius

https://github.com/tobijk/caius.git

Download: Selenium Server

https://github.com/SeleniumHQ/selenium/releases/download/selenium-4.9.0/selenium-server-4.9.1.jar

Execute: Server Start

java -jar selenium-server-4.9.1.jar standalone --host "127.0.0.1" --port 4445 --http-logs true --log-level "FINE" ;

Execute: Client Start

package require WebDriver ;
set seleniumServer "http://127.0.0.1:4445" ;
set caps [namespace which [WebDriver::Capabilities #auto -browser_name firefox]] ;
set session [WebDriver::Session #auto $seleniumServer/wd/hub $caps] ;
# The previous cmd does not return. Several minutes later we get the following,
# and the server is still alive.
::unknownError0

Log: Server

23:25:03.365 INFO [LoggingOptions.configureLogEncoding] - Using the system default encoding
23:25:03.371 INFO [OpenTelemetryTracer.createTracer] - Using OpenTelemetry for tracing
23:25:03.459 DEBUG [AutoConfiguredOpenTelemetrySdkBuilder.build] - Global OpenTelemetry set to OpenTelemetrySdk{tracerProvider=SdkTracerProvider{clock=SystemClock{}, idGenerator=RandomIdGenerator{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.25.0"}}, spanLimitsSupplier=SpanLimitsValue{maxNumberOfAttributes=128, maxNumberOfEvents=128, maxNumberOfLinks=128, maxNumberOfAttributesPerEvent=128, maxNumberOfAttributesPerLink=128, maxAttributeValueLength=2147483647}, sampler=ParentBased{root:AlwaysOnSampler,remoteParentSampled:AlwaysOnSampler,remoteParentNotSampled:AlwaysOffSampler,localParentSampled:AlwaysOnSampler,localParentNotSampled:AlwaysOffSampler}, spanProcessor=SimpleSpanProcessor{spanExporter=org.openqa.selenium.remote.tracing.opentelemetry.SeleniumSpanExporter$1@11c20519}}, meterProvider=SdkMeterProvider{clock=SystemClock{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.25.0"}}, metricReaders=[], views=[]}, loggerProvider=SdkLoggerProvider{clock=SystemClock{}, resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.19.0, attributes={service.name="unknown_service:java", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.25.0"}}, logLimits=LogLimits{maxNumberOfAttributes=128, maxAttributeValueLength=2147483647}, logRecordProcessor=io.opentelemetry.sdk.logs.NoopLogRecordProcessor@701fc37a}, propagators=DefaultContextPropagators{textMapPropagator=MultiTextMapPropagator{textMapPropagators=[W3CTraceContextPropagator, W3CBaggagePropagator]}}} by autoconfiguration
23:25:04.101 INFO [NodeOptions.getSessionFactories] - Detected 4 available processors
23:25:04.115 INFO [NodeOptions.discoverDrivers] - Driver(s) already present on the host: 1
23:25:04.145 INFO [NodeOptions.report] - Adding Firefox for {"browserName": "firefox","platformName": "linux"} 4 times (Host)
23:25:04.166 INFO [Node.<init>] - Binding additional locator mechanisms: relative
23:25:04.203 DEBUG [GridModel.add] - Adding node with id fd1e12ec-cf8a-43c5-8681-4b88a67c1e3e and URI http://127.0.0.1:4445
23:25:04.204 DEBUG [LocalDistributor.updateNodeAvailability] - Health check result for http://127.0.0.1:4445 was UP
23:25:04.205 INFO [GridModel.setAvailability] - Switching Node fd1e12ec-cf8a-43c5-8681-4b88a67c1e3e (uri: http://127.0.0.1:4445) from DOWN to UP
23:25:04.206 INFO [LocalDistributor.add] - Added node fd1e12ec-cf8a-43c5-8681-4b88a67c1e3e at http://127.0.0.1:4445. Health check every 120s
23:25:04.226 DEBUG [MultithreadEventLoopGroup.<clinit>] - -Dio.netty.eventLoopThreads: 8
23:25:04.234 DEBUG [GlobalEventExecutor.<clinit>] - -Dio.netty.globalEventExecutor.quietPeriodSeconds: 1
23:25:04.244 DEBUG [InternalThreadLocalMap.<clinit>] - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
23:25:04.244 DEBUG [InternalThreadLocalMap.<clinit>] - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
23:25:04.266 DEBUG [PlatformDependent0.explicitNoUnsafeCause0] - -Dio.netty.noUnsafe: false
23:25:04.267 DEBUG [PlatformDependent0.javaVersion0] - Java version: 11
23:25:04.267 DEBUG [PlatformDependent0.<clinit>] - sun.misc.Unsafe.theUnsafe: available
23:25:04.268 DEBUG [PlatformDependent0.<clinit>] - sun.misc.Unsafe.copyMemory: available
23:25:04.268 DEBUG [PlatformDependent0.<clinit>] - sun.misc.Unsafe.storeFence: available
23:25:04.268 DEBUG [PlatformDependent0.<clinit>] - java.nio.Buffer.address: available
23:25:04.268 DEBUG [PlatformDependent0.<clinit>] - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
23:25:04.269 DEBUG [PlatformDependent0.<clinit>] - java.nio.Bits.unaligned: available, true
23:25:04.270 DEBUG [PlatformDependent0.<clinit>] - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @c05fddc
23:25:04.271 DEBUG [PlatformDependent0.<clinit>] - java.nio.DirectByteBuffer.<init>(long, int): unavailable
23:25:04.273 DEBUG [PlatformDependent.unsafeUnavailabilityCause0] - sun.misc.Unsafe: available
23:25:04.274 DEBUG [PlatformDependent.estimateMaxDirectMemory] - maxDirectMemory: 4167041024 bytes (maybe)
23:25:04.274 DEBUG [PlatformDependent.tmpdir0] - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
23:25:04.274 DEBUG [PlatformDependent.bitMode0] - -Dio.netty.bitMode: 64 (sun.arch.data.model)
23:25:04.275 DEBUG [PlatformDependent.<clinit>] - -Dio.netty.maxDirectMemory: -1 bytes
23:25:04.275 DEBUG [PlatformDependent.<clinit>] - -Dio.netty.uninitializedArrayAllocationThreshold: -1
23:25:04.276 DEBUG [CleanerJava9.<clinit>] - java.nio.ByteBuffer.cleaner(): available
23:25:04.276 DEBUG [PlatformDependent.<clinit>] - -Dio.netty.noPreferDirect: false
23:25:04.278 DEBUG [NioEventLoop.<clinit>] - -Dio.netty.noKeySetOptimization: false
23:25:04.278 DEBUG [NioEventLoop.<clinit>] - -Dio.netty.selectorAutoRebuildThreshold: 512
23:25:04.283 DEBUG [PlatformDependent$Mpsc.<clinit>] - org.jctools-core.MpscChunkedArrayQueue: available
23:25:04.305 DEBUG [DefaultChannelId.<clinit>] - -Dio.netty.processId: 190894 (auto-detected)
23:25:04.308 DEBUG [NetUtil.<clinit>] - -Djava.net.preferIPv4Stack: false
23:25:04.309 DEBUG [NetUtil.<clinit>] - -Djava.net.preferIPv6Addresses: false
23:25:04.310 DEBUG [NetUtilInitializations.determineLoopback] - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
23:25:04.311 DEBUG [NetUtil$SoMaxConnAction.run] - /proc/sys/net/core/somaxconn: 4096
23:25:04.312 DEBUG [DefaultChannelId.<clinit>] - -Dio.netty.machineId: 94:c6:91:ff:fe:11:df:a9 (auto-detected)
23:25:04.319 DEBUG [ResourceLeakDetector.<clinit>] - -Dio.netty.leakDetection.level: simple
23:25:04.319 DEBUG [ResourceLeakDetector.<clinit>] - -Dio.netty.leakDetection.targetRecords: 4
23:25:04.341 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.numHeapArenas: 8
23:25:04.341 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.numDirectArenas: 8
23:25:04.341 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.pageSize: 8192
23:25:04.341 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.maxOrder: 9
23:25:04.342 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.chunkSize: 4194304
23:25:04.342 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.smallCacheSize: 256
23:25:04.342 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.normalCacheSize: 64
23:25:04.342 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
23:25:04.342 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.cacheTrimInterval: 8192
23:25:04.342 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.cacheTrimIntervalMillis: 0
23:25:04.342 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.useCacheForAllThreads: false
23:25:04.343 DEBUG [PooledByteBufAllocator.<clinit>] - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
23:25:04.350 DEBUG [ByteBufUtil.<clinit>] - -Dio.netty.allocator.type: pooled
23:25:04.350 DEBUG [ByteBufUtil.<clinit>] - -Dio.netty.threadLocalDirectBufferSize: 0
23:25:04.350 DEBUG [ByteBufUtil.<clinit>] - -Dio.netty.maxThreadLocalCharBufferSize: 16384
23:25:04.369 DEBUG [LoggingHandler.channelRegistered] - [id: 0xb3aa8f7e] REGISTERED
23:25:04.373 DEBUG [LoggingHandler.bind] - [id: 0xb3aa8f7e] BIND: /127.0.0.1:4445
23:25:04.376 INFO [Standalone.execute] - Started Selenium Standalone 4.9.1 (revision eb2032df7f): http://127.0.0.1:4445
23:25:04.376 DEBUG [LoggingHandler.channelActive] - [id: 0xb3aa8f7e, L:/127.0.0.1:4445] ACTIVE
################################################################################
# WebDriver::Session
# set session [WebDriver::Session #auto $seleniumServer/wd/hub $caps] ;
################################################################################
23:26:05.668 DEBUG [LoggingHandler.channelRead] - [id: 0xb3aa8f7e, L:/127.0.0.1:4445] READ: [id: 0x1cb980c8, L:/127.0.0.1:4445 - R:/127.0.0.1:33345]
23:26:05.669 DEBUG [LoggingHandler.channelReadComplete] - [id: 0xb3aa8f7e, L:/127.0.0.1:4445] READ COMPLETE
23:26:05.690 DEBUG [AbstractByteBuf.<clinit>] - -Dio.netty.buffer.checkAccessible: true
23:26:05.690 DEBUG [AbstractByteBuf.<clinit>] - -Dio.netty.buffer.checkBounds: true
23:26:05.691 DEBUG [ResourceLeakDetectorFactory$DefaultResourceLeakDetectorFactory.newResourceLeakDetector] - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@57973a03
23:26:05.735 DEBUG [ZlibCodecFactory.<clinit>] - -Dio.netty.noJdkZlibDecoder: false
23:26:05.736 DEBUG [ZlibCodecFactory.<clinit>] - -Dio.netty.noJdkZlibEncoder: false
23:26:05.747 DEBUG [Recycler.<clinit>] - -Dio.netty.recycler.maxCapacityPerThread: 4096
23:26:05.747 DEBUG [Recycler.<clinit>] - -Dio.netty.recycler.ratio: 8
23:26:05.747 DEBUG [Recycler.<clinit>] - -Dio.netty.recycler.chunkSize: 32
23:26:05.747 DEBUG [Recycler.<clinit>] - -Dio.netty.recycler.blocking: false
23:26:05.747 DEBUG [Recycler.<clinit>] - -Dio.netty.recycler.batchFastThreadLocalOnly: true
23:26:05.774 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /wd/hub/session HTTP/1.1
Host: 127.0.0.1:4445
User-Agent: Caius FT Framework
Connection: close
Accept: application/json;charset=utf-8
Accept-Encoding: gzip,deflate,compress
Content-Type: application/json;charset=utf-8
Content-Length: 487
23:26:05.774 DEBUG [RequestConverter.channelRead0] - Start of http request: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
POST /wd/hub/session HTTP/1.1
Host: 127.0.0.1:4445
User-Agent: Caius FT Framework
Connection: close
Accept: application/json;charset=utf-8
Accept-Encoding: gzip,deflate,compress
Content-Type: application/json;charset=utf-8
Content-Length: 487
23:26:05.781 DEBUG [RequestConverter.channelRead0] - Incoming message: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 0, widx: 487, cap: 487/487, unwrapped: PooledUnsafeDirectByteBuf(ridx: 740, widx: 740, cap: 2048)), decoderResult: success)
23:26:05.785 DEBUG [RequestConverter.channelRead0] - End of http request: DefaultLastHttpContent(data: PooledSlicedByteBuf(ridx: 487, widx: 487, cap: 487/487, unwrapped: PooledUnsafeDirectByteBuf(ridx: 740, widx: 740, cap: 2048)), decoderResult: success)
23:26:05.837 DEBUG [SpanWrappedHttpHandler.execute] - Wrapping request. Before OpenTelemetryContext{tracer=io.opentelemetry.sdk.trace.SdkTracer@120d6fe6, context={}, span id=0000000000000000, trace id=00000000000000000000000000000000} and after OpenTelemetryContext{tracer=io.opentelemetry.sdk.trace.SdkTracer@120d6fe6, context={opentelemetry-trace-span-key=SdkSpan{traceId=0a4d56167357df63bfdb9deb447bebb5, spanId=60a7f44bf24dca0c, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=session_queue, kind=INTERNAL, attributes=AttributesMap{data={random.key=c66470e0-6cf5-4a7a-beba-2b71564e43fc}, capacity=128, totalAddedValues=1}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1686032765835000000, endEpochNanos=0}}, span id=60a7f44bf24dca0c, trace id=0a4d56167357df63bfdb9deb447bebb5}
23:26:05.840 DEBUG [HttpTracing.inject] - Injecting (POST) /session into OpenTelemetrySpan{traceId=0a4d56167357df63bfdb9deb447bebb5,spanId=60a7f44bf24dca0c} at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler:84
23:27:03.698 DEBUG [LocalDistributor.lambda$asRunnableHealthCheck$9] - Running healthcheck for Node http://127.0.0.1:4445
23:27:03.699 DEBUG [LocalDistributor.updateNodeAvailability] - Health check result for http://127.0.0.1:4445 was UP
23:29:03.698 DEBUG [LocalDistributor.lambda$asRunnableHealthCheck$9] - Running healthcheck for Node http://127.0.0.1:4445
23:29:03.699 DEBUG [LocalDistributor.updateNodeAvailability] - Health check result for http://127.0.0.1:4445 was UP
asifhg commented 1 year ago

Your tutorial documentation uses: selenium-server-standalone-2.38.0.jar I want to try using WebDriver with selenium-server-standalone-2.38.0.jar. But I'm unable to find this version of the jar file: https://github.com/SeleniumHQ/selenium/tags?after=selenium-2.40.0 I will have to download the src tarball and build it, which I do not want to do.

asifhg commented 1 year ago

I tried each of the Selenium-Server versions backwards and found that version 4.8.3 works. The tcl instruction: set session [WebDriver::Session #auto $seleniumServer/wd/hub $caps] ; starts an instance of the Firefox browser.

tobijk commented 1 year ago

I wonder if this is simply an incompatibility between the browser and the latest selenium server. Are you using a recent version of Firefox or some LTS version maybe?

tobijk commented 1 year ago

Hm, no, it also doesn't work with chromedriver.

tobijk commented 1 year ago

Then probably some backwards-compatibility was removed. This Tcl module was written years ago, even before WebDriver became a W3C standard, so there may be things in there that simply aren't compliant with the standard.

tobijk commented 1 year ago

It's probably about how the session is being requested. I see things have changed. There are no desiredCapabilites and requiredCapabilities anymore. Instead there is alwaysMatch and firstMatch.

asifhg commented 1 year ago

Firefox Version Whatever the latest for Ubuntu 22.04 is: 113.0.2 (64-bit)

Would be useful to get the minimal tcl code to make Selenium-Webdriver start chrome instead of firefox. I tried the following to no avail: java -jar selenium-server/selenium-server-4.8.3.jar standalone --host 127.0.0.1 --port 4445 --http-logs true --log-level INFO --service-configuration stereotype='{"browserName": "chrome"}' ;

But then the same cmd in the following for firefox fails: java -jar selenium-server/selenium-server-4.8.3.jar standalone --host 127.0.0.1 --port 4445 --http-logs true --log-level INFO --service-configuration stereotype='{"browserName": "firefox"}' ;

I am able to do the following successfully.

Server

java -jar selenium-server/selenium-server-4.8.3.jar standalone --host 127.0.0.1 --port 4445 --http-logs true --log-level INFO ;

Client

/usr/bin/tclsh ;
package require WebDriver ;

set caps [ namespace which [ WebDriver::Capabilities #auto -browser_name firefox ]] ;
set session [ WebDriver::Session #auto http://127.0.0.1:4445/wd/hub $caps ] ;
# Now we have a running instance of the firefox browser.

set window [$session active_window] ;
set url "https://duckduckgo.com" ;
$window set_url $url ;
# Now the browser has opened the website.

set elements_name [ $window elements by_xpath {//*[@name]} ] ;
set e_search_field [ lindex $elements_name end-1 ] ;
set e_search_button [ lindex $elements_name end ] ;

$e_search_field click ;
$e_search_field clear ;
$e_search_field send_keys "google" ;
$e_search_button submit ;

I'm no Itcl expert, so debugging this is a challenge. But then again, there's nothing to debug for the above simple experiment. I'm going to keep going further with my experimentations such as loging into a legit a/c and report back and take it further.

asifhg commented 1 year ago

I want to use Caius to be able to launch the Firefox browser with a specific user profile. I tried to look through the source code, but I'm not sure whether passing cmdline args to Firefox is supported. Any help would be much appreciated. To illustrate, here is a simple example in Python3 Selenium. 1) Init user profile in cmdline options passed to Firefox. 2) Launch Firefox. 3) Load URL: about:profiles 4) Exit

OS Platform

lsb_release -a ;
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:    22.04
Codename:   jammy

Installation

sudo apt-get install python3 python3-pip ;
pip3 install selenium webdriver-manager ;
pip3 install -U selenium ;
pip3 install -U webdriver-manager ;

Prequisites

# Create Firefox user profile and note the location of the user profile directory.
firefox -ProfileManager ;

web_user_profile.py

import sys
import time
import os

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.service import Service
from webdriver_manager.firefox import GeckoDriverManager

options = Options()
options.page_load_strategy = 'normal'

profile_arg_param = '-profile'
# This is usually where the Firefox user profile is created.
profile_arg_value = os.getenv('HOME') + '/snap/firefox/common/.mozilla/firefox/WHATEVER_NAME_OF_THE_USER_PROFILE'

options.add_argument(profile_arg_param)
options.add_argument(profile_arg_value)

# Start an instance of the Firefox browser.
browser = webdriver.Firefox(service=Service(GeckoDriverManager().install()), options=options)

# Load URL
url = "about:profiles"
browser.get(url)

# Sleep, close browser and exit.
time.sleep(10) ;# This should give you enough time to look at the user profile used by the browser.
browser.close()
exit()