Closed soosrol closed 2 years ago
whimboo
commented
6 years ago So when you log into the node, can you see that Firefox is getting started at all? I would expect at least the Firefox log to be present in the trace log. I assume you followed https://firefox-source-docs.mozilla.org/testing/geckodriver/geckodriver/TraceLogs.html?
andreastt
commented
6 years ago I am also surprised to not find any stdout output from Firefox after the Running command: … log line. If you see connection attempt going up to 600 and then timing out, it means geckodriver is either unable to launch Firefox or that it is unable to connect to the Marionette remote protocol in the Firefox that gets started.
2mf
commented
6 years ago I'm having the same issue with Selenium 3.7.0 and geckodriver 0.19.1. However in my case firefox is starting and geckodriver can't connect to firefox. When I use the geckodriver 0.18.0 it is working again.
soosrol
commented
6 years ago @whimboo, @andreastt:
Yes, if I log into the node, I can see Firefox starting up, and it's responsive, I can interact with it manually. But somehow Geckodriver can't connect. If I don't try to load the profile, it usually connects on the 2nd-3rd try.
I'll see if I can squeeze out more log detailed logs, but I'm already using {"moz:firefoxOptions": {"log": {"level": "trace"}}}. Is there a separate Firefox log file somewhere? Currently I'm getting these logs from the hub/node's console out.
soosrol
commented
6 years ago @2mf: Firefox starts for me too, but geckodriver can't connect. I'll check if it works with v0.18. Are you also trying to load a profile with command line args, or just starting FF normally?
EDIT: Checked with geckodriver v0.18, no difference.
whimboo
commented
6 years ago @soosrol, does Firefox in those failing cases show the robot in the location bar? Also have a look at the browser console (tools menu) if there are some errors reported for Marionette.
Regarding the logs, usually all output should (for the time being) end up in the same log file. But I don't know anything about the remove driver to give a qualified answer in how it works.
soosrol
commented
6 years ago @whimboo, yes, it has the robot in the address bar. Also checked the browser console, and as far as I can tell, it doesn't have any relevant errors. See attached screenshots.
2mf
commented
6 years ago @soosrol I'm loading a profile in a few cases but most cases do not use profiles at all. What I'm doing is running Firefox headless or within Xvfb inside a docker container.
whimboo
commented
6 years ago @soosrol interesting! It reminds me on a case we occasionally see in our automated tests but which is so seldom that I wasn't able to hit it so far.
Is that reproducible for you all the time? Does it also happen when you are not creating a new profile but let geckodriver do it? I would like to know what the minimal testcase is for you.
Also I would like to see a real trace level log. Would be fantastic if you can find out where this is stored on the hub.
Btw. is that a new issue you are facing with the 57 release, or did it already exist with 56?
soosrol
commented
6 years ago @whimboo I can reproduce it every time. Everything works flawlessly if I don't specify a profile to load. This has been the same ever since I first tried it 2-3 Firefox versions ago and since Selenium 3.4 and geckodriver 17 or so...
What's really strange, is that I only encounter this issue if I pass in the command line arg for loading the profile from local drive. If I generate/load the profile within my tests by using
FirefoxProfile profile = new FirefoxProfile(new File("path/to/profile"));
FirefoxOptions options = new FirefoxOptions();
options.setProfile(profile);
capabilities.setCapability(FirefoxOptions.FIREFOX_OPTIONS, options);the profile gets loaded and and the session is started without an issue.
You may ask why I'm not doing it this way, and the answer is simple: I am running my tests on Selenium grid on multiple threads and this would require me to have selenium transfer the profile upon every browser start/restart. My custom profile contains a browser extension, that is ~20MB in size. Having a profile of this size transferred by selenium grid is not just very slow, but also very fragile, as after 2-3 tries I always get Java heap space exception, regardless of the allowed memory in JVM settings.
whimboo
commented
6 years ago Thank you for the update @soosrol. It looks great to have a reduced testcase now. One thing you could help us with is, can you please try if you can get the same issue reproduced when not running the test on the hub but locally? Does that produce the same failure?
Otherwise, @andreastt do you have an idea? Without having trace logs for that problem, I'm meanwhile out of ideas.
soosrol
commented
6 years ago @whimboo: I've tried running the same comparison (starting with/without the profile arg) on my local machine and I got the exact same results. I think this rules out most of the environmental causes, as my local machine is a Mac, unlike my grid that runs on Windows VMs.
See attached logs for both sessions: Without profile arg: good_ff.log With profile arg: bad_ff.log
Selenium version: 3.7.1 Geckodriver version: 19.1 Firefox version: 57.0 (64-bit)
whimboo
commented
6 years ago Amazing to see. It means we are getting closer! Given those results I took the opportunity and had a look myself. Here the results...
When trying to use an already existent profile the profiles key in moz:firefoxOptions cannot be used. It indeed only supports base64 encoded profile data. But as our documentation says: To have geckodriver pick up an existing profile on the filesystem, please set the args field to {"args": ["-profile", "/path/to/your/profile"]}
So I tried to use args and passed-in the path to a specific profile directory. Firefox launches, by using this profile but we are not able to make any connection to Marionette even it is running. After the connection failure the test quits without killing the Firefox instance. I will continue diving deeper into that...
@soosrol can you please also test the args key. Do you get the correct profile loaded? Or do you have the same connection problems?
whimboo
commented
6 years ago The problem with not being able to make a connection to Marionette for the given profile path is, that by default Marionette uses port 2828, but geckodriver takes a random free port. Right now I do not see a way to allow pass-in a fixed port to geckodriver when not starting it manually. Checking the code I can only see that the command line argument --marionette-port gets evaluated.
So whether we need a capability under moz:firefoxOptions to let users specify a port for Marionette, or geckodriver would have to update the prefs.js file first with the wanted port before starting Firefox. @andreastt what do you think?
soosrol
commented
6 years ago @whimboo: Yes, the args works, that is how I am loading the profile. The correct profile gets loaded both on Windows/Mac, regardless if I do it with the grid or locally.
In my opinion you may have just found the root cause with the marionette port!
When I don't specify the profile and let it be generated for the session, I can see on about:config that the port was set via prefs.js. However, when loading my profile with -profile, it shows as default just like you said.
At this point I'm trying to see if I can somehow force the geckodriver --marionette-port command line parameter from either the code or some config.
whimboo
commented
6 years ago I will file a bug soon so that it can be fixed.
As workaround you could start geckodriver manually by passing in --marionette-port 2828 or whatever you have in the profile setup. Sadly there is no other way possible right now.
soosrol
commented
6 years ago Yes, that worked. I ended up pointing webdriver.gecko.driver to a shell script like this:
#!/bin/bash
path/to/geckodriver "$@" --marionette-port 2828Bug 1421766 is now taking care of it.
Thank you for reporting! We really wonder why no-one else has seen this yet.
whimboo
commented
6 years ago And good to know that there is a workaround at least.
stewart-r
commented
6 years ago Any chance someone could clarify the workaround steps for this? I'm using the selenium standalone-firefox-debug docker container and selenium remote webdriver on .NET
I tried specifying --marionette-port 2828 in options.AddArgument and can see from the container logs that it is being passed to the CLI:
1512734718509 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "--profile /home/seluser/.mozilla/firefox/5vgqsnyv.default/" "--marionette-port 2828"but I still have the same behaviour. I'm not at all sure how to check which port is in the profile setup? :-/ I'm not too familiar with selenium or geckodriver and feel like I am missing the point.
stewart-r
commented
6 years ago Ah i understand now... In case anyone else was struggling in the same way as I was the --marionette-port command line option is applied when starting geckodriver not starting firefox.
So workaround steps are:
Many thanks for the workaround, guys! All back up and running now :-)
phandinhloc
commented
6 years ago I got the same issue. Specifying the marionette port makes it very difficult to write the tests that start multiple firefox profiles at the same time.
usmanpasha
commented
6 years ago FAILED CONFIGURATION: @BeforeClass beforeMethod org.openqa.selenium.WebDriverException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:32703 [localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) Build info: version: '3.7.0', revision: '2321c73', time: '2017-11-02T22:22:35.584Z' System info: host: 'MV-IMAGE-E17-8.local', ip: '2001:f40:902:a86d:4920:6e2e:35df:85c6', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '1.8.0_131' Driver info: driver.version: FirefoxDriver
oneumyvakin
commented
6 years ago For applying workaround on Windows I've used following Go code:
package main
import (
"os"
"os/exec"
"fmt"
)
func main() {
var args []string = os.Args[1:]
args = append(args, "--marionette-port")
args = append(args, "2828")
cmd := exec.Command(
"geckodriver2.exe",
args...,
)
fmt.Println(args)
err := cmd.Run()
if err != nil {
fmt.Printf("failed to execute '%s' '%s': %s\r\n", cmd.Path, cmd.Args, err)
return
}
}Well, the actual workaround for now in case you are using the Python bindings will look like the following:
profile = tempfile.mkdtemp(".selenium")
opts = Options()
opts.binary = binary
opts.add_argument("-profile")
opts.add_argument(profile)
capabilities = DesiredCapabilities.FIREFOX.copy()
driver = webdriver.Firefox(options=opts,
capabilities=capabilities,
service_args=["--marionette-port", "2828"])Other bindings should also expose the given options and the services_args argument.
alkayata
commented
6 years ago I've found same problem with selenium 3.12, firefox 60.1 and geckodriver 0.20.1 The workaround worked fine.
abalfour
commented
6 years ago @stewart-r Thank you for the tips!
Unfortunately I'm still struggling to get this working. Is the shell script being saved as a .sh, or is there a way to convert this to an .exe instead? When I run my program it seems that the Selenium driver is still looking for geckodriver.exe at the original location and isn't invoking the script that I've left in its place. I assume if I were to just leave the original geckodriver.exe in the original location that Selenium would just run that rather than invoking the .sh.
Also, does adding Ooptions.AddArgument($"--marionette-port 2828"); not do anything at all, or is that also necessary for the fix to work?
whimboo
commented
6 years ago @abalfour please see my last comment. It explains it in Python but should be similar for other Selenium bindings.
alkayata
commented
6 years ago @abalfour. It seems you are using windows. You can tell selenium to look for the geckodriver executable like this: "java -Dwebdriver.gecko.driver='path to your geckodriver executable' -jar ./selenium-server-standalone-3.12.0.jar". Don't forget to use the -D flag before the -jar or it doesn't work. Besides, I think if you want to use a shell script in Windows, you need to use a .bat file. So taking that into account, to start the selenium server you should use something like that: "java -Dwebdriver.gecko.driver='path to your geckodriver executable.BAT' -jar ./selenium-server-standalone-3.12.0.jar" Last, you need to adapt your shell script to Windows. The one provided works with bash, a command interpreter for unix, mainly linux. What it does is to spawn a geckodriver instance passing the parameters given to the script to it. For example, if I call 'geckodriver.sh -watermellon', it will in turn call 'geckodriver -watermellon --marionette-port 2828' . The '@' sign is the equivalente for 'all arguments passed to the script'. So you need to find the way to do the same in windows. I don't think it will be hard. I hope it works El jue, 21-06-2018 a las 22:24 +0000, Adrian Balfour escribió:
@stewart-r Thank you for the tips!
Unfortunately I'm still struggling to get this working. Is the shell script being saved as a .sh, or is there a way to convert this to an .exe instead? When I run my program it seems that the Selenium driver is still looking for geckodriver.exe at the original location and isn't invoking the script that I've left in its place. I assume if I were to just leave the original geckodriver.exe in the original location that Selenium would just run that rather than invoking the .sh.
Also, does adding Ooptions.AddArgument($"--marionette-port 2828"); not do anything at all, or is that also necessary for the fix to work?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
{"@context":"http://schema.org","@type":"EmailMessage","potentialActi on":{"@type":"ViewAction","target":"https://github.com/mozilla/geckod river/issues/1058#issuecomment- 399262245","url":"https://github.com/mozilla/geckodriver/issues/1058# issuecomment-399262245","name":"View Issue"},"description":"View this Issue on GitHub","publisher":{"@type":"Organization","name":"GitHub","url":"ht tps://github.com"}} {"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c5 5493e4bb","name":"GitHub"},"entity":{"external_key":"github/mozilla/g eckodriver","title":"mozilla/geckodriver","subtitle":"GitHub repository","main_image_url":"https://assets-cdn.github.com/images/em ail/message_cards/header.png","avatar_image_url":"https://assets- cdn.github.com/images/email/messagecards/avatar.png","action":{"name ":"Open in GitHub","url":"https://github.com/mozilla/geckodriver"}},"updates":{" snippets":[{"icon":"PERSON","message":"@abalfour in #1058: @stewart-r Thank you for the tips! \r\n\r\nUnfortunately I'm still struggling to get this working. Is the shell script being saved as a .sh, or is there a way to convert this to an .exe instead? When I run my program it seems that the Selenium driver is still looking for geckodriver.exe at the original location and isn't invoking the script that I've left in its place. I assume if I were to just leave the original geckodriver.exe in the original location that Selenium would just run that rather than invoking the .sh.\r\n\r\nAlso, does adding
Ooptions.AddArgument($\"--marionette-port 2828\");not do anything at all, or is that also necessary for the fix to work?"}],"action":{"name":"View Issue","url":"https://github.com/mozilla/geckodriver/issues/1058#issu ecomment-399262245"}}} { "@type": "MessageCard", "@context": "http://schema.org/extensions", "hideOriginalBody": "false", "originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB", "title": "Re: [mozilla/geckodriver] Geckodriver can't connect to Marionette if custom profile via local path is in use (#1058)", "sections": [ { "text": "", "activityTitle": "Adrian Balfour", "activityImage": "https://assets-cdn.github.com/images/email/message cards/avatar.png", "activitySubtitle": "@abalfour", "facts": [] } ], "potentialAction": [ { "name": "Add a comment", "@type": "ActionCard", "inputs": [ { "isMultiLine": true, "@type": "TextInput", "id": "IssueComment", "isRequired": false } ], "actions": [ { "name": "Comment", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n\"commandName\": \"IssueComment\",\n\"repositoryFullName\": \"mozilla/geckodriver\",\n\"issueId\": 1058,\n\"IssueComment\": \"{{IssueComment.value}}\"\n}" } ] }, { "name": "Close issue", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n\"commandName\": \"IssueClose\",\n\"repositoryFullName\": \"mozilla/geckodriver\",\n\"issueId\": 1058\n}" }, { "targets": [ { "os": "default", "uri": "https://github.com/mozilla/geckodriver/issues/1058#issuecomme nt-399262245" } ], "@type": "OpenUri", "name": "View on GitHub" }, { "name": "Unsubscribe", "@type": "HttpPOST", "target": "https://api.github.com", "body": "{\n\"commandName\": \"MuteNotification\",\n\"threadId\": 276678876\n}" } ], "themeColor": "26292E" }
hornschorsch
commented
6 years ago Is there any chance that this bug will be corrected some time?
umarch06
commented
6 years ago @alkayata can you please suggest me solution for C#, here is what I am doing but browser doesn't do anything just open and then no action is performed.
FirefoxOptions firefoxOptions = new FirefoxOptions();
firefoxOptions.AddArgument("-profile " + Configuration.BrowserProfilePath);
firefoxOptions.AddArgument("--marionette-port " + "2828");
PropertiesCollection.Driver = new FirefoxDriver(firefoxOptions);@umarch06 so Firefox doesn't have an argument marionette-port. You have to add it to the call of the FirefoxDriver constructor instead. But I don't know how the C# implementation handles that exactly.
umarch06
commented
6 years ago Can you share sample for any other language? I'll highly appreciate it.
whimboo
commented
6 years ago I cannot help here given that I didn't work with C# yet. And as such I don't know the driver at all. But you can check the API docs at https://seleniumhq.github.io/selenium/docs/api/dotnet/.
pmaddi2
commented
6 years ago Even after following work around steps, still able to encounter marionette connection issue.
Geckodriver20 or 18 Firefox 52 Selenium 3.13 or 3.4 Java
With custom or default profile path arguments, always listening port is 2828 and not entering into next step "ie.:- driver.get(URL); and traces are showing these:--- "Marionette INFO Listening on port 2828" "geckodriver::marionette TRACE connection attempt 6/600" "geckodriver::marionette TRACE connection attempt 6/600"
If there is no profile argument provided, Marionette is taking some random available port, and Marionette connection is successful and able to execute further URL navigation step.
Code Used with or without Marionette and with Custom or default profile path arguments is:- //ffoptions.addArguments("--marionette-port 2828"); ffoptions.setAcceptInsecureCerts(true); ffoptions.setLogLevel(FirefoxDriverLogLevel.TRACE); //ffoptions.addArguments("--binary C:\Program Files (x86)\Mozilla Firefox\firefox.exe"); ffoptions.addArguments("-profile"); //ffoptions.addArguments("C:\defaultprofilepath\Optum.default"); ffoptions.addArguments(customProfilePath); driver = new FirefoxDriver(ffoptions); driver.get(URL);
Also Tried with below code able to navigate further steps (i.e.:- App URL, etc..,), but unable to replace/append custom(temp) & default profile cookies with successful authenticated cookies. ffoptions.setAcceptInsecureCerts(true); ffoptions.setLogLevel(FirefoxDriverLogLevel.TRACE); driver = new FirefoxDriver(ffoptions); driver.get(url);
Could anyone please advise me solution for navigating geckodriver with custom profile. Please let me know if you need any further details in-order to provide solution
andreastt
commented
6 years ago Firefox 52 is not supported by geckodriver. Can we please only add comments that are actually relevant to the issue.
pmaddi2
commented
6 years ago Actually I tried with version 62 nightly build, still able to reproduce issue, showing browser connection refused, not going to navigate url step... Sorry correct FF version is 52.8.1 i tested was this.
Could you please advise which least version combinations works for my scenario? So that I’ll get that versions..
whimboo
commented
6 years ago @pmaddi2 as it looks like the Java bindings of Selenium don't allow you to specify a port for Marionette. As least I cannot find anything in the Java API docs. So you will have to wait until this issue has been fixed.
pmaddi2
commented
6 years ago Thank you Henrik,
I'll wait for the fix. In parallel if you get any alternate solution for profile loading. (like extension to override Marionette listening port before driver triggers it),etc. then could you please advise me/share in this thread. It is happening only with Custom profile Version 52.8.1 & 55. And Geckodriver 21 or 20.1 with firefox 62/63 nightly build is starting by taking some random free port(not 2828), but showing connection refused 500 internal server error. I retried by restarting machine and & clearing cache. still no luck.
pmaddi2
commented
6 years ago Thank you @whimboo
I’ll wait
mnemyx
commented
6 years ago @whimboo Thanks for the clarification in Python.
My scenario isn't exactly one-to-one; I'm using a remote Selenium Grid on a Windows Server with the profile on a local machine, not on the node itself. So if it helps anyone with a similar setup:
1.) I followed @soosrol's lead but the batch file ends up as:
C:\bin\geckodriver.exe %* --marionette-port 2828
2.) Node init on the grid:
java -Dwebdriver.gecko.driver=C:\bin\geckodriver.bat -jar C:\selenium\selenium-server-standalone-3.13.0.jar -role node -hub http://localhost:4444/grid/register
3.) And python:
opts = webdriver.FirefoxOptions()
prof = webdriver.FirefoxProfile("/path/to/local/profile")
opts.profile = prof
driver = webdriver.Remote(command_executor="http://remote.hub", options=opts)Thanks again for saving me several hours of frustration!
artemklushin
commented
6 years ago @umarch06 For c# you can try next solution:
FirefoxDriverService ffDriverService = FirefoxDriverService.CreateDefaultService(<driver path>);
ffDriverService.BrowserCommunicationPort = 2828;
IWebDriver driver = new FirefoxDriver(ffDriverService, <instance of FirefoxOptions>);It's works for me with profile in custom directory.
xbxolivesupport
commented
6 years ago follow this code package demo1;
import org.openqa.selenium.WebDriver; import org.openqa.selenium.firefox.FirefoxDriver; import org.openqa.selenium.remote.DesiredCapabilities;
public class Executer2 {
public static void main(String[] args){
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability("acceptInsecureCerts", true);
capabilities.setJavascriptEnabled(true);
capabilities.setCapability("marionette", true);
WebDriver driver = new FirefoxDriver(capabilities);
System.setProperty("webdriver.gecko.driver", System.getProperty("user.dir") + "/geckodriver.exe");
driver.get("http://www.google.com");
driver.close();} any issue regarding browser get help from this
cloudinfra
commented
6 years ago The workaround is awesome. we renamed the original gekodriver-v0.21 to gekodriver_original, and the wapper script to gekodriver-v0.21. the script used gekodriver_original with the --marionette-port 2828. Also, the port 2828 is specified in the user.js file inside the profile you are passing. so if you find a different port number there, you have to accordingly use the correct one. (marionette.port)
This approach avoids specifying webdriver.gecko.driver. Just another approach if you dont want to change the launching of the hub jar.
Deinceps
commented
5 years ago I found that the prefs.js file contains the port on which marionette runs and on that basis I made my decision on c#
FirefoxOptions options = new FirefoxOptions();
options.AddArguments("-profile", profliePath);
var service = FirefoxDriverService.CreateDefaultService();
string port = Regex.Match(
File.ReadAllText(profliePath + "/prefs.js"),
"marionette.port\", (\\d+?)" + Regex.Escape(")"))
.Groups[1]
.Value;
service.BrowserCommunicationPort = int.Parse(port);
var driver = new FirefoxDriver(service, options);
bamBrewing
commented
5 years ago Version: 0.23.0
Platform: Linux
Firefox: 60.3esrI am not able to create a new session using a custom profile.
{
"capabilities":{
"alwaysMatch":{
"moz:firefoxOptions":{
"binary":"/bin/firefox",
"args":[
"-profile",
"/path/to/my/profile"
]
}
}
}
}Running command: "/bin/firefox" "-marionette" "-profile" "/path/to/my/profile" "-foreground" "-no-remote"{
"capabilities":{
"moz:firefoxOptions":{
"binary":"/bin/firefox",
"args":[
"-profile",
"/path/to/my/profile"
]
}
}
}Running command: "/bin/firefox" "-marionette" "-profile" "/tmp/rust_mozprofile.6JBkouREH02c" "-foreground" "-no-remote"
parthibd
commented
5 years ago I want to confirm that this issue still persists with latest version of selenium and geckodriver . Marionette will always listen on port 2828 . Is there a way to temporarily fix this issue ?
hornschorsch
commented
5 years ago I have made a patch that fixes the problem by opening the file prefs.js in the profile directory and replacing the value of marionette.port with the actual portnumber.
https://manitu.koblenz-net.de/~horn/export/geckodriver-0.24.0-horn.patch https://manitu.koblenz-net.de/~horn/export/geckodriver-0.24.0
whimboo
commented
5 years ago @hornschorsch that is great to hear. Would you mind to upload your patch for review so that we can work on it to get it into an official release? The related bug is https://bugzilla.mozilla.org/show_bug.cgi?id=1421766. The process for review can be found at https://firefox-source-docs.mozilla.org/testing/marionette/marionette/NewContributors.html. Thanks a lot!
hornschorsch
commented
5 years ago Hello Henrik,
Am Mo den 11. März 2019 um 02:20 Uhr schrieb Henrik Skupin:
@hornschorsch that is great to hear. Would you mind to upload your patch for review so that we can work on it to get it into an official release? The related bug is https://bugzilla.mozilla.org/show_bug.cgi?id=1421766. The process for review can be found at https://firefox-source-docs.mozilla.org/testing/marionette/marionette/NewContributors.html.
I am very sorry but this procedure is to complicated. I donate this patch to you. Would you mind to review it, possibly rewrite it (i am not experienced in programming in rust and possibly this can be done better or eleganter) and then integrate it into geckodriver if you want?
Thanks a lot!
Kind Regards, Georg
System
Testcase
Expected result
Actual result
Notes
fo.addArguments("-profile", "./firefoxprofile");firefox.exe -profile "path/to/profile"also works.Stacktrace
Trace-level log