When testing more than 3 browsers, 4th browser reuse a closed tunnel

[vvo]

zuul.yml:

tunnel:
  type: ngrok
browsers:
  - name: safari
    version: 5..latest

- testing: safari @ Mac 10.6: 5
- testing: safari @ Mac 10.8: 6
- testing: safari @ Mac 10.9: 7
- testing: safari @ Mac 10.10: 8
  zuul control server active on port 39339 +0ms
  zuul:sauce:safari:5 running +1ms
  zuul:setup using zuul-ngrok to tunnel +4ms
  zuul:user-server user server: ./test/support-server/index.js +0ms
  zuul:sauce:safari:6 running +1ms
  zuul:setup using zuul-ngrok to tunnel +0ms
  zuul:user-server user server: ./test/support-server/index.js +0ms
  zuul:sauce:safari:7 running +1ms
  zuul:setup using zuul-ngrok to tunnel +0ms
  zuul:user-server user server: ./test/support-server/index.js +0ms
  zuul:user-server user server port 44874 +3ms
  zuul:user-server user server port 60080 +4ms
  zuul:user-server user server port 41516 +4ms
  zuul:setup bouncer active on port 40921 +4ms
  zuul:setup bouncer active on port 55625 +6ms
  zuul:setup bouncer active on port 35919 +7ms
- queuing: <safari 6 on Mac 10.8>
  zuul:sauce:safari:6 queuing +876ms
- queuing: <safari 5 on Mac 10.6>
  zuul:sauce:safari:5 queuing +9ms
- queuing: <safari 7 on Mac 10.9>
  zuul:sauce:safari:7 queuing +5ms
  zuul:sauce:safari:7 open https://e715f7a.ngrok.com/__zuul +7s

[...]

zuul:sauce:safari:8 open https://e715f7a.ngrok.com/__zuul +524ms

So "https://e715f7a.ngrok.com/__zuul" is reused by safari 8 while it was started by safari 7. Am I missing something?

[vvo]

Seems like ngrok gives sometimes the same subdomain name, so when we close it it fails. I have done a test where I do not .disconnect() the tunnel and it works fine.

https://github.com/bubenshchykov/ngrok#disconnect says so too.

But the real problem here is that we should not have the same tunnel domain for different tests. What do you think?

[rase-]

Thanks for the detailed report!

I'll take a closer look a bit later today.

[rase-]

@vvo: Looking into a better way to avoid this issue, but in the meanwhile I merged this to create subdomains locally to avoid reuse: https://github.com/rase-/zuul-ngrok/pull/2, with a special subdomain config value.

[vvo]

Not closing the tunnel and letting ngrok do it seems to be a good way to solve this issue too. My tests are stable now that I did this very small change. And it does not requires an authtoken.

[rase-]

@vvo: Cool. Thanks for experimenting with it. I'll take a look at that, too.

[rase-]

@vvo: yeah seems to work well, but I don't think leaving a whole bunch of tunnels open is ideal either.

Maybe we could offer the subdomain generation and a keepalive option as alternatives. Neither is perfect, but would serve their purpose, I guess.

[vvo]

I understand but forcing users to use authtoken is too much.

it seems that if ngrok reuse the same tunnel, theres no problem at all. there s a problem when we try to close a reused (in use) tunnel.

maybe we are trying to be smarter than ngrok is, there's no problem in not closing a tunnel because ngrok handles it all.

I would go for no closing the tunnel manually, this would be good so that we do not have to register to ngrok.

Also I would like ngrok to be the default tunnel to zuul, it's more stable it seems than localtunnel unfortunately.

It would help get zuul to a pretty stable tool in cloud testing env

[rase-]

I just have doubts about not closing any tunnels manually when launching a ton of browsers, which is why I was suggesting the two approaches as alternatives, meaning we would not be "forcing" users to provide an authtoken.

Using ngrok has it's downsides too, since websockets seem to be totally broken unless using tcp-mode, which in turn requires an authtoken as well.

[rase-]

Having two ways to accomplish essentially the same thing is not ideal either. I think out of the two not closing manually is better, so maybe we just go with that.

[rase-]

@vvo: currently more than one tunnel can be killed at once, even ones that haven't completely finished running the tests (/me facepalm). It shouldn't matter if we're reusing the same subdomain with the latest patch: https://github.com/rase-/zuul-ngrok/commit/c259c46ec4c111e71a671b00240fda2eb65df4ff

Could you try this with your set up as well to verify?

[vvo]

works well @rase- , lets release this! :)

[rase-]

@vvo done!

Thanks! :)