Login via Proxy / VPN

[acegilz]

My server is located on a country where betfair could not be accessed, is there any simple way or some relevant tutorial /gems you could share that could help me fixing this? Like connect via proxy / openvpn, etc (I have a PIA account..) thanks

[mikecmpbll]

the httpi gem which we use supports proxying, just at the moment there isn't a way to pass the options through. I'll release a new version in the next few days to support this.

:+1:

[acegilz]

That would be awesome, I'll be waiting an update. Thanks

[acegilz]

On betfair gem (https://github.com/lukebyrne/betfair) they already have a proxy configuration, you probably already know but just to make sure :)

If you want to use a proxy or turn on Savon's logging then just pass in like so:

This is a local squid proxy I tunnel to from my local machine to access the host server in UK for dev purposes.

proxy = 'http://localhost:8888' logging = true bf = Betfair::API.new(proxy, logging)

Proxies can be useful if you want to host on a cloud service such as Heroku, as you will be denied access to the Betfair API from the USA. Just proxy via a server from a country that Betfair allows, such >as the UK.

[acegilz]

Any update on this?

[mikecmpbll]

been busy. will take a look this week :).

[acegilz]

Thanks! I'm impatiently waiting for it :+1: Don't want to change server just because of this..

[mikecmpbll]

rightio, this should be possible now with 3b28053e4fd426786a3e9526a7151b4ceab481f3. you can pass any options that HTTPI's Request initializer accepts (http://httpirb.com/#options) as the 2nd parameter to Betfair::Client initialization, for example proxy:

client = Betfair::Client.new(
    { "X-Application" => "your application code" },
    { proxy: 'https://my.proxy.address' }
  )

if you could test it by using the latest code from github, then if it's all good, I'll release a new version to rubygems.

[acegilz]

2.1.0 :019 > client.interactive_login("user@mail", "mypass") ArgumentError: Invalid URL: 92.110.173.139

[acegilz]

i cant login anything rather than http:// (ip, https, etc.. dont work). I would like to use PIA socks 5 proxy, do you know if its possible to use a socks5 proxy with this? Also how do I send authentication credentials for proxy on that? (https://www.privateinternetaccess.com/forum/discussion/258/private-internet-access-proxy-now-available-now-open)

Sorry I never used proxy, just VPN.

Thanks

[mikecmpbll]

i don't know anything about PIA socks 5 proxy, tbh, but a quick bit of research shows that the net-http adapter for httpi supports Net::HTTP.SOCKSProxy which sounds like it might be what you're after.

if you pass a proxy uri with the socks scheme, so something like socks://92.110.173.139, then it will initialize this socksproxy, i guess.

[mikecmpbll]

for authentication, we may need to create a pull request for httpi to allow username and password to be passed through to Net::HTTP.SOCKSProxy.

[mikecmpbll]

actually, that patch to support socks proxy in http hasn't been released to rubygems yet. i could probably submit a patch to httpi for the auth stuff and ask them to release it so we can update our dependency here, but i dunno when i'll have the time to look at it. feel free to take a look if you wish.

[acegilz]

Don't worry, ill try to get an http proxy, thanks for your support. I will update here when I get success.

[acegilz]

I am tryng some of the free proxies here http://proxylist.hidemyass.com/search-1297445#listable but seems like I can only authenticate via http which may bring security problems. when i try https i get the following error

[2015-12-06T00:11:13.164867 #5774] DEBUG -- : HTTPI POST request to identitysso.betfair.com (httpclient)
ArgumentError: unsupported proxy https://178.22.148.122:3129

[acegilz]

None of the http proxies worked, dont know if proxy problem or betfair doesnt allow non secure connections (but should be proxy since sometimes i get error 400 bad request and error 500 internal error), but when I force https proxy to run http the request is made and in fact i receive a strange response telling the appkey is wrong (faultstring"=>"ANGX-0007") when its clearly not. If I test it again without the proxy param the response is correct.

RSA 1024 bit CA certificates are loaded due to old openssl compatibility

sports = client.list_event_types(filter: {})

2.1.0 :120 >       sports = client.list_event_types(filter: {})
D, [2015-12-06T00:15:38.505963 #5774] DEBUG -- : HTTPI POST request to api.betfair.com (httpclient)
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
 => {"detail"=>{"exceptionname"=>"APINGException", "APINGException"=>{"errorDetails"=>"", "errorCode"=>"INVALID_APP_KEY", "requestUUID"=>"prdang036-11240513-003a872e31"}}, "faultcode"=>"Client", "faultstring"=>"ANGX-0007"} 

When request is made without proxy params:

2.1.0 :100 > sports = client.list_event_types(filter: {})
D, [2015-12-06T00:14:27.886429 #5774] DEBUG -- : HTTPI POST request to api.betfair.com (httpclient)
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
 => [{"eventType"=>{"id"=>"468328", "name"=>"Handball"}, "marketCount"=>93}, {"eventType"=>{"id"=>"1", "name"=>"Soccer"}, "marketCount"=>13482}, {"eventType"=>{"id"=>"2", "name"=>"Tennis"}, "marketCount"=>19}, {"eventType"=>{"id"=>"3", "name"=>"Golf"}, "marketCount"=>53}, {"eventType"=>{"id"=>"4", "name"=>"Cricket"}, "marketCount"=>169}, {"eventType"=>{"id"=>"998917", "name"=>"Volleyball"}, "marketCount"=>15}, {"eventType"=>{"id"=>"5", "name"=>"Rugby Union"}, "marketCount"=>35}, {"eventType"=>{"id"=>"6", "name"=>"Boxing"}, "marketCount"=>45}, {"eventType"=>{"id"=>"8", "name"=>"Motor Sport"}, "marketCount"=>1}, {"eventType"=>{"id"=>"7524", "name"=>"Ice Hockey"}, "marketCount"=>645}, {"eventType"=>{"id"=>"11", "name"=>"Cycling"}, "marketCount"=>1}, {"eventType"=>{"id"=>"26420387", "name"=>"Mixed Martial Arts"}, "marketCount"=>90}, {"eventType"=>{"id"=>"451485", "name"=>"Winter Sports"}, "marketCount"=>19}, {"eventType"=>{"id"=>"3503", "name"=>"Darts"}, "marketCount"=>34}, {"eventType"=>{"id"=>"136332", "name"=>"Chess"}, "marketCount"=>14}, {"eventType"=>{"id"=>"7522", "name"=>"Basketball"}, "marketCount"=>507}, {"eventType"=>{"id"=>"72382", "name"=>"Pool"}, "marketCount"=>1}, {"eventType"=>{"id"=>"1477", "name"=>"Rugby League"}, "marketCount"=>3}, {"eventType"=>{"id"=>"2152880", "name"=>"Gaelic Games"}, "marketCount"=>4}, {"eventType"=>{"id"=>"6422", "name"=>"Snooker"}, "marketCount"=>26}, {"eventType"=>{"id"=>"6423", "name"=>"American Football"}, "marketCount"=>347}, {"eventType"=>{"id"=>"27388198", "name"=>"Current Affairs"}, "marketCount"=>2}, {"eventType"=>{"id"=>"7511", "name"=>"Baseball"}, "marketCount"=>1}, {"eventType"=>{"id"=>"4339", "name"=>"Greyhound Racing"}, "marketCount"=>269}] 

[mikecmpbll]

the https issue that you're having appears to be related to a bug in the adapter that you're using, httpclient, not allowing https proxy servers for some reason :/. you can switch adapters, to net-http for instance, by doing:

HTTPI.adapter = :net_http

i've tested with a UK proxy from http://www.xroxy.com/proxy-country-GB.htm (https://88.150.158.151:80) and it worked fine for me.

[acegilz]

It's really strange when I request using proxy I always get error on appkey:

2.1.0 :171 >       sports = client.list_event_types(filter: {})
D, [2015-12-06T01:16:52.955375 #5774] DEBUG -- : HTTPI POST request to api.betfair.com (net_http)

 => {"detail"=>{"exceptionname"=>"APINGException", "APINGException"=>{"errorDetails"=>"", "errorCode"=>"INVALID_APP_KEY", "requestUUID"=>"prdang025-11240513-002db72638"}}, "faultcode"=>"Client", "faultstring"=>"ANGX-0007"} 

[mikecmpbll]

maybe you're not initializing the betfair client correctly when you pass the proxy option, and it's interferring with your app key stuff.

[acegilz]

I dont know what do do, already replaced the client file again from my files and rebooted server but the problem persists.

I noticed now that the X-Authentication comes empty when I use the proxy, and that's probably the reason why I don't stop receiving the error I told yesterday:

client.interactive_login("mail@gmail.com", "pass")
D, [2015-12-07T00:32:46.387787 #26958] DEBUG -- : HTTPI POST request to identitysso.betfair.com (net_http)
 => {"X-Application"=>"my_app_token", "Accept"=>"application/json", "Content-Type"=>"application/json", "X-Authentication"=>""} 

[mikecmpbll]

that means that the log-in is failing, it's not returning an authentication token.

how are you initialising the client?

[acegilz]

HTTPI.adapter = :net_http

1

    client = Betfair::Client.new({ "X-Application" => "myappkey" }, { proxy: 'https://128.199.235.21:3128' })

2

    client.interactive_login("mail@gmail.com", "pass")

[mikecmpbll]

not sure what to advise, it works fine for me. the authentication is definitely failing for some reason and it's not an issue with the proxy server because the request is going through fine, we can tell that because you're getting a legitimate response.

[acegilz]

Finally I could get a success response. tried a lot of proxies from the site you gave yesterday and only worked with one: https://217.37.168.132:80

Dunno maybe betfair it only accepts on port 80?

[mikecmpbll]

/shrugs. no idea. you're not likely to have great success with free proxies for anything serious anyway as they usually don't stay up for very long.

[acegilz]

Yeah I know, I would buy something without problem but I need to know if its possible to buy some proxy without user:pw credentials as there's currently no credential supports here (if it were i would be already using my paid socks 5 proxy)

I will be looking on here for updates during the following weeks.

And once again. Thanks for your support !

[mikecmpbll]

i'm pretty sure that for a regular proxy credentials can be passed through by setting the proxy to something like "https://username:password@www.someproxy.org"

as for the socks proxy, it's not out of the realms of possibility, it just needs some small work in httpi by the looks of it.

[acegilz]

Yeah but it seems different for socks authentication. I will keep trying while wait for a reply there https://github.com/astro/socksify-ruby/issues/1#issuecomment-162905681

[mikecmpbll]

okayyyyy, so, putting all the peices of this puzzle together, you should be able to get this working now by:

apply httpi changes to support socks proxy. these changes aren't released to rubygems yet so we cannot have them as a dependency of betfair, so probably the easiest way would be to load a monkey patch:

require "socksify/http"

module HTTPI
  module Adapter
    class NetHTTP
      def create_client
        proxy_url = @request.proxy || URI("")
        if URI(proxy_url).scheme == 'socks'
          proxy = Net::HTTP.SOCKSProxy(proxy_url.host, proxy_url.port)
        else
          proxy = Net::HTTP::Proxy(proxy_url.host, proxy_url.port, proxy_url.user, proxy_url.password)
        end
        proxy.new(@request.url.host, @request.url.port)
      end
    end
  end
end

module HTTPI
  class Request
    def normalize_url!(url)
      raise ArgumentError, "Invalid URL: #{url}" unless url.to_s =~ /^http|socks/
      url.kind_of?(URI) ? url : URI(url)
    end
  end
end

in your app code:

require 'betfair'
require 'httpi_patch' # require the patch above

HTTPI.adapter = :net_http

TCPSocket.socks_username = "username" # socks username
TCPSocket.socks_password = "password" # socks password

client = Betfair::Client.new({ "X-Application" => "myappkey" }, { proxy: 'socks://ipaddress:port' })

hopefully that works!

[acegilz]

Simply amazing. I will try it tonight and confirm you.

Once again, I'm very thankfully for all your support and dedication to solve this issue

[acegilz]

Absolutely brilliant Mike! Works smoothly

[mikecmpbll]

great, no problem. in order to get this working 'out the box':

1) i'll wait for the socksify PR to be merged & released 2) write a PR for httpi to allow the socks credentials to be set 3) update betfair's httpi dependency

[acegilz]

I'll be following this topic and make the appropriate changes when all dependencies get updated

[acegilz]

Is there any update regarding this issue? Every-time I make a new installation I have to come here to check and apply the monkey patch you provided in order to put socks proxy working

Edit: At the moment I can't even do it like I used before:

client = Betfair::Client.new({ "X-Application" => "key" }, { proxy: "socks://#{ip}:1080" })

TypeError: wrong argument type Hash (expected Module)
    from /var/www/bbtc/website/shared/bundle/ruby/2.1.0/gems/betfair-ng-0.2.0/lib/betfair/client.rb:60:in `extend'
    from /var/www/bbtc/website/shared/bundle/ruby/2.1.0/gems/betfair-ng-0.2.0/lib/betfair/client.rb:60:in `extend_api'
    from /var/www/bbtc/website/shared/bundle/ruby/2.1.0/gems/betfair-ng-0.2.0/lib/betfair/client.rb:26:in `initialize'

[acegilz]

Nevermind.. I forgot to update the client.rb like this https://github.com/mikecmpbll/betfair/commit/3b28053e4fd426786a3e9526a7151b4ceab481f3

Hope this could be supported by default soon to avoid having to hardcode 3 different files to have socks proxy working.

[mikecmpbll]

iirc i was waiting for this PR on socksify-ruby https://github.com/astro/socksify-ruby/pull/24 but it doesn't look like anything's happening with it.