search

wspek / uboat

0 stars 0 forks source link

Implement user login due to OpenSubtitles rate limiting #46

Closed wspek closed 8 months ago

wspek commented 5 years ago

https://forum.opensubtitles.org/viewtopic.php?f=8&t=16072

Perhaps we should use a service that can make requests from different IPs etc.

if you get 407 Download limit reached, do not use persistent connections (HTTP keep-alive) and don't download more than 200 subtitles per 24 hour per IP/User. If user wants more, he can ​Become OpenSubtitles VIP member - thats one of the many reasons, why your app should allow user to login.

If you are registering User Agent to raise download limits from opensubtitles - it is wrong and it will not work. Download Limits are per User IP and User ID, it doesn't have nothing common with UA.

it is REQUIRED that your implementation supports LogIn() with user credentials (manually input user and password), because there are quite a lot of IPs, which are proxy (some service providers NAT, Singapore, Iran and others), and there is download limit per IP, so those users will hit limit very soon and OS will be not usable for them. Also there is other important reasons why to support this (VIP membership and so on). Also NEVER put in your application registered username and password, it might be blocked.

VIP Users should communicate with VIP API server, please read more in LogIn()

when downloading subtitles from server, you can set up 3-5 retries, 1-5 seconds delay. Don't abuse server with unlimited download retries and small (or no) delay

VIP: 1000 subtitles per 24 hours.

40 subtitles, per 10 seconds, per IP. But here, under the VIP section I see something different:

https://www.opensubtitles.org/en/support#vip

See X-RateLimit-Remaining: 39:

POST /xml-rpc HTTP/1.1
Host: api.opensubtitles.org
Accept-Encoding: gzip
Content-Type: text/xml
User-Agent: TemporaryUserAgent
Content-Length: 338

<?xml version='1.0'?>
<methodCall>
<methodName>LogIn</methodName>
<params>
<param>
<value><string>batchsubs</string></value>
</param>
<param>
<value><string>madabamiti</string></value>
</param>
<param>
<value><string>en</string></value>
</param>
<param>
<value><string>TemporaryUserAgent</string></value>
</param>
</params>
</methodCall>
HTTP/1.1 200 OK
Date: Sat, 10 Aug 2019 15:05:17 GMT
Content-Type: text/xml;charset=UTF-8
Content-Length: 318
Connection: keep-alive
Set-Cookie: __cfduid=de468b3875b2ec7f37128ef67b538c90a1565449516; expires=Sun, 09-Aug-20 15:05:16 GMT; path=/; domain=.opensubtitles.org; HttpOnly
Set-Cookie: weblang=en; expires=Sun, 09-Aug-2020 15:05:17 GMT; Max-Age=31536000; path=/; domain=.opensubtitles.org
Set-Cookie: PHPSESSID=k75Dy9NIN0ibwIK2YkAL2Kvqch6; expires=Sat, 10-Aug-2019 21:05:17 GMT; Max-Age=21600; path=/; domain=.opensubtitles.org; HttpOnly
Download-Quota: 200
Content-Encoding: gzip
X-Content-Encoding: gzip
X-Uncompressed-Content-Length: 972
X-Compressed-Content-Length: 318
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Origin,X-Requested-With,Content-Type,Accept,DNT,Keep-Alive,User-Agent,If-Modified-Since,Cache-Control
X-Cache-Backend: web3
Vary: Accept-Encoding
Age: 0
X-Cache: MISS
X-RateLimit-Remaining: 39
X-Via: fw1.int.opensubtitles.org
Accept-Ranges: bytes
Server: cloudflare
CF-RAY: 5042ddf8b8246795-EZE

............_O.0....Bxw-..K..F^...%j|,....H[..^0,q....'....;...J..F.Rlmg.m...I...m..............B.'Reference to deleted milestone 5'kX........U.1\..U.
%.U@.,@............M.D8..A...w{7l.y.&h.!hx.N...4.F.....C.TJ.fSgZO.?)h.3z..`.zi...r^D.a.....2...<0Q.s.d..<..X...d...."^...(....v.....=..a./..w.2.P...........kt.........;.i..(^agC.pZ.c.6t.>4../.YdL....POST /xml-rpc HTTP/1.1
Host: api.opensubtitles.org
Accept-Encoding: gzip
Content-Type: text/xml
User-Agent: TemporaryUserAgent
Content-Length: 547

<?xml version='1.0'?>
<methodCall>
<methodName>SearchSubtitles</methodName>
<params>
<param>
<value><string>k75Dy9NIN0ibwIK2YkAL2Kvqch6</string></value>
</param>
<param>
<value><array><data>
<value><struct>
<member>
<name>sublanguageid</name>
<value><string>eng,spa</string></value>
</member>
<member>
<name>moviehash</name>
<value><string>2a22b36853eb835b</string></value>
</member>
<member>
<name>moviebytesize</name>
<value><string>376635392</string></value>
</member>
</struct></value>
</data></array></value>
</param>
</params>
</methodCall>
HTTP/1.1 200 OK
Date: Sat, 10 Aug 2019 15:05:17 GMT
Content-Type: text/xml;charset=UTF-8
Content-Length: 1771
Connection: keep-alive
Set-Cookie: __cfduid=d3bb2d06219c25ed6fe2838d57f8709cf1565449517; expires=Sun, 09-Aug-20 15:05:17 GMT; path=/; domain=.opensubtitles.org; HttpOnly
Set-Cookie: PHPSESSID=k75Dy9NIN0ibwIK2YkAL2Kvqch6; expires=Sat, 10-Aug-2019 21:05:17 GMT; Max-Age=21600; path=/; domain=.opensubtitles.org; HttpOnly
Content-Encoding: gzip
X-Content-Encoding: gzip
X-Uncompressed-Content-Length: 21468
X-Compressed-Content-Length: 1771
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Origin,X-Requested-With,Content-Type,Accept,DNT,Keep-Alive,User-Agent,If-Modified-Since,Cache-Control
X-Cache-Backend: web3
Vary: Accept-Encoding
Age: 0
X-Cache: MISS
X-RateLimit-Remaining: 39
X-Via: fw1.int.opensubtitles.org
Accept-Ranges: bytes
Server: cloudflare
CF-RAY: 5042ddfc8a076795-EZE

.............r.8.._.....~......tHH.&.
..t....<...
i.V...b+.......LI.'.._.::...F...Oq.Y.,.......^.%^......y.t....~..i._.|.&9..g4.q^...........^Q.:vY6.'4f....<...._...0@.t...z...W_.+...|Z.':4...._~...SZxS.........Mi>..e<....._.....:......K...
.sE.r.i..\..WIv..`....Pl.&&.AJ.Wa.N'..@i......#t......*.,....z..)..1.B.&.......C........2..:.iyV(.9..9..F..PP...pLl+(....]........p]J....Q.F~...9.yq%Z
.g.=.b...}..g...q...,b4gk...U...YL.z.r..h..S..J..aSo
....Z...r..E....cKK-WCr=..L...k.@^3.j.9.......... .c......R..G....'$..p....z...]........E.i..~...T....H.&...(...$J....&~`C.$JU......J....%K
.......h.9.P....+(..qlb*..xN........<L..J..f|.I#.a..K.1..
......@T2...u.ck...2.1.w..s..L.g.w#v..4.Y....W....\D..{....,3[}..5dUn.w.WG......U.}.9...)..8..P.I..Sz..&7.2...$...]"i#.@.Ox.L.A..I...<.E.M)...D0cl5..#.]..w7>.._.LT.......k.7..j.......Xmw...2.yT..J3.^'....<..
..b..W.</..............0)J../.........Sla3..I....A..m...{.0'.(..E1....i..%y....4..Y.....g^..B}..].P.<......Xdt....@........ :^..MM=.m...Pk..em.;.....1...^..\NWllV]R.h...m....Vc7.....]ml.^....W...>..W6. .%.c..j........I...u.x..-.(....%...)QU.CQ.X.e..{oX?..~O.......@.j7.+..l..D..ZD.1D...~ZDI,Si.,\P&.9..
/`.&...s.D......... Q..Q.*-....5..P.a[D.....aiP......|v.BD...X. .e.-.l....(.<.2..Q.......r.(1..f..5..[.(m...<...l6.(
.2..QV...7^3...*.t.(7.(M. ..1.....VZ.}^.../...'..K..V:.Rq./-....}...x.P....0....c".....@....JhiC.K..zh  .e.-..Zn.ZB....y .D..CKl..:.c.....!vI...-.....X.....c......B..p...y.-[n.r....n.<B.......nI0p][.[*57.....[n.r..
lS..-.....;.<...r....H)15b.....>...>.F..............m...QB......S...k..9..|B..Q.6,.....{....ZO).....).iH7........I..j...AJ..p..jd...$..y.(.X.Z0.....&e...b.|s.r2...`.f..8t...........\.R.......P..d.$o.ul&..iI..e..%....:JW..:6.IP......#...B.:.......Q6.#......j0.h.[N....o...q.O...?.l...S..POST /xml-rpc HTTP/1.1
Host: api.opensubtitles.org
Accept-Encoding: gzip
Content-Type: text/xml
User-Agent: TemporaryUserAgent
Content-Length: 177

<?xml version='1.0'?>
<methodCall>
<methodName>LogOut</methodName>
<params>
<param>
<value><string>k75Dy9NIN0ibwIK2YkAL2Kvqch6</string></value>
</param>
</params>
</methodCall>
HTTP/1.1 200 OK
Date: Sat, 10 Aug 2019 15:05:18 GMT
Content-Type: text/xml;charset=UTF-8
Content-Length: 171
Connection: keep-alive
Set-Cookie: __cfduid=d3bb2d06219c25ed6fe2838d57f8709cf1565449517; expires=Sun, 09-Aug-20 15:05:17 GMT; path=/; domain=.opensubtitles.org; HttpOnly
Set-Cookie: PHPSESSID=k75Dy9NIN0ibwIK2YkAL2Kvqch6; expires=Sat, 10-Aug-2019 21:05:18 GMT; Max-Age=21600; path=/; domain=.opensubtitles.org; HttpOnly
Content-Encoding: gzip
X-Content-Encoding: gzip
X-Uncompressed-Content-Length: 283
X-Compressed-Content-Length: 171
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Origin,X-Requested-With,Content-Type,Accept,DNT,Keep-Alive,User-Agent,If-Modified-Since,Cache-Control
X-Cache-Backend: web1
Vary: Accept-Encoding
Age: 0
X-Cache: MISS
X-RateLimit-Remaining: 39
X-Via: fw1.int.opensubtitles.org
Accept-Ranges: bytes
Server: cloudflare
CF-RAY: 5042ddff1b426795-EZE

..........e.A..0...R.N.z..........@.&AqR.|
J..........M..Q..[...................[.g.{...w...2  g3.e+).!....H..c.dR....J.....:...D(6.._....-...OL...A(....s....k...|...j3....
wspek commented 5 years ago

I always see the same number for rate limiting per IP (per 10 seconds). Maybe I should first bring website online and then send the guys an email. And then implement rate limiting.

wspek commented 5 years ago

closed