pwsacademy / around-the-table

An open source platform that supports tabletop gaming communities, written in Swift.
BSD 2-Clause "Simplified" License
19 stars 4 forks source link

Gateway timeout when searching BGG #71

Open svanimpe opened 6 years ago

svanimpe commented 6 years ago

When a query has many results, the aggregate query to BGG can time out. For example, a search for "magic" causes the following aggregate query to be sent in GameRepository.games(forIDs:):

GET https://boardgamegeek.com/xmlapi2/thing?id=190022,156356,99665,13432,13438,9639,186185,109139,176634,43384,232929,20395,246100,36106,14813,13506,5426,46006,30956,111816,18492,23143,29975,69035,33797,34867,111497,118618,211659,232034,1738,7647,229698,58312,122975,250530,255079,233950,29714,21921,72427,35876,227525,253757,28066,33842,15163,170171,170170,124900,155435,155436,155438,13166,22150,212802,216128,219567,37193,63808,8306,183405,209452,225588,256780,203560,173784,239122,238660,238658,120658,229723,14916,23309,218358,151230,203982,17869,23849,34916,38339,170356,173639,195543,158544,199508,68032,177535,95607,145058,118637,121091,3699,137827,146466,235093,90198,254721,235089,10911,23741,10674,21785,6248,47462,14443,14442,6673,20621,9103,198730,1219,437,219426,65333,90012,255994,157292,18569,173673,99399,14297,136712,60342,176057,167512,40816,97978,59172,2763,10593,31812,104814,143744,123850,66086,7780,136382,60203,164420,21771,26359,148785,2087,63531,37812,129253,194033,210388,115832,119503,225144,35200,1319,169438,151378,36449,41916,128991,3923,209778,242740,238664,234154,230359,243342,25778,143282,45984,21756,228165,41998,153568,167314,154586,22,176070,194882,63408,17568,123417,111722,17003,31111,11028,52618,180041,111508,19757,141468,8232,30676,30675,15864,130792,28846,234204,115228,7105,22515,23178,48906,1891,463,159349,159351,153816,240307,244179,220837,244183,240369,239756,248708,218388,240299,218386,242690,242694,231392,242693,242692,167698,183271,203204,240393,240313,240310,240342,240367,240392,240308,240336,240523,183750,210402,203381,210396,210412,210400,210390,248927,210397,210410,210405,210395,210411,210401,238506,210465,210414,210399,210406,207905,210403,212298,240353,239762,231309,239754,240309,221378,240350,240524,240328,248763,239755,240614,239765,240391,239768,244279,244281,240303,238614,240305,240351,239752,239757,240386,240352,239751,240385,244187,240300,240304,240348,218575,198474,242683,242685,198476,193917,198477,242687,242689,239767,239750,240321,240525,240378,240317,240383,240306,240354,240356,239761,239760,220928,248762,203309,240347,240301,244188,239753,239764,239763,239758,248709,239759,240374,240371,256727,22371,131003,244160,14792,7741,24565,61486,161112,33720,114911,175586,2136,2002,11397,248971,128418,148521,171910,25240,5432,178645,163926,84417,29493,180157,117532,199900,212721,95231,85245,19996,30655,112897,187488,241198,195696,41001,27960,99820,169429,28574,28580,28577,28579,28575,28578,11336,60861,24922,22059,23384,146498,21909,18029,200915,167819,131149,212579,108344,149253,244618,38074,34093,26504,100419,69639,189160,25129,239698,28719,6680,14066,43150,168236,257571,228805,43100,207591,40153,131474,188363,24733,236383,70570,121919,13325,175800,91513,57116,253423,1624,43452,226119,8770,170342,7683,228765,3081,209322,92081,32210,145372,145370,181763,136085,98743,133992,43287,43289,113224,203462,27627,228438,191374,15150,2362,69733,190913,1012,9635,242208,9293,128,34772,201250,256966,22983,97928,13962,67706,39942,12233,19953,80311,96959,24092,174365,20592,165354,18028,202945,79800

BGG cannot respond to this query in time, causing a gateway time out, which gets displayed as a "BoardGameGeek did not return valid XML" error.

For large queries, we should split the query into sizeable chunks to avoid this error.