failing sub-queries due to biothings rate limit, timeouts?

[colleenXu]

For sub-queries with many IDs (the later hops of a Predict-style).....it seems like some of the sub-queries BTE sends to biothings and TRAPI APIs fail.

@tokebe has said that:

• for biothings APIs, it seems to be a rate limit issue (perhaps sending too many queries per minute)? *for TRAPI APIs it seems to be a timeout issue.

We'd like to explore this in more detail to see when / why it occurs and if this needs to be addressed and how...

[colleenXu]

Example of logs for semmeddb with timeouts? rate limits? happening with the following query:

bte:call-apis:query Failed to make to following query: {"url":"https://biothings.ncats.io/semmeddb/query","params":{"fields":"subject,predicate,predication_id,pmid,object","size":1000},"data":{"q":[["C0919509","INTERACTS_WITH","horm",1,1],["C1527757","INTERACTS_WITH","horm",1,1],["C1334291","INTERACTS_WITH","horm",1,1],["C1527617","INTERACTS_WITH","horm",1,1],["C1420214","INTERACTS_WITH","horm",1,1],["C1456457","INTERACTS_WITH","horm",1,1],["C1412097","INTERACTS_WITH","horm",1,1],["C1439337","INTERACTS_WITH","horm",1,1],["C1414313","INTERACTS_WITH","horm",1,1],["C1368111","INTERACTS_WITH","horm",1,1],["C1739039","INTERACTS_WITH","horm",1,1],["C1413043","INTERACTS_WITH","horm",1,1],["C0376622","INTERACTS_WITH","horm",1,1],["C1738970","INTERACTS_WITH","horm",1,1],["C1334306","INTERACTS_WITH","horm",1,1],["C3840403","INTERACTS_WITH","horm",1,1],["C3849882","INTERACTS_WITH","horm",1,1],["C0812215","INTERACTS_WITH","horm",1,1],["C0139905","INTERACTS_WITH","horm",1,1],["C1421249","INTERACTS_WITH","horm",1,1],["C1448662","INTERACTS_WITH","horm",1,1],["C1334134","INTERACTS_WITH","horm",1,1],["C1334201","INTERACTS_WITH","horm",1,1],["C1413041","INTERACTS_WITH","horm",1,1],["C1332419","INTERACTS_WITH","horm",1,1],["C3713783","INTERACTS_WITH","horm",1,1],["C1367731","INTERACTS_WITH","horm",1,1],["C0380051","INTERACTS_WITH","horm",1,1],["C0812303","INTERACTS_WITH","horm",1,1],["C1442788","INTERACTS_WITH","horm",1,1],["C0879468","INTERACTS_WITH","horm",1,1],["C4082532","INTERACTS_WITH","horm",1,1],["C1333206","INTERACTS_WITH","horm",1,1],["C0294276","INTERACTS_WITH","horm",1,1],["C0812385","INTERACTS_WITH","horm",1,1],["C0105421","INTERACTS_WITH","horm",1,1],["C1366876","INTERACTS_WITH","horm",1,1],["C1456380","INTERACTS_WITH","horm",1,1],["C1412332","INTERACTS_WITH","horm",1,1],["C1332830","INTERACTS_WITH","horm",1,1],["C3887685","INTERACTS_WITH","horm",1,1],["C1176140","INTERACTS_WITH","horm",1,1],["C1142644","INTERACTS_WITH","horm",1,1],["C1332828","INTERACTS_WITH","horm",1,1],["C0960580","INTERACTS_WITH","horm",1,1],["C1332829","INTERACTS_WITH","horm",1,1],["C0608437","INTERACTS_WITH","horm",1,1],["C1539082","INTERACTS_WITH","horm",1,1],["C3887583","INTERACTS_WITH","horm",1,1],["C1420175","INTERACTS_WITH","horm",1,1],["C0166441","INTERACTS_WITH","horm",1,1],["C1825588","INTERACTS_WITH","horm",1,1],["C2348047","INTERACTS_WITH","horm",1,1],["C1337109","INTERACTS_WITH","horm",1,1],["C1173600","INTERACTS_WITH","horm",1,1],["C0525037","INTERACTS_WITH","horm",1,1],["C4082578","INTERACTS_WITH","horm",1,1],["C2827404","INTERACTS_WITH","horm",1,1],["C0796520","INTERACTS_WITH","horm",1,1],["C1568122","INTERACTS_WITH","horm",1,1],["C0086661","INTERACTS_WITH","horm",1,1],["C0080065","INTERACTS_WITH","horm",1,1],["C1333715","INTERACTS_WITH","horm",1,1],["C1447799","INTERACTS_WITH","horm",1,1],["C1332803","INTERACTS_WITH","horm",1,1],["C1531189","INTERACTS_WITH","horm",1,1],["C1382176","INTERACTS_WITH","horm",1,1],["C0534137","INTERACTS_WITH","horm",1,1],["C0919428","INTERACTS_WITH","horm",1,1],["C1136197","INTERACTS_WITH","horm",1,1],["C0694891","INTERACTS_WITH","horm",1,1],["C0385939","INTERACTS_WITH","horm",1,1]],"scopes":["subject.umls","predicate","object.semantic_type_abbreviation","subject.novelty","object.novelty"]},"method":"post","timeout":50000}. The error is Error: timeout of 50000ms exceeded +1ms

query from demo>

``` { "message": { "query_graph": { "nodes": { "n0": { "categories": ["biolink:Gene"], "is_set": true }, "n1": { "ids": ["CHEMBL.COMPOUND:CHEMBL941"], "categories": ["biolink:SmallMolecule"], "is_set": false }, "n2": { "ids": ["MONDO:0005301"], "categories": ["biolink:Disease"], "is_set": false }, "n3": { "categories": ["biolink:SmallMolecule"], "is_set": false } }, "edges": { "e01": { "predicates": ["biolink:interacts_with"], "subject": "n0", "object": "n1" }, "e02": { "predicates": ["biolink:genetic_association"], "subject": "n0", "object": "n2" }, "e03": { "predicates": ["biolink:interacts_with"], "subject": "n0", "object": "n3" } } } } } ```

[colleenXu]

Note to @andrewsu @newgene that this issue affects demo queries and may be an issue...

[tokebe]

It's worth noting that I've since been unable to replicate my results of getting 503 service unavailable from BioThings, using a test query that had previously caused such a return when being sent repeatedly in quick succession.

This isn't the case anymore as best as I can tell, so I think my initial claim of rate limiting is now dubious at best. This leaves the timeout issue, which may be on the BioThings end, as @marcodarko has noted relatively simple queries going through normally using GET but taking a long time/not completing through POST.

[colleenXu]

Closing since this issue hasn't been easily reproducible (isn't a big deal right now) and is somewhat addressed by commits related to batch-querying like this: biothings/bte_trapi_query_graph_handler@21a9c15 and biothings/bte_trapi_query_graph_handler@2313aa5