Closed sstemann closed 1 year ago
It appears that BTE is executing this query as expected, but that none of our queried APIs are returning any info. When I swap Aicardi Syndrome for asthma, the same set of API is queried and successfully returns results (see log snippet below). When I broaden the query to ask for any NamedThing
connected to Aicardi Syndrome and no predicate constraint, I get 505 results of type Gene
, SequenceVariant
, and Disease
.
During today's standup Brett ran the query with the more general DO Identifier, results are here:
(no MolPro results) https://arax.ncats.io/?r=51a79358-4ba4-4c84-a9d4-cc2469a6585f
After the standup, I ran the query with the MONDO synonym of the DO identifier, those results are here:
(with MolPro, but looks like ARAX is hanging) https://arax.ncats.io/?r=65fe6fa9-82d3-449a-99d4-63600a5f7fa7
I posted the Query with the MONDO directly to ARAX and it gets 330 results, not sure if the hanging from ARS submission is an issue with ARS or ARAX @MarkDWilliams @dkoslicki https://arax.ncats.io/?r=49550
So I looked into this a little bit and situation is puzzling. It seems like we have this situation quite a lot where Aragorn and ARAX appear to not return results, but then when retried manually, everything seems fine. What is the cause?
Starting here from above: https://arax.ncats.io/?r=65fe6fa9-82d3-449a-99d4-63600a5f7fa7 showing that ARS thinks that ARS and Aragorn are still running.
BTE's first log message: 2022-06-28T16:28:10.210Z INFO: job status available at: http://bte.transltr.io/v1/check_query_status/YJAUqcqutr Explanatories first msg: 2022-06-28T16:27:57 DEBUG: Executing 4 predicate searches.
What came in to ARAX during that time:
ARAX logs: 2022-06-28T16:27:55.830921 INFO: asynchronous Query launching on incoming Query 2022-06-28T16:27:56.603797 INFO: Everything seems in order to begin processing the query asynchronously. Processing will continue and Response will be posted to https://ars-prod.transltr.io/ars/api/messages/cb542053-49c2-4c6e-9c23-690a20284db9 ... 2022-06-28T16:28:38.479569 DEBUG: Writing response JSON to S3 bucket
so ARAX worked on it for 45 seconds and got an answer, this one: https://arax.ncats.io/?r=49540 But when trying to POST the response to https://ars-prod.transltr.io/ars/api/messages/cb542053-49c2-4c6e-9c23-690a20284db9 it received an HTTP 500 in response.
So it seems that in the cases where ARAX appears to be not responding, it appears (at least in this case) that ARAX is doing the work but then receives a 500 when trying to POST the result back to ARS.
Maybe we need to build in some more retries? or maybe something in the response is causing a problem? I wonder if it is too large?
I wrote a little script that fetches 49540 and tries to repost it to the callback:
import requests
arax_repsonse_url= 'https://arax.ncats.io/api/arax/v1.2/response/49540'
callback_url = 'https://ars-prod.transltr.io/ars/api/messages/cb542053-49c2-4c6e-9c23-690a20284db9'
response_content = requests.get(arax_repsonse_url, headers={'accept': 'application/json'})
status_code = response_content.status_code
if status_code != 200:
print("ERROR returned with status "+str(status_code))
print(response_content)
exit()
response_dict = response_content.json()
print(f"Retrieved a message with {len(response_dict['message']['results'])} results")
print(f"Submitting to callback {callback_url}")
response_content = requests.post(callback_url, json=response_dict, headers={'accept': 'application/json'})
status_code = response_content.status_code
print(f"Status code is {status_code}")
if status_code != 200:
print("ERROR returned with status "+str(status_code))
print(response_content.content)
exit()
and it yields:
Retrieved a message with 330 results
Submitting to callback https://ars-prod.transltr.io/ars/api/messages/cb542053-49c2-4c6e-9c23-690a20284db9
Status code is 500
ERROR returned with status 500
b'Internal server error'
My wild guess is that it's a size thing. We had trouble with this in the past. could it be that ars-prod doesn't have the fix to let it receive POSTs larger than X MB. I recall that nginx had to be tweaked for ars.transltr.io? Just guessing...
Query: aircardi.json PK: 729a0522-8861-4564-bc32-24e33d7c16f0
This query was submitted to ars-prod (ITRB). Added the treats predicate specifically based on update from Vlado re: MolPro/tree.