Closed github-actions[bot] closed 1 year ago
Hi @GemmaTuron and @pittmanriley , I have tested the model on Docker and cli and it is working very slow. I have bee able o fetch it in colab but it is taking too long to make predictions. I will post the results of Colab later
eos3ae7_cli_fetch.log eos3ae7_cli_eml_pred.csv
@samuelmaina and @GemmaTuron it is also taking extremely long to run on Colab, and when I test the model on Codespaces and Docker, I'm getting all null outputs.
Codespaces: eos3ae7_single_input.log eos3ae7_codespaces.csv
Docker: eos3ae7_docker.log
It might be a memory issue as well, if it works in Colab but produces null outputs in docker. Wehn you are running the Docker container @pittmanriley you can look at the resources it uses, is it going to 8GB? because it is probably the limit!
@pittmanriley Did you check the size required by Docker to run it?
Sorry @GemmaTuron I didn't see your earlier message. I will check this soon.
@GemmaTuron when I'm running this in Docker, the highest memory usage that I saw was 635MB / 7.67GB.
I should note that due to Mac issues, I have to pull the AMD64 version of this model, which makes the model run slower. Perhaps this makes the memory usage lower than expected?
I'm also still getting null outputs on any input in Docker.
Hi all,
I tested this model out too, upon Riley's request. I agree that it is taking a very long time, even for a single string input. However, I was able to get a non-null output on Docker: eos3ae7_docker_fetch.log
FYI I also have a Mac, but my pulled model image is not AMD64. I wonder if this could be the reason why it is working on my end, but not Riley's end.
Hi,
I am also getting null outputs in my Linux.. @emmakodes could you check if you can run it?
This is the stats I get when the model is running. It is using up to 200% of CPU but space does not go above 1.5 GB
f3492f267829 eos3ae7_3880 172.61% 1.024GiB / 31.05GiB 3.30% 287kB / 25.8kB 0B / 926kB 70
Sure @GemmaTuron I will run it. Can I also maybe get the input smiles you used.
I just tested this file: test.csv
Hello @GemmaTuron
True, the model takes time to make predictions even for a single smile.
The model fetches successfully but I get a Status code: 504
and null values when I make predictions via CLI (fetch via Docker).
ersilia -v api run -i "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
{
"input": {
"key": "NQQBNZBOOHHVQP-UHFFFAOYSA-N",
"input": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]",
"text": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
},
"output": {
"outcome": [
null,
null,
null,
The model fetch successfully and generates actual predictions via CLI (fetch via Github):
ersilia -v api run -i "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
{
"input": {
"key": "NQQBNZBOOHHVQP-UHFFFAOYSA-N",
"input": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]",
"text": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
},
"output": {
"outcome": [
"Sc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(S/N2cc([N+](=O)[O-])s2)s1",
"Nc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2nnc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(-c2ncc([N+](=O)[N-])s2)s1",
"Nc1cnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Oc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
"N1cnnc(Sc2ncc([N+](=O)[O-])s2)s1",
"N=c1nc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1ncc(Sc2ncc([N+](=O)[O-])s2)s1",
"Oc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ccc([N+](=O)[O-])s2)s1",
"C1nnnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc(S(N)(=O)[O-])s2)s1",
"Fc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Cc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[O-])o2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[N-])s2)s1",
"Nc1nn(CSc2ncc([N+](=O)[O-])s2)s1"
]
}
}
The model generates actual predictions eos3ae7_docker_pred.csv
The model generates actual predictions
eos3ae7_output.csv
https://colab.research.google.com/drive/1clrqL4ZBdYLIQGo6vSv58gF1K-3Hhizy?usp=sharing
My thoughts: since the model works well via CLI (fetch via Github), Colab, and DockerHub but gives Status code: 504
and null values when I fetch via Docker on CLI. I guess the issue might be related to a Timeout error. Maybe the server (Nginx) that routes requests to the model docker container has a shorter timeout configuration and don't really wait for the model to generate predictions.
Thanks @emmakodes for testing That's interesting pointing to a timeout error, I'll add to the list of things to check!
Hi,
I have re-tested this model in different manners on my Linux system (to use the AMD64 Docker version efficiently) to be sure we are onto the right issue:
{
"input": {
"key": "CXBDYQVECUFKRK-UHFFFAOYSA-N",
"input": "CCCCOC",
"text": "CCCCOC"
},
"output": {
"outcome": [
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null,
null
]
}
}
docker run ersiliaos/eos3ae7:latest
& docker exec -it <container_name> bash
, I do get results for a single smiles:
ersilia run -i "CCCCOC"
{
"input": {
"key": "CXBDYQVECUFKRK-UHFFFAOYSA-N",
"input": "CCCCOC",
"text": "CCCCOC"
},
"output": {
"outcome": [
null,
"ClC(CO)COCl",
"ClC(C(=O)C)OCO",
"ClC(Cl)COC",
"ClC(C(=O)NCOC)=O",
"ClC(CO)COC",
"ClC(C)=O",
"ClC(CO)COC(C)/C",
"ClC(COC)COCOC(F)F",
"ClC(C=O)NC1OC1",
"ClC(Cl)/COCOCl",
"ClC(C=C)CO",
"ClC(CO)NCOCOCOCl",
"ClC(=O)O",
"ClCOC(=C)OCOC",
"ClC(C=O)COCOCl",
"ClCOC(=C)OC/C/F",
"ClC(Cl)CO",
"ClC(CO)OCO",
"ClC(=O)NCOCOCl"
]
}
}
- When I fetch the model __from_github I am also able to get outputs:
{ "input": { "key": "CXBDYQVECUFKRK-UHFFFAOYSA-N", "input": "CCCCOC", "text": "CCCCOC" }, "output": { "outcome": [ null, "ClC(C=C)CO", "ClC(C=O)OC", "ClC(C)=C(C)OCl", "ClC(C)=O", "BrC(Cl)COC", "ClC(CO)OCOCOCl", "ClCOC(=O)OCOCl", "ClC(C=O)CO", "ClC(CO)COCl", "ClC(C=C)OC", "ClC(C)=COC(NC)N", "ClC(Cl)O", "ClC(C)=CO", "ClC(Cl)/COCOCl", "ClC(Cl)CNC", "ClC(=O)c1NC1", "ClC(C/C)CO", "ClC(C)=COC", "ClC(C=O)COCOC/F" ] } }
Hi @emmakodes and @GemmaTuron the model works now when serving with a pulled docker image. The issue was, indeed, a timeout issue in nginx. This has now been resolved in Ersilia, starting from v0.1.23.
Please try the model and feel free to close the issue.
Update on the above...
Some molecules still crash, for example C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]
.
Could you maybe give them a try?
Hi @miquelduranfrigola
If you fetch the model from GitHub and wait a while (over 5 mins) you can get the predictions for this molecule:
{
"input": {
"key": "NQQBNZBOOHHVQP-UHFFFAOYSA-N",
"input": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]",
"text": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
},
"output": {
"outcome": [
"Nc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[O-])o2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1ncc(Sc2ncc([N+](=O)[O-])s2)s1",
"Cc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Sc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(S/N2cc([N+](=O)[O-])s2)s1",
"Nc1nnc(-c2ncc([N+](=O)[N-])s2)s1",
"C1nnnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nn(CSc2ncc([N+](=O)[O-])s2)s1",
"Oc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
"Oc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ccc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2nnc([N+](=O)[O-])s2)s1",
"N=c1nc(Sc2ncc([N+](=O)[O-])s2)s1",
"Fc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"N1cnnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[N-])s2)s1",
"Nc1cnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc(S(N)(=O)[O-])s2)s1"
]
}
}
If you fetch it from dockerhub, you can run it inside the docker container (via docker exec -it
{
"input": {
"key": "NQQBNZBOOHHVQP-UHFFFAOYSA-N",
"input": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]",
"text": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
},
"output": {
"outcome": [
"Nc1nnc(Sc2ncc(S(N)(=O)[O-])s2)s1",
"Nc1cnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1ncc(Sc2ncc([N+](=O)[O-])s2)s1",
"Fc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Oc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(S/N2cc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[O-])o2)s1",
"C1nnnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(-c2ncc([N+](=O)[N-])s2)s1",
"Nc1nn(CSc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Cc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Oc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
"N=c1nc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
"N1cnnc(Sc2ncc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ncc([N+](=O)[N-])s2)s1",
"Nc1nnc(Sc2nnc([N+](=O)[O-])s2)s1",
"Nc1nnc(Sc2ccc([N+](=O)[O-])s2)s1",
"Sc1nnc(Sc2ncc([N+](=O)[O-])s2)s1"
]
}
}
the molecule that was failing previously CCCOC, now works through Ersilia-Docker, so the timeout of nginx seems to be the cause, but for some molecules that are very slow to calculate it still fails. We can either leave it as is or increase the timeout even more.
This model is ready for testing. If you are assigned to this issue, please try it out using the CLI, Google Colab and DockerHub and let us know if it works!