New model ready for testing!

github-actions[bot] commented 1 year ago

This model is ready for testing. If you are assigned to this issue, please try it out using the CLI, Google Colab and DockerHub and let us know if it works!

samuelmaina commented 1 year ago

Hi @GemmaTuron and @pittmanriley , I have tested the model on Docker and cli and it is working very slow. I have bee able o fetch it in colab but it is taking too long to make predictions. I will post the results of Colab later

CLI

eos3ae7_cli_fetch.log eos3ae7_cli_eml_pred.csv

Docker

eos3ae7_docker_pred.csv

pittmanriley commented 1 year ago

@samuelmaina and @GemmaTuron it is also taking extremely long to run on Colab, and when I test the model on Codespaces and Docker, I'm getting all null outputs.

Codespaces: eos3ae7_single_input.log eos3ae7_codespaces.csv

Docker: eos3ae7_docker.log

GemmaTuron commented 1 year ago

It might be a memory issue as well, if it works in Colab but produces null outputs in docker. Wehn you are running the Docker container @pittmanriley you can look at the resources it uses, is it going to 8GB? because it is probably the limit!

GemmaTuron commented 1 year ago

@pittmanriley Did you check the size required by Docker to run it?

pittmanriley commented 1 year ago

Sorry @GemmaTuron I didn't see your earlier message. I will check this soon.

pittmanriley commented 1 year ago

@GemmaTuron when I'm running this in Docker, the highest memory usage that I saw was 635MB / 7.67GB.

I should note that due to Mac issues, I have to pull the AMD64 version of this model, which makes the model run slower. Perhaps this makes the memory usage lower than expected?

I'm also still getting null outputs on any input in Docker.

febielin commented 1 year ago

Hi all,

I tested this model out too, upon Riley's request. I agree that it is taking a very long time, even for a single string input. However, I was able to get a non-null output on Docker: eos3ae7_docker_fetch.log

FYI I also have a Mac, but my pulled model image is not AMD64. I wonder if this could be the reason why it is working on my end, but not Riley's end.

GemmaTuron commented 1 year ago

Hi,

I am also getting null outputs in my Linux.. @emmakodes could you check if you can run it?

GemmaTuron commented 1 year ago

This is the stats I get when the model is running. It is using up to 200% of CPU but space does not go above 1.5 GB

f3492f267829 eos3ae7_3880 172.61% 1.024GiB / 31.05GiB 3.30% 287kB / 25.8kB 0B / 926kB 70

emmakodes commented 1 year ago

Sure @GemmaTuron I will run it. Can I also maybe get the input smiles you used.

GemmaTuron commented 1 year ago

I just tested this file: test.csv

emmakodes commented 1 year ago

Hello @GemmaTuron

True, the model takes time to make predictions even for a single smile.

CLI ( fetch via Docker )

The model fetches successfully but I get a Status code: 504 and null values when I make predictions via CLI (fetch via Docker). ersilia -v api run -i "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"

{
    "input": {
        "key": "NQQBNZBOOHHVQP-UHFFFAOYSA-N",
        "input": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]",
        "text": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
    },
    "output": {
        "outcome": [
            null,
            null,
            null,

CLI ( fetch via Github )

The model fetch successfully and generates actual predictions via CLI (fetch via Github): ersilia -v api run -i "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"

{
    "input": {
        "key": "NQQBNZBOOHHVQP-UHFFFAOYSA-N",
        "input": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]",
        "text": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
    },
    "output": {
        "outcome": [
            "Sc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(S/N2cc([N+](=O)[O-])s2)s1",
            "Nc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2nnc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(-c2ncc([N+](=O)[N-])s2)s1",
            "Nc1cnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Oc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
            "N1cnnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "N=c1nc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1ncc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Oc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ccc([N+](=O)[O-])s2)s1",
            "C1nnnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc(S(N)(=O)[O-])s2)s1",
            "Fc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Cc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[O-])o2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[N-])s2)s1",
            "Nc1nn(CSc2ncc([N+](=O)[O-])s2)s1"
        ]
    }
}

eos3ae7_cli_pred.csv

DockerHub

The model generates actual predictions eos3ae7_docker_pred.csv

Colab

The model generates actual predictions eos3ae7_output.csv https://colab.research.google.com/drive/1clrqL4ZBdYLIQGo6vSv58gF1K-3Hhizy?usp=sharing

My thoughts: since the model works well via CLI (fetch via Github), Colab, and DockerHub but gives Status code: 504 and null values when I fetch via Docker on CLI. I guess the issue might be related to a Timeout error. Maybe the server (Nginx) that routes requests to the model docker container has a shorter timeout configuration and don't really wait for the model to generate predictions.

GemmaTuron commented 1 year ago

Thanks @emmakodes for testing That's interesting pointing to a timeout error, I'll add to the list of things to check!

GemmaTuron commented 1 year ago

Hi,

I have re-tested this model in different manners on my Linux system (to use the AMD64 Docker version efficiently) to be sure we are onto the right issue:

If I fetch the model through Ersilia using the CLI --from_dockerhub, I get None Outputs even for a single smiles:

{
"input": {
    "key": "CXBDYQVECUFKRK-UHFFFAOYSA-N",
    "input": "CCCCOC",
    "text": "CCCCOC"
},
"output": {
    "outcome": [
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null
    ]
}
}

For the same smiles, if I run the Docker container that has been fetched directly using the docker commands docker run ersiliaos/eos3ae7:latest & docker exec -it <container_name> bash, I do get results for a single smiles: ersilia run -i "CCCCOC"


{
"input": {
    "key": "CXBDYQVECUFKRK-UHFFFAOYSA-N",
    "input": "CCCCOC",
    "text": "CCCCOC"
},
"output": {
    "outcome": [
        null,
        "ClC(CO)COCl",
        "ClC(C(=O)C)OCO",
        "ClC(Cl)COC",
        "ClC(C(=O)NCOC)=O",
        "ClC(CO)COC",
        "ClC(C)=O",
        "ClC(CO)COC(C)/C",
        "ClC(COC)COCOC(F)F",
        "ClC(C=O)NC1OC1",
        "ClC(Cl)/COCOCl",
        "ClC(C=C)CO",
        "ClC(CO)NCOCOCOCl",
        "ClC(=O)O",
        "ClCOC(=C)OCOC",
        "ClC(C=O)COCOCl",
        "ClCOC(=C)OC/C/F",
        "ClC(Cl)CO",
        "ClC(CO)OCO",
        "ClC(=O)NCOCOCl"
    ]
}
}

- When I fetch the model __from_github I am also able to get outputs:

{ "input": { "key": "CXBDYQVECUFKRK-UHFFFAOYSA-N", "input": "CCCCOC", "text": "CCCCOC" }, "output": { "outcome": [ null, "ClC(C=C)CO", "ClC(C=O)OC", "ClC(C)=C(C)OCl", "ClC(C)=O", "BrC(Cl)COC", "ClC(CO)OCOCOCl", "ClCOC(=O)OCOCl", "ClC(C=O)CO", "ClC(CO)COCl", "ClC(C=C)OC", "ClC(C)=COC(NC)N", "ClC(Cl)O", "ClC(C)=CO", "ClC(Cl)/COCOCl", "ClC(Cl)CNC", "ClC(=O)c1NC1", "ClC(C/C)CO", "ClC(C)=COC", "ClC(C=O)COCOC/F" ] } }

miquelduranfrigola commented 1 year ago

Hi @emmakodes and @GemmaTuron the model works now when serving with a pulled docker image. The issue was, indeed, a timeout issue in nginx. This has now been resolved in Ersilia, starting from v0.1.23.

Please try the model and feel free to close the issue.

miquelduranfrigola commented 1 year ago

Update on the above...

Some molecules still crash, for example C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-].

Could you maybe give them a try?

GemmaTuron commented 1 year ago

Hi @miquelduranfrigola

If you fetch the model from GitHub and wait a while (over 5 mins) you can get the predictions for this molecule:

{
    "input": {
        "key": "NQQBNZBOOHHVQP-UHFFFAOYSA-N",
        "input": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]",
        "text": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
    },
    "output": {
        "outcome": [
            "Nc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[O-])o2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1ncc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Cc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Sc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(S/N2cc([N+](=O)[O-])s2)s1",
            "Nc1nnc(-c2ncc([N+](=O)[N-])s2)s1",
            "C1nnnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nn(CSc2ncc([N+](=O)[O-])s2)s1",
            "Oc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
            "Oc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ccc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2nnc([N+](=O)[O-])s2)s1",
            "N=c1nc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Fc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "N1cnnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[N-])s2)s1",
            "Nc1cnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc(S(N)(=O)[O-])s2)s1"
        ]
    }
}

If you fetch it from dockerhub, you can run it inside the docker container (via docker exec -it bash) and get the predictions after a while, but not through Ersilia.

{
    "input": {
        "key": "NQQBNZBOOHHVQP-UHFFFAOYSA-N",
        "input": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]",
        "text": "C1=C(SC(=N1)SC2=NN=C(S2)N)[N+](=O)[O-]"
    },
    "output": {
        "outcome": [
            "Nc1nnc(Sc2ncc(S(N)(=O)[O-])s2)s1",
            "Nc1cnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1ncc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Fc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Oc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(S/N2cc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[O-])o2)s1",
            "C1nnnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(-c2ncc([N+](=O)[N-])s2)s1",
            "Nc1nn(CSc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Cc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Oc1nnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "N=c1nc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(-c2ncc([N+](=O)[O-])s2)s1",
            "N1cnnc(Sc2ncc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ncc([N+](=O)[N-])s2)s1",
            "Nc1nnc(Sc2nnc([N+](=O)[O-])s2)s1",
            "Nc1nnc(Sc2ccc([N+](=O)[O-])s2)s1",
            "Sc1nnc(Sc2ncc([N+](=O)[O-])s2)s1"
        ]
    }
}

the molecule that was failing previously CCCOC, now works through Ersilia-Docker, so the timeout of nginx seems to be the cause, but for some molecules that are very slow to calculate it still fails. We can either leave it as is or increase the timeout even more.

ersilia-os / eos3ae7