Extend indexing to more proteins

[georgeamccarthy]

Is your feature request related to a problem? Please describe.

• Currently only 42 proteins are being indexed.
• We want to index more proteins. If computationally feasible we will index the entire dataset which includes 10,000 proteins.

Describe the solution you'd like

Option 1 - theoretical computation time

• Figure out how much time each additional protein adds to encoding time. E.g. try with 20/50/100 proteins and measure the difference.
• Calculate the time to index 10,000 proteins.

Option 2 - attempt to compute

• Try to compute all 10,000 on our machines and see if it will complete in a few hours. @fissoreg Can you try this? Currently my machine is taking too long to compute the embeddings, probably due to a bug - see #27. I recall that you can compute the 42 quite quickly.

[georgeamccarthy]

@fissoreg let's disable flow.block() and run backend/app.py with 42 proteins and compare our computation time.

My output was: python backend/app.py 190.66s user 8.11s system 102% cpu 3:13.73 total

[fissoreg]

Here is my time: python backend/app.py 179.57s user 34.62s system 302% cpu 1:10.85 total

Not a big difference, if I correctly interpret the numbers. It just seems like more cores are used on my PCs.

[georgeamccarthy]

Could you share your CPU, RAM and GPU specs please?

[fissoreg]

Nothing exceptional:

• Intel i5 @2.80 GHz (4 cores)
• 16 GB RAM
• no GPU

I guess this conversation is relevant for #27

[georgeamccarthy]

Closing as it looks like the indexing up to 100k proteins should be feasible on low-mid spec hardware.