Shuffle protein data on first load and then store in memmap (on disk) instead of in memory.

[georgeamccarthy]

PR type

• 🏆 Enhancements
• 🏠 Internal

Purpose

• Shuffles proteins on first index.
• Uses memmap to store proteins instead of keeping them in memory due to large size (~50 MB+ for ~100k proteins).

Why?

• Currently embeddings are computed sequentially with increasing PDB ID. This makes embedding a fraction of the dataset index favour lower PDB ID which makes evaluating metric similarity less meaningful.
• Memmap is preferred for large doc arrays.
• Helps avoid memory issues on small deployment server.

Feedback required over

• A quick pair of :eyes: on the code

Mentions

• @fissoreg

Future work

• Currently using pandas to shuffle the data. One could use the jina built in .shuffle (see cookbook). However I couldn't get this working properly.

References

• Previous meeting with Jina AI devs.

Legal

• [x] I have read and agreed to the terms of contributing.

[georgeamccarthy]

Added a feature to log number of culled proteins.

[fissoreg]

Future work

• Currently using pandas to shuffle the data. One could use the jina built in .shuffle (see cookbook). However I couldn't get this working properly.

Apparently the shuffle method is a recent addition: https://github.com/jina-ai/jina/commit/2302e456165810c9d9f8d6df1505a0aabd2edc76

It will work if you upgrade:

pip install --upgrade jina

[georgeamccarthy]

Great find! TODO :)