Gensim docker

[sverhoeven]

• Improve insert speed by
  • Replacing calls to save in a loop by single bulk create. Uses more memory, but much less database queries
  • Compute overlap scores using single SQL statement instead of fetching all data to Python and updating scores. Only used when uploading gensim result
• Adds gensim for lda calculation
  • run_gensim.py script to run
    • corpus sub command to convert masspec file to corpus json
    • gensim sub command to perform lda using gensim outputting single json file with corpus and lda results
    • insert sub command to insert json file with corpus and lda results into database
    • gensim --ldaformat gensim sub command to perform lda using gensim outputting gensim formatted output
    • insert_gensim sub command to insert json file with corpus and gensim formatted lda results files into database
  • Upload gensim form to upload gensim results into database
• Docker-compose file for production version ms2lda website
• Use Redis as Celery job queue instead of RabbitMQ, as it is easier to setup
• Improved README
• Database schema has not been changed
• Updated requirements.txt and use pipenv to setup environment

[sdrogers]

All looks ok - just a couple of queries:

• Will we need to change the setup on the server re redis / rabbitmq?
• What changed to allow pipenv?

[sverhoeven]

To use redis you will need to run a redis daemon on the server and the settings.py must include the line CELERY_BROKER_URL= "redis://localhost:6379/0".

Pipenv files are generated from the requirements.txt, so only the requirements.txt file was made more complete.

[sverhoeven]

1. Redis and debug moved to own settings file
2. Upload gensim form is only visible to members of extra users table

[sverhoeven]

3. Created CHEMSPIDER_APIKEY const in settings files which is used by all chemspider code.

PS Chembl API has a image endpoint, eg. (https://www.ebi.ac.uk/chembl/api/data/image/KXEBOYWJADRGHW-SFHVURJKSA-N)

[justinjjvanderhooft]

What does the Chembl API need to draw a structure? :)

[justinjjvanderhooft]

Is it just the InchIKEY?

[sverhoeven]

Yep, ChEMBL ID or Standard InChI Key

[sverhoeven]

@sdrogers PR is ready to be merged.

[sdrogers]

Ok, done, thanks!