search

wincowgerDEV / OpenSpecy-shiny

This is a companion repo that holds the code for www.openspecy.org website and shiny application. The Open Specy package repo https://github.com/wincowgerDEV/OpenSpecy-package
Creative Commons Attribution 4.0 International
2 stars 1 forks source link

Force Library Matching on Web to be read primarily from disk instead of RAM #6

Closed wincowgerDEV closed 1 year ago

wincowgerDEV commented 2 years ago

The web server we have now is working alright but the RAM limit is 1 GB so we need to be more careful than before about how we utilize the RAM. We are currently parking all of library data in the RAM every time the software is run. This is ok right now since the database is only ~50mb. The new database we have is 2GB. I have a routine I have written to read these files and match them in chunks and have improved it for speed. Unfortunately, this is going to break our current file format, at least for the web files. The routine basically splits every spectrum up into its own file and reads them in one at a time. Alternatively, we could try to invest in BigQuery or something like that which would allow us to scale even larger in the future.

wincowgerDEV commented 1 year ago

More efficient to keep on RAM