denisri
commented
2 years ago Looking at the code in populse_mia.main_window in MainWindow.switch_project() and printing time at different points, I see that all the time is taken while calling problem_list = data_loader.verify_scans(temp_database).
So I had a look at this function, which does the following:
- iterate over every document (file) in the database
- check that the file exists
- read the whole file in memory
- process a md5 sum for it
- check the md5 against the one stored in the database
This is very good for sanity checking, but is, in my opinion, totally inappropriate here.
- a database is meant to index many many items. Many people are working on large cohorts in neuroimaging now. The UKBiobank dataset is about 30000 subjects now, which in a processed database context means several million files indexed. Just iterating over every item is a flaw.
- reading every file (in the main thread, in addition) is totally surrealistic
- reading whole files in memory is also not an option. We are working with people dealing with histology, where files are 10-100 Gb large and cannot fit in memory.
- we want to be able to address remote data which are not present on the local filesystem, thus cannot be checked locally.
In large databases this operation may take several weeks of IO and processing (I'm not kidding !).
I propose to make this check an option (available in a menu or via a button somewhere), because being able to perform this check is sometimes useful, but it should be done only on demand. I will bypass this step in switch_project(), at least unless a specific config option in the project says that the check should be done.
servoz
Expected behavior Opening a project should be fast
Actual behavior Opening a rather small project takes about 10 seconds on my laptop (with local files), which seems not reasonable and surprisingly long. For another (a bit larger) database it's more than 20 seconds, thus seems to depend on the size of the database. While developing we restart populse_mia very often, and this time is taken at each trial/test. Even for users, the use of a database is supposed to be fast and allow smooth data opening and navigation, which is not what we observe today.