The performance of many database management operations is suffering supra-linearly with the number of datasets.
There are no usages yet that require co-mingling of datasets. For this issue, alter the dataset import and readers to use a separate database for each dataset.
Although this is a simple change, it affects every occurrence of DBCursor and similar, since the connections made are now to be study-scoped. The mechanism for discovery of the correct database for a given study specifier could perhaps be a simple lookup table in a shared default database, e.g. default_study_lookup.
The performance of many database management operations is suffering supra-linearly with the number of datasets. There are no usages yet that require co-mingling of datasets. For this issue, alter the dataset import and readers to use a separate database for each dataset.
Although this is a simple change, it affects every occurrence of
DBCursorand similar, since the connections made are now to be study-scoped. The mechanism for discovery of the correct database for a given study specifier could perhaps be a simple lookup table in a shared default database, e.g.default_study_lookup.