Improve the efficiency of adding large numbers of entries to a dataset
[X] Fix expensive SQL queries. No need to retrieve all existing entries when adding a single entry. Use batching for large numbers
[x] Improve usability of add_entries functions in dataset models
For the first, local tests of adding 10,000 entries to a singlepoint dataset one-by-one show a reduction of 75%, with consistent (rather than steadily increasing) insertion time.
codecov[bot]
commented
9 months ago
Description
Improve the efficiency of adding large numbers of entries to a dataset
add_entriesfunctions in dataset modelsFor the first, local tests of adding 10,000 entries to a singlepoint dataset one-by-one show a reduction of 75%, with consistent (rather than steadily increasing) insertion time.
See https://github.com/openmm/spice-dataset/issues/82#issuecomment-1753480715 for the initial motivation
Changelog description
Improve the efficiency of adding large numbers of entries to a dataset
Status