search

aiidateam / aiida-core

The official repository for the AiiDA code
https://aiida-core.readthedocs.io
Other
412 stars 184 forks source link

Archive: Respect `filter_size` in query for existing nodes #6404

Closed sphuber closed 1 month ago

sphuber commented 1 month ago

Fixes #6402

The QueryParams dataclass defines the filter_size attribute which is used in all queries to limit the number of parameters used in a query. This is necessary because without it, large archives would result in queries with a lot of parameters which can cause exceptions in database backends, such as SQLite, which define a limit of 1000 by default.

The aiida.tools.archive._import_nodes function was not respecting this setting when determining the set of nodes from the archive that already exist in the target storage. This would result in an exception when trying to import a large archive into a storage using SQLite. The problem is fixed by using the batch_iter utility to retrieve the existing UUIDs in batches of size filter_size.

sphuber commented 1 month ago

With this fix, I have successfully imported the MC3D archive into a core.sqlite_dos profile.