search

aiidateam / aiida-core

The official repository for the AiiDA code
https://aiida-core.readthedocs.io
Other
435 stars 188 forks source link

SQLite Exception during archive import #5735

Closed ahkole closed 6 months ago

ahkole commented 2 years ago

Describe the bug

During the import of an archive I created I got the following exception:

Report: starting import: pseudos.aiida
Report: Parameters
-------------------------------  ----------------
Archive                          pseudos.aiida
New Node Extras                  keep
Merge Node Extras (in database)  (k)eep
Merge Node Extras (in archive)   do (n)ot create
Merge Node Extras (in both)      (l)eave existing
Merge Comments                   leave
Computer Authinfos               exclude

Report: Skipping 1 existing User(s)
Report: Collecting Node(s) ...
Report: Skipping 1 existing Node(s)
Report: Adding 69 new node(s)
Report: Adding 1 new group(s)                                                                                                                                                                                                                                  
Report: Adding 70 Node(s) to new Group(s)                                                                                                                                                                                                                      
Report: Created new import Group: PK=5, label=20221101-151034                                                                                                                                                                                                  
Report: Checking keys against repository ...                                                                                                                                                                                                                   
Report: Skipping 1 existing repository files
Report: Adding 69 new repository files
63/6911/01/2022 03:10:44 PM <8935> sqlalchemy.pool.impl.NullPool: [ERROR] Exception during reset or similar
Traceback (most recent call last):
  File "/home/arnold/Documents/PhD/projects/aiida-venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 739, in _finalize_fairy
    fairy._reset(pool)
  File "/home/arnold/Documents/PhD/projects/aiida-venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 988, in _reset
    pool._dialect.do_rollback(self)
  File "/home/arnold/Documents/PhD/projects/aiida-venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 682, in do_rollback
    dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140676517820224 and this is thread id 140676480427712.
11/01/2022 03:10:44 PM <8935> sqlalchemy.pool.impl.NullPool: [ERROR] Exception closing connection <sqlite3.Connection object at 0x7ff1aee9b3f0>
Traceback (most recent call last):
  File "/home/arnold/Documents/PhD/projects/aiida-venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 739, in _finalize_fairy
    fairy._reset(pool)
  File "/home/arnold/Documents/PhD/projects/aiida-venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 988, in _reset
    pool._dialect.do_rollback(self)
  File "/home/arnold/Documents/PhD/projects/aiida-venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 682, in do_rollback
    dbapi_connection.rollback()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140676517820224 and this is thread id 140676480427712.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/arnold/Documents/PhD/projects/aiida-venv/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 247, in _close_connection
    self._dialect.do_close(connection)
  File "/home/arnold/Documents/PhD/projects/aiida-venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 688, in do_close
    dbapi_connection.close()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140676517820224 and this is thread id 140676480427712.
Report: Committing transaction to database...                                                                                                                                                                                                                  
Success: imported archive pseudos.aiida

It does report a success at the end of the import, and all the nodes in the archive seem to be present after the import, but this exception makes me a bit worried that maybe something is corrupted in the database now.

Is there anyway to check whether the profile's database is not corrupted?

Steps to reproduce

I don't know how to reproduce the error. This is the first time I get this exception during an import of an archive. I tried reproducing it by creating a clean new profile and importing the same archive there, but then the import completed successfully without any exceptions. I might try to delete the nodes that were imported and import them again in the same profile to see if it reproduces the error, but I haven't tried that yet.

Expected behavior

No sqlite exceptions during import

Your environment

ahkole commented 2 years ago

Not sure if related, but whenever I shutdown the AiiDA daemon I get a lot of these in the postgresql logs:

2022-11-02 21:21:27.638 CET [15338] ([unknown]) {aiida_qs_arnold_4f9e95979a19b7e84b704aaa3c5b9d4d : BdG_arnold_4f9e95979a19b7e84b704aaa3c5b9d4d} LOG: unexpected EOF on client connection with an open transaction

You can see in the log line that this is caused by a transaction with one of the AiiDA database tables. Maybe the connections aren't cleanly closed when stopping the daemon?

chrisjsewell commented 2 years ago

Heya, can you provide the procedure you used to initiate the archive; from the CLI or another way?

ahkole commented 2 years ago

Heya, can you provide the procedure you used to initiate the archive; from the CLI or another way?

I created the archive using the CLI with

verdi -p profile1 archive create pseudos.aiida --groups pk

and then I tried to import it in a different profile using

verdi -p profile2 archive import pseudos.aiida

where pk was the group pk of the pseudopotentials that was created by installing a set of pseudopotentials with aiida-pseudo, profile1 was the name of the profile where I had installed the pseudos and profile2 the name of the profile where I wanted to copy the pseudos to.

chrisjsewell commented 2 years ago

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140676517820224 and this is thread id 140676480427712.

I think its related to closing of the object store, similar to: https://github.com/aiidateam/aiida-core/issues/4899

ahkole commented 2 years ago

sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140676517820224 and this is thread id 140676480427712.

I think its related to closing of the object store, similar to: #4899

Sounds like a very similar issue yes. The archive that was imported consisted of pseudopotentials, which have an associated file stored in the repository. So importing of the archive would require access to the object store.

I also see that that issue was listed as being fixed. Was that fix already included in v2.0.4. of aiida-core? Could something similar be used to fix this issue?