It seems like using Magnitude into a Torch DataLoader makes the database fail when using more than one worker.
The problem is, I'm converting into embedding my dataset on running time with the DataLoader worker, but when I use more than one, the process fails with this error. Everything work well using only one, or in any other context.
results = self._db().execute(search_query,
sqlite3.DatabaseError: database disk image is malformed
Since the DataLoader manages its own worker process, I cannot create every time a new magnitude object. Any ideas for a solution?
It seems like using Magnitude into a Torch
DataLoadermakes the database fail when using more than one worker.The problem is, I'm converting into embedding my dataset on running time with the
DataLoaderworker, but when I use more than one, the process fails with this error. Everything work well using only one, or in any other context.Since the
DataLoadermanages its own worker process, I cannot create every time a new magnitude object. Any ideas for a solution?