search

obiba / opal

OBiBa’s core database application for biobanks or epidemiological studies.
http://www.obiba.org/pages/products/opal/
GNU General Public License v3.0
29 stars 22 forks source link

Binary import problem #3903

Open ymarcon opened 2 weeks ago

ymarcon commented 2 weeks ago

I am finding that on some tables, I am unable to import. Both Opals are 4.7.3. On the server with the data, the Mongo version is 7.0.8 and on the server performing the import the version is 7.0.2. The error is as follows, below. Guidance would be appreciated. Other tables import successfully.

Thanks.

2024-08-20 10:13:54,354 [pool-14-thread-1] INFO  org.obiba.opal.rest.client.magma.OpalJavaClient - Disconnecting from Opal: https://opal.clsa-elcv.ca:8843/ws/
2024-08-20 10:13:54,370 [pool-14-thread-1] ERROR org.obiba.opal.shell.commands.ImportCommand - Runtime error while importing data
java.lang.RuntimeException: java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.bson.Document
    at org.obiba.opal.core.service.DataImportServiceImpl.copyValueTables(DataImportServiceImpl.java:137)
    at org.obiba.opal.core.service.DataImportServiceImpl.importData(DataImportServiceImpl.java:103)
    at org.obiba.opal.core.service.DataImportServiceImpl.importData(DataImportServiceImpl.java:83)
    at org.obiba.opal.shell.commands.ImportCommand.importFromTables(ImportCommand.java:166)
    at org.obiba.opal.shell.commands.ImportCommand.executeImports(ImportCommand.java:100)
    at org.obiba.opal.shell.commands.ImportCommand.execute(ImportCommand.java:77)
    at org.obiba.opal.shell.CommandJob.run(CommandJob.java:156)
    at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
    at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.bson.Document
    at org.obiba.magma.datasource.mongodb.MongoDBValueTableWriter$MongoDBValueSetWriter.removeFile(MongoDBValueTableWriter.java:182)
    at org.obiba.magma.datasource.mongodb.MongoDBValueTableWriter$MongoDBValueSetWriter.updateBinary(MongoDBValueTableWriter.java:176)
    at org.obiba.magma.datasource.mongodb.MongoDBValueTableWriter$MongoDBValueSetWriter.writeValue(MongoDBValueTableWriter.java:152)
    at org.obiba.magma.support.DatasourceCopier.copyValues(DatasourceCopier.java:304)
    at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetWriter.copyValue(MultithreadedDatasourceCopier.java:371)
    at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetWriter.run(MultithreadedDatasourceCopier.java:360)
    at org.obiba.magma.support.MultithreadedDatasourceCopier.write(MultithreadedDatasourceCopier.java:195)
    at org.obiba.magma.support.MultithreadedDatasourceCopier.copy(MultithreadedDatasourceCopier.java:182)
    at org.obiba.opal.core.service.CopyValueTablesLockingAction$CopyAction.execute(CopyValueTablesLockingAction.java:130)
    at org.obiba.opal.core.magma.concurrent.LockingActionTemplate$1.doInTransactionWithoutResult(LockingActionTemplate.java:47)
    at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:36)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
    at org.obiba.opal.core.magma.concurrent.LockingActionTemplate.doInTransaction(LockingActionTemplate.java:42)
    at org.obiba.opal.core.magma.concurrent.LockingActionTemplate.execute(LockingActionTemplate.java:31)
    at org.obiba.opal.core.service.DataImportServiceImpl.copyValueTables(DataImportServiceImpl.java:129)
    ... 13 common frames omitted
2024-08-20 10:13:54,469 [pool-14-thread-1] WARN  org.obiba.opal.shell.commands.ImportCommand - Import failed in 33.91 s
ymarcon commented 2 weeks ago

Working with this, we tried to export one of the affected tables. We are unable to. It quits pretty quickly and provides the following error:

java.lang.RuntimeException: org.obiba.magma.MagmaRuntimeException: Cannot retrieve content of gridFsFile [636b27bc808de225bb577a0e]
    at org.obiba.opal.core.service.DataExportServiceImpl.exportTablesToDatasource(DataExportServiceImpl.java:88)
    at org.obiba.opal.shell.commands.CopyCommand.execute(CopyCommand.java:168)
    at org.obiba.opal.shell.CommandJob.run(CommandJob.java:156)
    at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
    at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.obiba.magma.MagmaRuntimeException: Cannot retrieve content of gridFsFile [636b27bc808de225bb577a0e]
    at org.obiba.magma.datasource.mongodb.MongoDBValueLoaderFactory$MongoDBBinaryValueLoader.getByteArray(MongoDBValueLoaderFactory.java:84)
    at org.obiba.magma.datasource.mongodb.MongoDBValueLoaderFactory$MongoDBBinaryValueLoader.getValue(MongoDBValueLoaderFactory.java:71)
    at org.obiba.magma.Value.getValue(Value.java:60)
    at org.obiba.magma.type.AbstractValueType.toString(AbstractValueType.java:130)
    at org.obiba.magma.Value.toString(Value.java:104)
    at org.obiba.magma.xstream.converter.ValueConverter.marshal(ValueConverter.java:43)
    at com.thoughtworks.xstream.core.TreeMarshaller.convert(TreeMarshaller.java:71)
    at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
    at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
    at org.obiba.magma.xstream.converter.ValueSequenceConverter.marshal(ValueSequenceConverter.java:51)
    at com.thoughtworks.xstream.core.TreeMarshaller.convert(TreeMarshaller.java:71)
    at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:270)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:174)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMarshal(AbstractReflectionConverter.java:262)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:90)
    at com.thoughtworks.xstream.core.TreeMarshaller.convert(TreeMarshaller.java:71)
    at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:270)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:174)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMarshal(AbstractReflectionConverter.java:258)
    at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:90)
    at com.thoughtworks.xstream.core.TreeMarshaller.convert(TreeMarshaller.java:71)
    at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
    at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
    at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
    at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
    at com.thoughtworks.xstream.XStream.marshal(XStream.java:1303)
    at com.thoughtworks.xstream.XStream.marshal(XStream.java:1292)
    at com.thoughtworks.xstream.XStream.toXML(XStream.java:1265)
    at org.obiba.magma.datasource.fs.FsValueTableWriter$XStreamValueSetWriter.close(FsValueTableWriter.java:109)
    at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetWriter.copyValue(MultithreadedDatasourceCopier.java:372)
    at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetWriter.run(MultithreadedDatasourceCopier.java:360)
    at org.obiba.magma.support.MultithreadedDatasourceCopier.write(MultithreadedDatasourceCopier.java:195)
    at org.obiba.magma.support.MultithreadedDatasourceCopier.copy(MultithreadedDatasourceCopier.java:182)
    at org.obiba.opal.core.service.DataExportServiceImpl$ExportActionTemplate$ExportAction.exportTableToDatasource(DataExportServiceImpl.java:197)
    at org.obiba.opal.core.service.DataExportServiceImpl$ExportActionTemplate$ExportAction.execute(DataExportServiceImpl.java:166)
    at org.obiba.opal.core.magma.concurrent.LockingActionTemplate$1.doInTransactionWithoutResult(LockingActionTemplate.java:47)
    at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:36)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
    at org.obiba.opal.core.magma.concurrent.LockingActionTemplate.doInTransaction(LockingActionTemplate.java:42)
    at org.obiba.opal.core.magma.concurrent.LockingActionTemplate.execute(LockingActionTemplate.java:31)
    at org.obiba.opal.core.service.DataExportServiceImpl.exportTablesToDatasource(DataExportServiceImpl.java:80)
    ... 9 more
Caused by: org.bson.BsonInvalidOperationException: Value expected to be of type STRING is of unexpected type NULL
    at org.bson.BsonValue.throwIfInvalidType(BsonValue.java:419)
    at org.bson.BsonValue.asString(BsonValue.java:69)
    at com.mongodb.client.gridfs.codecs.GridFSFileCodec.decode(GridFSFileCodec.java:66)
    at com.mongodb.client.gridfs.codecs.GridFSFileCodec.decode(GridFSFileCodec.java:47)
    at com.mongodb.internal.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52)
    at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42)
    at org.bson.internal.LazyCodec.decode(LazyCodec.java:48)
    at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:104)
    at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42)
    at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:51)
    at com.mongodb.internal.connection.InternalStreamConnection.getCommandResult(InternalStreamConnection.java:535)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:420)
    at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:337)
    at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:116)
    at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:643)
    at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:71)
    at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:240)
    at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:226)
    at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:126)
    at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:116)
    at com.mongodb.internal.connection.DefaultServer$OperationCountTrackingConnection.command(DefaultServer.java:345)
    at com.mongodb.internal.operation.CommandOperationHelper.createReadCommandAndExecute(CommandOperationHelper.java:232)
    at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:695)
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$2(OperationHelper.java:566)
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:591)
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$3(OperationHelper.java:565)
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:591)
    at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:564)
    at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:690)
    at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:65)
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:722)
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:86)
    at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:191)
    at com.mongodb.client.internal.FindIterableImpl.first(FindIterableImpl.java:213)
    at com.mongodb.client.gridfs.GridFSBucketImpl.getFileInfoById(GridFSBucketImpl.java:547)
    at com.mongodb.client.gridfs.GridFSBucketImpl.openDownloadStream(GridFSBucketImpl.java:260)
    at com.mongodb.client.gridfs.GridFSBucketImpl.openDownloadStream(GridFSBucketImpl.java:255)
    at org.obiba.magma.datasource.mongodb.MongoDBValueLoaderFactory$MongoDBBinaryValueLoader.getByteArray(MongoDBValueLoaderFactory.java:81)
    ... 51 more
ymarcon commented 2 weeks ago

Most likely it is a mongodb gridfs upgrade issue. Cannot reproduce.