When adding to LMDB from both the crossref 2021 dump as well as the incremental dump i get tons of errors that the entry is rejected. I have no clue what could be the error or how to fix it. Is this something any of you have encountered and if so, what could be the error? Could it be that the file has been corrupted or is this due to a bug?
The db file was 300gb (About 90m records curling /service/data) when it stopped working and it feels like it stopped suddenly because the process chewed on but the file was not modified on disk at all.
Here is the full stack trace
ERROR [2023-02-10 08:33:49,853] com.scienceminer.lookup.storage.lookup.MetadataLookup: Cannot store the entry 10.4324/9780429428463-101, {"publisher-location":"London","edition-number":"1","reference-count":0,"publisher":"Routledge","content-domain":{"domain":[],"crossmark-restriction":false},"published-print":{"date-parts":[[2021,5,27]]},"DOI":"10.4324/9780429428463-101","type":"book-chapter","created":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T19:15:44Z","timestamp":1622142944000},"page":"1-2","source":"Crossref","is-referenced-by-count":0,"title":["General introduction"],"prefix":"10.4324","author":[{"given":"Bernard","family":"Schmitt","sequence":"first","affiliation":[]},{"given":"Alvaro","family":"Cencini","sequence":"additional","affiliation":[]},{"given":"Xavier","family":"Bradley","sequence":"additional","affiliation":[]}],"member":"301","container-title":["Inflation, Unemployment and Capital Malformations"],"deposited":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T19:15:45Z","timestamp":1622142945000},"score":0.0,"resource":{"primary":{"URL":"https://www.taylorfrancis.com/books/9780429428463/chapters/10.4324/9780429428463-101"}},"issued":{"date-parts":[[2021,5,27]]},"references-count":0,"URL":"http://dx.doi.org/10.4324/9780429428463-101","published":{"date-parts":[[2021,5,27]]}}
! org.lmdbjava.Txn$BadException: Transaction must abort, has a child, or is invalid (-30782)
! at org.lmdbjava.ResultCodeMapper.checkRc(ResultCodeMapper.java:70)
! at org.lmdbjava.Dbi.put(Dbi.java:411)
! at com.scienceminer.lookup.storage.lookup.MetadataLookup.store(MetadataLookup.java:110)
! at com.scienceminer.lookup.storage.lookup.MetadataLookup.lambda$loadFromFile$0(MetadataLookup.java:95)
! at com.scienceminer.lookup.reader.CrossrefJsonlReader.lambda$load$0(CrossrefJsonlReader.java:39)
! at java.util.Iterator.forEachRemaining(Iterator.java:116)
! at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
! at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
! at com.scienceminer.lookup.reader.CrossrefJsonlReader.load(CrossrefJsonlReader.java:33)
! at com.scienceminer.lookup.storage.lookup.MetadataLookup.loadFromFile(MetadataLookup.java:86)
! at com.scienceminer.lookup.utils.crossrefclient.IncrementalLoaderTask$LoadCrossrefFile.run(IncrementalLoaderTask.java:268)
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
! at java.util.concurrent.FutureTask.run(FutureTask.java:266)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
! at java.lang.Thread.run(Thread.java:750)
ERROR [2023-02-10 08:33:49,853] com.scienceminer.lookup.utils.crossrefclient.IncrementalLoaderTask: Error while processing /app/data/incremental_dumps/2023-02-10/G1000132.json.gz
! org.lmdbjava.Txn$BadException: Transaction must abort, has a child, or is invalid (-30782)
! at org.lmdbjava.ResultCodeMapper.checkRc(ResultCodeMapper.java:70)
! at org.lmdbjava.Txn.commit(Txn.java:110)
! at com.scienceminer.lookup.storage.lookup.MetadataLookup.loadFromFile(MetadataLookup.java:99)
! at com.scienceminer.lookup.utils.crossrefclient.IncrementalLoaderTask$LoadCrossrefFile.run(IncrementalLoaderTask.java:268)
! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
! at java.util.concurrent.FutureTask.run(FutureTask.java:266)
! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
! at java.lang.Thread.run(Thread.java:750)
Hello,
When adding to LMDB from both the crossref 2021 dump as well as the incremental dump i get tons of errors that the entry is rejected. I have no clue what could be the error or how to fix it. Is this something any of you have encountered and if so, what could be the error? Could it be that the file has been corrupted or is this due to a bug?
The db file was 300gb (About 90m records curling
/service/data) when it stopped working and it feels like it stopped suddenly because the process chewed on but the file was not modified on disk at all.Here is the full stack trace
ERROR [2023-02-10 08:33:49,853] com.scienceminer.lookup.storage.lookup.MetadataLookup: Cannot store the entry 10.4324/9780429428463-101, {"publisher-location":"London","edition-number":"1","reference-count":0,"publisher":"Routledge","content-domain":{"domain":[],"crossmark-restriction":false},"published-print":{"date-parts":[[2021,5,27]]},"DOI":"10.4324/9780429428463-101","type":"book-chapter","created":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T19:15:44Z","timestamp":1622142944000},"page":"1-2","source":"Crossref","is-referenced-by-count":0,"title":["General introduction"],"prefix":"10.4324","author":[{"given":"Bernard","family":"Schmitt","sequence":"first","affiliation":[]},{"given":"Alvaro","family":"Cencini","sequence":"additional","affiliation":[]},{"given":"Xavier","family":"Bradley","sequence":"additional","affiliation":[]}],"member":"301","container-title":["Inflation, Unemployment and Capital Malformations"],"deposited":{"date-parts":[[2021,5,27]],"date-time":"2021-05-27T19:15:45Z","timestamp":1622142945000},"score":0.0,"resource":{"primary":{"URL":"https://www.taylorfrancis.com/books/9780429428463/chapters/10.4324/9780429428463-101"}},"issued":{"date-parts":[[2021,5,27]]},"references-count":0,"URL":"http://dx.doi.org/10.4324/9780429428463-101","published":{"date-parts":[[2021,5,27]]}} ! org.lmdbjava.Txn$BadException: Transaction must abort, has a child, or is invalid (-30782) ! at org.lmdbjava.ResultCodeMapper.checkRc(ResultCodeMapper.java:70) ! at org.lmdbjava.Dbi.put(Dbi.java:411) ! at com.scienceminer.lookup.storage.lookup.MetadataLookup.store(MetadataLookup.java:110) ! at com.scienceminer.lookup.storage.lookup.MetadataLookup.lambda$loadFromFile$0(MetadataLookup.java:95) ! at com.scienceminer.lookup.reader.CrossrefJsonlReader.lambda$load$0(CrossrefJsonlReader.java:39) ! at java.util.Iterator.forEachRemaining(Iterator.java:116) ! at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ! at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647) ! at com.scienceminer.lookup.reader.CrossrefJsonlReader.load(CrossrefJsonlReader.java:33) ! at com.scienceminer.lookup.storage.lookup.MetadataLookup.loadFromFile(MetadataLookup.java:86) ! at com.scienceminer.lookup.utils.crossrefclient.IncrementalLoaderTask$LoadCrossrefFile.run(IncrementalLoaderTask.java:268) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ! at java.lang.Thread.run(Thread.java:750) ERROR [2023-02-10 08:33:49,853] com.scienceminer.lookup.utils.crossrefclient.IncrementalLoaderTask: Error while processing /app/data/incremental_dumps/2023-02-10/G1000132.json.gz ! org.lmdbjava.Txn$BadException: Transaction must abort, has a child, or is invalid (-30782) ! at org.lmdbjava.ResultCodeMapper.checkRc(ResultCodeMapper.java:70) ! at org.lmdbjava.Txn.commit(Txn.java:110) ! at com.scienceminer.lookup.storage.lookup.MetadataLookup.loadFromFile(MetadataLookup.java:99) ! at com.scienceminer.lookup.utils.crossrefclient.IncrementalLoaderTask$LoadCrossrefFile.run(IncrementalLoaderTask.java:268) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ! at java.lang.Thread.run(Thread.java:750)