Open wbari opened 5 years ago
Hit the same error again with fresh system, 5 jobs were running and all got failed with this error
2019-02-26 12:30:49 [main] INFO HadoopLocalLoadVariantStoragePipeline:327 - ============================================================ 2019-02-26 12:30:49 [main] INFO HBaseStudyConfigurationDBAdaptor:165 - Update StudyConfiguration test@GRCh37-Tuesday:platinum-Tuesday 2019-02-26 12:30:49 [main] INFO HadoopLocalLoadVariantStoragePipeline:161 - end - start = 719.512s 2019-02-26 12:30:49 [main] INFO StorageEngine:123 - PostLoad 'file:/opt/opencga/sessions/jobs/J_2532/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.variants.avro.gz' 2019-02-26 12:30:49 [main] INFO PhoenixHelper:213 - Opening connection to PhoenixDriver org.apache.phoenix.jdbc.PhoenixDriver@22f4f8f2 2019-02-26 12:30:49 [main] INFO QueryUtil:331 - Creating connection with the jdbc url: jdbc:phoenix:zk6-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net,zk1-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net,zk2-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net:2181:/hbase-unsecure; 2019-02-26 12:30:50 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:50 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:50 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:50 [main] INFO RecoverableZooKeeper:120 - Process identifier=hconnection-0x1756f7cc connecting to ZooKeeper ensemble=zk6-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net:2181,zk1-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net:2181,zk2-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net:2181 2019-02-26 12:30:50 [main] INFO ZooKeeper:438 - Initiating client connection, connectString=zk6-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net:2181,zk1-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net:2181,zk2-cgahba.hhm4yxt2jm1ujkk4pjngiah14h.fx.internal.cloudapp.net:2181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@27df95e 2019-02-26 12:30:50 [main-SendThread(10.0.0.11:2181)] INFO ClientCnxn:975 - Opening socket connection to server 10.0.0.11/10.0.0.11:2181. Will not attempt to authenticate using SASL (unknown error) 2019-02-26 12:30:50 [main-SendThread(10.0.0.11:2181)] INFO ClientCnxn:852 - Socket connection established to 10.0.0.11/10.0.0.11:2181, initiating session 2019-02-26 12:30:50 [main-SendThread(10.0.0.11:2181)] INFO ClientCnxn:1235 - Session establishment complete on server 10.0.0.11/10.0.0.11:2181, sessionid = 0x369255acc75065d, negotiated timeout = 120000 2019-02-26 12:30:50 [main] INFO Metrics:44 - Initializing metrics system: phoenix 2019-02-26 12:30:50 [main] INFO MetricsConfig:112 - loaded properties from hadoop-metrics2.properties 2019-02-26 12:30:50 [main] INFO MetricsSystemImpl:376 - Scheduled snapshot period at 10 second(s). 2019-02-26 12:30:50 [main] INFO MetricsSystemImpl:192 - phoenix metrics system started 2019-02-26 12:30:50 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:51 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:51 [main] INFO PhoenixHelper:216 - Opened Phoenix DB connection org.apache.phoenix.jdbc.PhoenixConnection@4c302f38 called from [java.lang.Thread.getStackTrace(Thread.java:1559), org.opencb.opencga.storage.hadoop.variant.index.phoenix.PhoenixHelper.newJdbcConnection(PhoenixHelper.java:215), org.opencb.opencga.storage.hadoop.variant.index.phoenix.VariantPhoenixHelper.newJdbcConnection(VariantPhoenixHelper.java:282), org.opencb.opencga.storage.hadoop.variant.adaptors.VariantHadoopDBAdaptor.getJdbcConnection(VariantHadoopDBAdaptor.java:141), org.opencb.opencga.storage.hadoop.variant.HadoopVariantStoragePipeline.registerLoadedFiles(HadoopVariantStoragePipeline.java:386), org.opencb.opencga.storage.hadoop.variant.HadoopVariantStoragePipeline.postLoad(HadoopVariantStoragePipeline.java:369), org.opencb.opencga.storage.core.StorageEngine.loadFile(StorageEngine.java:124), org.opencb.opencga.storage.core.StorageEngine.index(StorageEngine.java:99), org.opencb.opencga.storage.core.variant.VariantStorageEngine.index(VariantStorageEngine.java:338), org.opencb.opencga.storage.hadoop.variant.HadoopVariantStorageEngine.index(HadoopVariantStorageEngine.java:228), org.opencb.opencga.storage.core.manager.variant.operations.VariantFileIndexerStorageOperation.index(VariantFileIndexerStorageOperation.java:280), org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:185), org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:176), org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.index(VariantCommandExecutor.java:303), org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.execute(VariantCommandExecutor.java:115), org.opencb.opencga.app.cli.analysis.AnalysisMain.privateMain(AnalysisMain.java:98), org.opencb.opencga.app.cli.analysis.AnalysisMain.main(AnalysisMain.java:34)] 2019-02-26 12:30:51 [main] INFO VariantHadoopDBAdaptor:145 - Opened Phoenix Connection org.apache.phoenix.jdbc.PhoenixConnection@4c302f38 2019-02-26 12:30:51 [main] INFO PhoenixHelper:166 - Adding missing columns: [1_4_F, 1_4_S] 2019-02-26 12:30:51 [main] INFO PhoenixHelper:169 - ALTER VIEW "opencga_test_GRCh37-Tuesday_variants" ADD IF NOT EXISTS "1_4_F" VARCHAR ARRAY, "1_4_S" VARCHAR ARRAY 2019-02-26 12:30:51 [main] WARN HadoopVariantStoragePipeline:352 - Skip check loaded variants 2019-02-26 12:30:51 [main] INFO VariantStoragePipeline:674 - Add loaded samples to Default Cohort "ALL" 2019-02-26 12:30:51 [main] INFO HBaseStudyConfigurationDBAdaptor:165 - Update StudyConfiguration test@GRCh37-Tuesday:platinum-Tuesday 2019-02-26 12:30:51 [main] INFO StorageEngine:79 - Memory usage. MaxMemory: 7282.00 MiB TotalMemory: 2928.50 MiB FreeMemory: 875.15 MiB UsedMemory: 2053.35 MiB 2019-02-26 12:30:52 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:52 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:52 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:52 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:30:52 [main] INFO AbstractCellBaseVariantAnnotator:63 - Annotating with Cellbase REST. host 'http://bioinfo.hpc.cam.ac.uk/cellbase/', version 'v4', species 'hsapiens', assembly 'GRCh37' 2019-02-26 12:31:37 [main] WARN MetaClient:275 - CellBase REST fail. Error parsing query result for ids . hosts: http://bioinfo.hpc.cam.ac.uk/cellbase/, version: v4, category: meta, subcategory: null, resource: about, queryOptions: {"limit":1000}. Exception message: HTTP 504 Gateway Time-out 2019-02-26 12:31:37 [main] WARN MetaClient:303 - CellBase REST warning. Skipping id. 2019-02-26 12:31:37 [main] ERROR VariantFileIndexerStorageOperation:282 - Error executing INDEX org.opencb.opencga.storage.core.exceptions.StoragePipelineException: Error annotating. at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotateLoadedFiles(VariantStorageEngine.java:404) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.index(VariantStorageEngine.java:340) at org.opencb.opencga.storage.hadoop.variant.HadoopVariantStorageEngine.index(HadoopVariantStorageEngine.java:228) at org.opencb.opencga.storage.core.manager.variant.operations.VariantFileIndexerStorageOperation.index(VariantFileIndexerStorageOperation.java:280) at org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:185) at org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:176) at org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.index(VariantCommandExecutor.java:303) at org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.execute(VariantCommandExecutor.java:115) at org.opencb.opencga.app.cli.analysis.AnalysisMain.privateMain(AnalysisMain.java:98) at org.opencb.opencga.app.cli.analysis.AnalysisMain.main(AnalysisMain.java:34) Caused by: org.opencb.opencga.storage.hadoop.utils.HBaseLock$IllegalLockStatusException: Inconsistent lock status. You don't have the lock! Row: 'project', column: 'lock'. -1512091646 != 942238370 from [CURRENT-C5aBUkWSKh:1551184277562] at org.opencb.opencga.storage.hadoop.utils.HBaseLock.unlock(HBaseLock.java:189) at org.opencb.opencga.storage.hadoop.variant.metadata.HBaseProjectMetadataDBAdaptor.unLockProject(HBaseProjectMetadataDBAdaptor.java:63) at org.opencb.opencga.storage.core.metadata.StudyConfigurationManager.lockAndUpdateProject(StudyConfigurationManager.java:416) at org.opencb.opencga.storage.core.variant.annotation.DefaultVariantAnnotationManager.annotate(DefaultVariantAnnotationManager.java:113) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotate(VariantStorageEngine.java:361) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotateLoadedFiles(VariantStorageEngine.java:402) ... 9 more 2019-02-26 12:31:37 [main] INFO VariantFileIndexerStorageOperation:157 - Scanning files from /opt/opencga/sessions/jobs/J_2532 to move to file:///opt/opencga/sessions/users/test/projects/2516/2517/tmplsls/ 2019-02-26 12:31:38 [main] INFO FileManager:2331 - Study file path: /opt/opencga/sessions/users/test/projects/2516/2517/tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.file.json.gz 2019-02-26 12:31:38 [main] INFO FileManager:2332 - File path: /opt/opencga/sessions/users/test/projects/2516/2517/tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.file.json.gz 2019-02-26 12:31:38 [main] WARN FileManager:182 - The file platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.file.json.gz is not a variant transformed file 2019-02-26 12:31:38 [main] INFO FileUtils:694 - Copying file from file:///opt/opencga/sessions/jobs/J_2532/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.file.json.gz to file:///opt/opencga/sessions/users/test/projects/2516/2517/tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.file.json.gz 2019-02-26 12:31:38 [main] INFO FileUtils:200 - Checksum matches 4ba108abc497af6c6fc20789762858d0 2019-02-26 12:31:38 [main] INFO FileScanner:317 - Added file tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.file.json.gz 2019-02-26 12:31:38 [main] INFO FileManager:2331 - Study file path: /opt/opencga/sessions/users/test/projects/2516/2517/tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.variants.avro.gz 2019-02-26 12:31:38 [main] INFO FileManager:2332 - File path: /opt/opencga/sessions/users/test/projects/2516/2517/tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.variants.avro.gz 2019-02-26 12:31:38 [main] INFO FileManager:187 - Looking for vcf file in path tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz 2019-02-26 12:31:38 [main] INFO FileManager:198 - Looking for vcf file by name platinum-genomes-vcf-NA12880_S1.genome.vcf.gz 2019-02-26 12:32:09 [main] INFO FileUtils:694 - Copying file from file:///opt/opencga/sessions/jobs/J_2532/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.variants.avro.gz to file:///opt/opencga/sessions/users/test/projects/2516/2517/tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.variants.avro.gz 2019-02-26 12:33:48 [main] INFO FileUtils:200 - Checksum matches 1ce59ed33c4b3d2e48e68a9040472956 2019-02-26 12:33:48 [main] INFO FileScanner:317 - Added file tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.variants.avro.gz 2019-02-26 12:33:49 [main] INFO VariantFileIndexerStorageOperation:341 - 2 files found 2019-02-26 12:33:49 [main] WARN FileManager:182 - The file platinum-genomes-vcf-NA12880_S1.genome.vcf.gz.file.json.gz is not a variant transformed file 2019-02-26 12:33:49 [main] INFO FileManager:187 - Looking for vcf file in path tmplsls/platinum-genomes-vcf-NA12880_S1.genome.vcf.gz 2019-02-26 12:33:49 [main] INFO FileManager:198 - Looking for vcf file by name platinum-genomes-vcf-NA12880_S1.genome.vcf.gz 2019-02-26 12:33:49 [main] WARN VariantFileIndexerStorageOperation:426 - Unexpected index status. Expected TRANSFORMING, LOADING or INDEXING and got Status{name='TRANSFORMED', date='20190226123138', message='Found transformed file'} 2019-02-26 12:33:49 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:33:49 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:33:49 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:33:49 [main] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. 2019-02-26 12:33:49 [main] INFO CatalogStudyConfigurationFactory:104 - Updating StudyConfiguration 1 2019-02-26 12:33:50 [main] INFO HBaseManager:110 - Close HBase connection hconnection-0x76b74e9c 2019-02-26 12:33:50 [main] INFO ConnectionManager$HConnectionImplementation:1703 - Closing zookeeper sessionid=0x369255acc750636 2019-02-26 12:33:50 [main] INFO ZooKeeper:684 - Session: 0x369255acc750636 closed 2019-02-26 12:33:50 [main-EventThread] INFO ClientCnxn:512 - EventThread shut down 2019-02-26 12:33:50 [main] INFO HBaseManager:113 - Remaining HBase open connections: 0 2019-02-26 12:33:50 [main] INFO VariantHadoopDBAdaptor:231 - Close Phoenix connection org.apache.phoenix.jdbc.PhoenixConnection@4c302f38 called from [java.lang.Thread.getStackTrace(Thread.java:1559), org.opencb.opencga.storage.hadoop.variant.adaptors.VariantHadoopDBAdaptor.close(VariantHadoopDBAdaptor.java:230), org.opencb.opencga.storage.hadoop.variant.adaptors.VariantHadoopDBAdaptor.close(VariantHadoopDBAdaptor.java:222), org.opencb.opencga.storage.hadoop.variant.HadoopVariantStorageEngine.close(HadoopVariantStorageEngine.java:853), org.opencb.opencga.storage.core.manager.variant.operations.VariantFileIndexerStorageOperation.index(VariantFileIndexerStorageOperation.java:320), org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:185), org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:176), org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.index(VariantCommandExecutor.java:303), org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.execute(VariantCommandExecutor.java:115), org.opencb.opencga.app.cli.analysis.AnalysisMain.privateMain(AnalysisMain.java:98), org.opencb.opencga.app.cli.analysis.AnalysisMain.main(AnalysisMain.java:34)] org.opencb.opencga.storage.core.exceptions.StoragePipelineException: Error annotating. at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotateLoadedFiles(VariantStorageEngine.java:404) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.index(VariantStorageEngine.java:340) at org.opencb.opencga.storage.hadoop.variant.HadoopVariantStorageEngine.index(HadoopVariantStorageEngine.java:228) at org.opencb.opencga.storage.core.manager.variant.operations.VariantFileIndexerStorageOperation.index(VariantFileIndexerStorageOperation.java:280) at org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:185) at org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:176) at org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.index(VariantCommandExecutor.java:303) at org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.execute(VariantCommandExecutor.java:115) at org.opencb.opencga.app.cli.analysis.AnalysisMain.privateMain(AnalysisMain.java:98) at org.opencb.opencga.app.cli.analysis.AnalysisMain.main(AnalysisMain.java:34) Caused by: org.opencb.opencga.storage.hadoop.utils.HBaseLock$IllegalLockStatusException: Inconsistent lock status. You don't have the lock! Row: 'project', column: 'lock'. -1512091646 != 942238370 from [CURRENT-C5aBUkWSKh:1551184277562] at org.opencb.opencga.storage.hadoop.utils.HBaseLock.unlock(HBaseLock.java:189) at org.opencb.opencga.storage.hadoop.variant.metadata.HBaseProjectMetadataDBAdaptor.unLockProject(HBaseProjectMetadataDBAdaptor.java:63) at org.opencb.opencga.storage.core.metadata.StudyConfigurationManager.lockAndUpdateProject(StudyConfigurationManager.java:416) at org.opencb.opencga.storage.core.variant.annotation.DefaultVariantAnnotationManager.annotate(DefaultVariantAnnotationManager.java:113) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotate(VariantStorageEngine.java:361) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotateLoadedFiles(VariantStorageEngine.java:402) ... 9 more 2019-02-26 12:33:50 [Thread-20] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
2019-02-26 12:35:04 [main] INFO VariantHadoopDBAdaptor:231 - Close Phoenix connection org.apache.phoenix.jdbc.PhoenixConnection@68b11545 called from [java.lang.Thread.getStackTrace(Thread.java:1559), org.opencb.opencga.storage.hadoop.variant.adaptors.VariantHadoopDBAdaptor.close(VariantHadoopDBAdaptor.java:230), org.opencb.opencga.storage.hadoop.variant.adaptors.VariantHadoopDBAdaptor.close(VariantHadoopDBAdaptor.java:222), org.opencb.opencga.storage.hadoop.variant.HadoopVariantStorageEngine.close(HadoopVariantStorageEngine.java:853), org.opencb.opencga.storage.core.manager.variant.operations.VariantFileIndexerStorageOperation.index(VariantFileIndexerStorageOperation.java:320), org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:185), org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:176), org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.index(VariantCommandExecutor.java:303), org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.execute(VariantCommandExecutor.java:115), org.opencb.opencga.app.cli.analysis.AnalysisMain.privateMain(AnalysisMain.java:98), org.opencb.opencga.app.cli.analysis.AnalysisMain.main(AnalysisMain.java:34)] org.opencb.opencga.storage.core.exceptions.StoragePipelineException: Error annotating. at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotateLoadedFiles(VariantStorageEngine.java:404) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.index(VariantStorageEngine.java:340) at org.opencb.opencga.storage.hadoop.variant.HadoopVariantStorageEngine.index(HadoopVariantStorageEngine.java:228) at org.opencb.opencga.storage.core.manager.variant.operations.VariantFileIndexerStorageOperation.index(VariantFileIndexerStorageOperation.java:280) at org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:185) at org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:176) at org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.index(VariantCommandExecutor.java:303) at org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.execute(VariantCommandExecutor.java:115) at org.opencb.opencga.app.cli.analysis.AnalysisMain.privateMain(AnalysisMain.java:98) at org.opencb.opencga.app.cli.analysis.AnalysisMain.main(AnalysisMain.java:34) Caused by: java.lang.IllegalStateException: No runners but queue with 16 items!!! at org.opencb.commons.run.ParallelTaskRunner.readLoop(ParallelTaskRunner.java:526) at org.opencb.commons.run.ParallelTaskRunner.run(ParallelTaskRunner.java:341) at org.opencb.commons.run.ParallelTaskRunner.run(ParallelTaskRunner.java:297) at org.opencb.opencga.storage.core.variant.annotation.DefaultVariantAnnotationManager.createAnnotation(DefaultVariantAnnotationManager.java:220) at org.opencb.opencga.storage.hadoop.variant.annotation.HadoopDefaultVariantAnnotationManager.createAnnotation(HadoopDefaultVariantAnnotationManager.java:94) at org.opencb.opencga.storage.core.variant.annotation.DefaultVariantAnnotationManager.annotate(DefaultVariantAnnotationManager.java:121) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotate(VariantStorageEngine.java:361) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotateLoadedFiles(VariantStorageEngine.java:402)
CellBase went down completely and hence job failed.
Created a clean Project, study and link file , transform, load went fine, annotations failed. This was working fine before applying the stats fix changes.
2019-02-22 11:44:09 [main] INFO VariantHadoopDBAdaptor:231 - Close Phoenix connection org.apache.phoenix.jdbc.PhoenixConnection@4e3f2908 called from [java.lang.Thread.getStackTrace(Thread.java:1559), org.opencb.opencga.storage.hadoop.variant.adaptors.VariantHadoopDBAdaptor.close(VariantHadoopDBAdaptor.java:230), org.opencb.opencga.storage.hadoop.variant.adaptors.VariantHadoopDBAdaptor.close(VariantHadoopDBAdaptor.java:222), org.opencb.opencga.storage.hadoop.variant.HadoopVariantStorageEngine.close(HadoopVariantStorageEngine.java:853), org.opencb.opencga.storage.core.manager.variant.operations.VariantFileIndexerStorageOperation.index(VariantFileIndexerStorageOperation.java:320), org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:185), org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:176), org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.index(VariantCommandExecutor.java:303), org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.execute(VariantCommandExecutor.java:115), org.opencb.opencga.app.cli.analysis.AnalysisMain.privateMain(AnalysisMain.java:98), org.opencb.opencga.app.cli.analysis.AnalysisMain.main(AnalysisMain.java:34)] org.opencb.opencga.storage.core.exceptions.StoragePipelineException: Error annotating. at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotateLoadedFiles(VariantStorageEngine.java:404) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.index(VariantStorageEngine.java:340) at org.opencb.opencga.storage.hadoop.variant.HadoopVariantStorageEngine.index(HadoopVariantStorageEngine.java:228) at org.opencb.opencga.storage.core.manager.variant.operations.VariantFileIndexerStorageOperation.index(VariantFileIndexerStorageOperation.java:280) at org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:185) at org.opencb.opencga.storage.core.manager.variant.VariantStorageManager.index(VariantStorageManager.java:176) at org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.index(VariantCommandExecutor.java:303) at org.opencb.opencga.app.cli.analysis.executors.VariantCommandExecutor.execute(VariantCommandExecutor.java:115) at org.opencb.opencga.app.cli.analysis.AnalysisMain.privateMain(AnalysisMain.java:98) at org.opencb.opencga.app.cli.analysis.AnalysisMain.main(AnalysisMain.java:34) Caused by: java.lang.IllegalArgumentException: Missing annotator source version for VariantAnnotator: class org.opencb.opencga.storage.core.variant.annotation.annotators.CellBaseRestVariantAnnotator at org.opencb.opencga.storage.core.variant.annotation.VariantAnnotationManager.checkCurrentAnnotation(VariantAnnotationManager.java:84) at org.opencb.opencga.storage.core.variant.annotation.DefaultVariantAnnotationManager.lambda$annotate$0(DefaultVariantAnnotationManager.java:114) at org.opencb.opencga.storage.core.metadata.StudyConfigurationManager.lockAndUpdateProject(StudyConfigurationManager.java:407) at org.opencb.opencga.storage.core.variant.annotation.DefaultVariantAnnotationManager.annotate(DefaultVariantAnnotationManager.java:113) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotate(VariantStorageEngine.java:361) at org.opencb.opencga.storage.core.variant.VariantStorageEngine.annotateLoadedFiles(VariantStorageEngine.java:402) ... 9 more 2019-02-22 11:44:09 [Thread-13] WARN Configuration:2825 - hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring. Collapse
11:57 AM