search

bulik / ldsc

LD Score Regression (LDSC)
GNU General Public License v3.0
643 stars 343 forks source link

Run Partitioned heritability have errors #103

Open xiangboyulan opened 6 years ago

xiangboyulan commented 6 years ago

Dear everyone,

When I run the partitioned heritability, there have some errors: xiangbos-MacBook-Pro:ldsc_mod xiangbo$ python ldsc.py --h2 sleep.sumstats.gz --w-ld-chr /Users/xiangbo/mtag/mtag/ldsc_mod/weights_hm3_no_hla/weights. --ref-ld-chr /Users/xiangbo/mtag/mtag/ldsc_mod/baseline_v1.1/baseline. --overlap-annot --frqfile-chr /Users/xiangbo/mtag/mtag/ldsc_mod/1000G_frq/1000G.mac5eur. --out sleep_baseline

Beginning analysis at Mon Feb 5 09:46:59 2018 Reading summary statistics from sleep.sumstats.gz ... Read summary statistics for 387424 SNPs. Reading reference panel LD Score from /Users/xiangbo/mtag/mtag/ldsc_mod/baseline_v1.1/baseline.[1-22] ... Read reference panel LD Scores for 1190321 SNPs. Removing partitioned LD Scores with zero variance. Reading regression weight LD Score from /Users/xiangbo/mtag/mtag/ldsc_mod/weights_hm3_no_hla/weights.[1-22] ... Read regression weight LD Scores for 1242190 SNPs. After merging with reference panel LD, 384737 SNPs remain. After merging with regression SNP LD, 348714 SNPs remain. Removed 0 SNPs with chi^2 > 113.006 (348714 SNPs remain) Total Observed scale h2: -0.1764 (0.0187) Categories: baseL2_0 Coding_UCSC.bedL2_0 Coding_UCSC.extend.500.bedL2_0 Conserved_LindbladToh.bedL2_0 Conserved_LindbladToh.extend.500.bedL2_0 CTCF_Hoffman.bedL2_0 CTCF_Hoffman.extend.500.bedL2_0 DGF_ENCODE.bedL2_0 DGF_ENCODE.extend.500.bedL2_0 DHS_peaks_Trynka.bedL2_0 DHS_Trynka.bedL2_0 DHS_Trynka.extend.500.bedL2_0 Enhancer_Andersson.bedL2_0 Enhancer_Andersson.extend.500.bedL2_0 Enhancer_Hoffman.bedL2_0 Enhancer_Hoffman.extend.500.bedL2_0 FetalDHS_Trynka.bedL2_0 FetalDHS_Trynka.extend.500.bedL2_0 H3K27ac_Hnisz.bedL2_0 H3K27ac_Hnisz.extend.500.bedL2_0 H3K27ac_PGC2.bedL2_0 H3K27ac_PGC2.extend.500.bedL2_0 H3K4me1_peaks_Trynka.bedL2_0 H3K4me1_Trynka.bedL2_0 H3K4me1_Trynka.extend.500.bedL2_0 H3K4me3_peaks_Trynka.bedL2_0 H3K4me3_Trynka.bedL2_0 H3K4me3_Trynka.extend.500.bedL2_0 H3K9ac_peaks_Trynka.bedL2_0 H3K9ac_Trynka.bedL2_0 H3K9ac_Trynka.extend.500.bedL2_0 Intron_UCSC.bedL2_0 Intron_UCSC.extend.500.bedL2_0 PromoterFlanking_Hoffman.bedL2_0 PromoterFlanking_Hoffman.extend.500.bedL2_0 Promoter_UCSC.bedL2_0 Promoter_UCSC.extend.500.bedL2_0 Repressed_Hoffman.bedL2_0 Repressed_Hoffman.extend.500.bedL2_0 SuperEnhancer_Hnisz.bedL2_0 SuperEnhancer_Hnisz.extend.500.bedL2_0 TFBS_ENCODE.bedL2_0 TFBS_ENCODE.extend.500.bedL2_0 Transcribed_Hoffman.bedL2_0 Transcribed_Hoffman.extend.500.bedL2_0 TSS_Hoffman.bedL2_0 TSS_Hoffman.extend.500.bedL2_0 UTR_3_UCSC.bedL2_0 UTR_3_UCSC.extend.500.bedL2_0 UTR_5_UCSC.bedL2_0 UTR_5_UCSC.extend.500.bedL2_0 WeakEnhancer_Hoffman.bedL2_0 WeakEnhancer_Hoffman.extend.500.bedL2_0 Lambda GC: 1.3238 Mean Chi^2: 2.108 Intercept: 2.4435 (0.0341) Ratio: 1.3028 (0.0307) Reading annot matrix from /Users/xiangbo/mtag/mtag/ldsc_mod/baseline_v1.1/baseline.[1-22] ... /Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py:120: UserWarning: Boolean Series key will be reindexed to match DataFrame index. df_annot = df_annot[(.95 > df_frq.FRQ) & (df_frq.FRQ > 0.05)] Error parsing .annot file. Traceback (most recent call last): File "ldsc.py", line 644, in sumstats.estimate_h2(args, log) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 371, in estimate_h2 overlap_matrix, M_tot = _read_annot(args, log) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 97, in _read_annot 'annot matrix', ps.annot, frqfile=args.frqfile_chr) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 153, in _read_chr_split_files out = parsefunc(_splitp(chr_arg), _N_CHR, **kwargs) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py", line 197, in annot for i, fh in enumerate(fh_list)] File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py", line 120, in annot_parser df_annot = df_annot[(.95 > df_frq.FRQ) & (df_frq.FRQ > 0.05)] File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 1958, in getitem return self._getitem_array(key) File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 1998, in _getitem_array key = check_bool_indexer(self.index, key) File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/indexing.py", line 1939, in check_bool_indexer raise IndexingError('Unalignable boolean Series provided as ' IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match

Analysis finished at Mon Feb 5 09:47:38 2018 Total time elapsed: 39.41s Traceback (most recent call last): File "ldsc.py", line 644, in sumstats.estimate_h2(args, log) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 371, in estimate_h2 overlap_matrix, M_tot = _read_annot(args, log) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 97, in _read_annot 'annot matrix', ps.annot, frqfile=args.frqfile_chr) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 153, in _read_chr_split_files out = parsefunc(_splitp(chr_arg), _N_CHR, **kwargs) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py", line 197, in annot for i, fh in enumerate(fh_list)] File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py", line 120, in annot_parser df_annot = df_annot[(.95 > df_frq.FRQ) & (df_frq.FRQ > 0.05)] File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 1958, in getitem return self._getitem_array(key) File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 1998, in _getitem_array key = check_bool_indexer(self.index, key) File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/indexing.py", line 1939, in check_bool_indexer raise IndexingError('Unalignable boolean Series provided as ' pandas.core.indexing.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match

Please give some advice. Thank you very much!

Bo

rkwalters commented 6 years ago

Hi Bo, The one time I’ve seen this error message occur previously it was caused by a mismatch between the set of variants in the LD score files and the set of variants in the frqfile (thread here https://github.com/bulik/ldsc/issues/85). From the filenames, I’m guessing you may have 1000 Genomes Phase 3 LD scores and a 1000 Genomes Phase 1 frqfile. Can you see if switching to the Phase 3 frqfile (available here https://data.broadinstitute.org/alkesgroup/LDSCORE/) fixes this? Cheers, Raymond

On Feb 4, 2018, at 8:54 PM, xiangboyulan notifications@github.com wrote:

Dear everyone,

When I run the partitioned heritability, there have some errors: xiangbos-MacBook-Pro:ldsc_mod xiangbo$ python ldsc.py --h2 sleep.sumstats.gz --w-ld-chr /Users/xiangbo/mtag/mtag/ldsc_mod/weights_hm3_no_hla/weights. --ref-ld-chr /Users/xiangbo/mtag/mtag/ldsc_mod/baseline_v1.1/baseline. --overlap-annot --frqfile-chr /Users/xiangbo/mtag/mtag/ldsc_mod/1000G_frq/1000G.mac5eur. --out sleep_baseline

LD Score Regression (LDSC) Version 1.0.0 (C) 2014-2015 Brendan Bulik-Sullivan and Hilary Finucane Broad Institute of MIT and Harvard / MIT Department of Mathematics GNU General Public License v3 Call: ./ldsc.py --h2 sleep.sumstats.gz --ref-ld-chr /Users/xiangbo/mtag/mtag/ldsc_mod/baseline_v1.1/baseline. --out sleep_baseline --overlap-annot --frqfile-chr /Users/xiangbo/mtag/mtag/ldsc_mod/1000G_frq/1000G.mac5eur. --w-ld-chr /Users/xiangbo/mtag/mtag/ldsc_mod/weights_hm3_no_hla/weights.

Beginning analysis at Mon Feb 5 09:46:59 2018 Reading summary statistics from sleep.sumstats.gz ... Read summary statistics for 387424 SNPs. Reading reference panel LD Score from /Users/xiangbo/mtag/mtag/ldsc_mod/baseline_v1.1/baseline.[1-22] ... Read reference panel LD Scores for 1190321 SNPs. Removing partitioned LD Scores with zero variance. Reading regression weight LD Score from /Users/xiangbo/mtag/mtag/ldsc_mod/weights_hm3_no_hla/weights.[1-22] ... Read regression weight LD Scores for 1242190 SNPs. After merging with reference panel LD, 384737 SNPs remain. After merging with regression SNP LD, 348714 SNPs remain. Removed 0 SNPs with chi^2 > 113.006 (348714 SNPs remain) Total Observed scale h2: -0.1764 (0.0187) Categories: baseL2_0 Coding_UCSC.bedL2_0 Coding_UCSC.extend.500.bedL2_0 Conserved_LindbladToh.bedL2_0 Conserved_LindbladToh.extend.500.bedL2_0 CTCF_Hoffman.bedL2_0 CTCF_Hoffman.extend.500.bedL2_0 DGF_ENCODE.bedL2_0 DGF_ENCODE.extend.500.bedL2_0 DHS_peaks_Trynka.bedL2_0 DHS_Trynka.bedL2_0 DHS_Trynka.extend.500.bedL2_0 Enhancer_Andersson.bedL2_0 Enhancer_Andersson.extend.500.bedL2_0 Enhancer_Hoffman.bedL2_0 Enhancer_Hoffman.extend.500.bedL2_0 FetalDHS_Trynka.bedL2_0 FetalDHS_Trynka.extend.500.bedL2_0 H3K27ac_Hnisz.bedL2_0 H3K27ac_Hnisz.extend.500.bedL2_0 H3K27ac_PGC2.bedL2_0 H3K27ac_PGC2.extend.500.bedL2_0 H3K4me1_peaks_Trynka.bedL2_0 H3K4me1_Trynka.bedL2_0 H3K4me1_Trynka.extend.500.bedL2_0 H3K4me3_peaks_Trynka.bedL2_0 H3K4me3_Trynka.bedL2_0 H3K4me3_Trynka.extend.500.bedL2_0 H3K9ac_peaks_Trynka.bedL2_0 H3K9ac_Trynka.bedL2_0 H3K9ac_Trynka.extend.500.bedL2_0 Intron_UCSC.bedL2_0 Intron_UCSC.extend.500.bedL2_0 PromoterFlanking_Hoffman.bedL2_0 PromoterFlanking_Hoffman.extend.500.bedL2_0 Promoter_UCSC.bedL2_0 Promoter_UCSC.extend.500.bedL2_0 Repressed_Hoffman.bedL2_0 Repressed_Hoffman.extend.500.bedL2_0 SuperEnhancer_Hnisz.bedL2_0 SuperEnhancer_Hnisz.extend.500.bedL2_0 TFBS_ENCODE.bedL2_0 TFBS_ENCODE.extend.500.bedL2_0 Transcribed_Hoffman.bedL2_0 Transcribed_Hoffman.extend.500.bedL2_0 TSS_Hoffman.bedL2_0 TSS_Hoffman.extend.500.bedL2_0 UTR_3_UCSC.bedL2_0 UTR_3_UCSC.extend.500.bedL2_0 UTR_5_UCSC.bedL2_0 UTR_5_UCSC.extend.500.bedL2_0 WeakEnhancer_Hoffman.bedL2_0 WeakEnhancer_Hoffman.extend.500.bedL2_0 Lambda GC: 1.3238 Mean Chi^2: 2.108 Intercept: 2.4435 (0.0341) Ratio: 1.3028 (0.0307) Reading annot matrix from /Users/xiangbo/mtag/mtag/ldsc_mod/baseline_v1.1/baseline.[1-22] ... /Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py:120: UserWarning: Boolean Series key will be reindexed to match DataFrame index. df_annot = df_annot[(.95 > df_frq.FRQ) & (df_frq.FRQ > 0.05)] Error parsing .annot file. Traceback (most recent call last): File "ldsc.py", line 644, in sumstats.estimate_h2(args, log) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 371, in estimate_h2 overlap_matrix, M_tot = _read_annot(args, log) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 97, in _read_annot 'annot matrix', ps.annot, frqfile=args.frqfile_chr) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 153, in _read_chr_split_files out = parsefunc(_splitp(chr_arg), _N_CHR, **kwargs) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py", line 197, in annot for i, fh in enumerate(fh_list)] File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py", line 120, in annot_parser df_annot = df_annot[(.95 > df_frq.FRQ) & (df_frq.FRQ > 0.05)] File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 1958, in getitem return self._getitem_array(key) File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 1998, in _getitem_array key = check_bool_indexer(self.index, key) File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/indexing.py", line 1939, in check_bool_indexer raise IndexingError('Unalignable boolean Series provided as ' IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match

Analysis finished at Mon Feb 5 09:47:38 2018 Total time elapsed: 39.41s Traceback (most recent call last): File "ldsc.py", line 644, in sumstats.estimate_h2(args, log) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 371, in estimate_h2 overlap_matrix, M_tot = _read_annot(args, log) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 97, in _read_annot 'annot matrix', ps.annot, frqfile=args.frqfile_chr) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/sumstats.py", line 153, in _read_chr_split_files out = parsefunc(_splitp(chr_arg), _N_CHR, **kwargs) File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py", line 197, in annot for i, fh in enumerate(fh_list)] File "/Users/xiangbo/mtag/mtag/ldsc_mod/ldscore/parse.py", line 120, in annot_parser df_annot = df_annot[(.95 > df_frq.FRQ) & (df_frq.FRQ > 0.05)] File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 1958, in getitem return self._getitem_array(key) File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/frame.py", line 1998, in _getitem_array key = check_bool_indexer(self.index, key) File "/Users/xiangbo/anaconda2/lib/python2.7/site-packages/pandas/core/indexing.py", line 1939, in check_bool_indexer raise IndexingError('Unalignable boolean Series provided as ' pandas.core.indexing.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match

Please give some advice. Thank you very much!

Bo

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bulik/ldsc/issues/103, or mute the thread https://github.com/notifications/unsubscribe-auth/AILEvcPzMrizlkD90T18uwgKYPTWLH5bks5tRl9FgaJpZM4R40zm.

xiangboyulan commented 6 years ago

Hi Raymond,

Based on your advice, I have solved the errors. Thank you so much!

Bo