omerwe
commented
4 years ago Thanks Brian, great idea!!! I'll test this soon and will merge this if there's no problem
On Mon, Dec 9, 2019 at 4:08 PM Brian M. Schilder notifications@github.com wrote:
Hey Omer,
I've been looking for ways to save space on my computer when using the UKB LD files, so I extended the load_ld() function in run_finemapper.py to be able to download LD directly from the alkesgroup server.
I also added a helper function find_ld_prefix() to identify which LD file you need based on some info.
Just wanted to share these in case you found them helpful to integrate into PolyFun.
Best, Brian
import numpy as np import pandas as pd import logging import os import scipy.sparse as sparse from io import BytesIO import requests
def find_ld_prefix(chrom, min_pos):
min_pos = 40590585; chrom=12
alkes_url = "https://data.broadinstitute.org/alkesgroup/UKBB_LD" bp_starts = list(range(1, 252000001, 1000000)) bp_ends = [x+3000000 for x in bp_starts] i = [i for i,x in enumerate(bp_starts) if x>=min_pos][0] prefix = "chr%d_%d_%d" % (chrom, bp_starts[i], bp_ends[i]) ld_prefix = os.path.join(alkes_url, prefix) return ld_prefixdef load_ld(ld_prefix, server=True):
ld_prefix="https://data.broadinstitute.org/alkesgroup/UKBB_LD/chr12_41000001_44000001"
# load SNPs info snps_filename_parquet = ld_prefix + '.parquet' snps_filename_gz = ld_prefix + '.gz' if os.path.exists(snps_filename_parquet) | server==False: df_ld_snps = pd.read_parquet(snps_filename_parquet) elif os.path.exists(snps_filename_gz) | server==True: df_ld_snps = pd.read_table(snps_filename_gz, delim_whitespace=True) else: raise ValueError('couldn\'t find SNPs file %s or %s' % (snps_filename_parquet, snps_filename_gz)) # load LD matrix R_filename = ld_prefix + '.npz' if server: print("Reading .npz file from alkesgroup server") r = requests.get(R_filename, stream=True) R = sparse.load_npz(BytesIO(r.raw.read())).toarray() else: if not os.path.exists(R_filename): raise IOError('%s not found' % (R_filename)) R = sparse.load_npz(R_filename).toarray() R = R + R.T assert np.allclose(np.diag(R), 1.0) logging.info('Loaded an LD matrix for %d SNPs from %s' % (R.shape[0], R_filename)) # sanity checks assert R.shape[0] == R.shape[1] if R.shape[0] != df_ld_snps.shape[0]: raise ValueError('LD matrix has a different number of SNPs than the SNPs file') return R, df_ld_snps— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/omerwe/polyfun/issues/17?email_source=notifications&email_token=ACNCB4ZL56IPMLVDX3PZ3ZDQX2XT7A5CNFSM4JYSH4PKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H7IOXOA, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNCB46MOCE2VMURFI7UXNDQX2XT7ANCNFSM4JYSH4PA .
bschilder
cmlakhan
Hey Omer,
I've been looking for ways to save space on my computer when using the UKB LD files, so I extended the
load_ld()function inrun_finemapper.pyto be able to download LD directly from the alkesgroup server.I also added a helper function
find_ld_prefix()to identify which LD file you need based on some info.Just wanted to share these in case you found them helpful to integrate into PolyFun.
Best, Brian