search

limix / bgen-reader-py

A BGEN file format reader.
MIT License
10 stars 3 forks source link

Can't open bgen using symbolic links #42

Open jordanero opened 2 years ago

jordanero commented 2 years ago

Hello,

I want to use the numpy inspired API to open an bgen using a symbolic link. I want to do this because I don't want to create the .metadata2.mmm file in the shared directory where the bgen lives. I thought I recalled being able to do this in a previous version of bgen-reader, but am running into the following problem now.

from bgen_reader import open_bgen wha = open_bgen('/n/holylfs05/LABS/liang_lab_l3/Lab/jordan/ssctpr/data/ukbb_symbolic_link/ukb_imp_chr1_v3.bgen') Traceback (most recent call last): File "", line 1, in File "/n/home_fasse/jrossen/.conda/envs/ssctpr2/lib/python3.9/site-packages/bgen_reader/_bgen2.py", line 133, in init assert_file_exist(filepath) File "/n/home_fasse/jrossen/.conda/envs/ssctpr2/lib/python3.9/site-packages/bgen_reader/_file.py", line 14, in assert_file_exist raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), filepath)

I don't have a very strong understanding of how PosixPath works, but after a little playing around, it seems like PosixPath precludes the use of symbolic links.

Either allowing for the use of symbolic links, or allowing the user to specify a directory where .metadata2.mmm should be created would solve my problem completely.

Thanks for making this software!

CarlKCarlK commented 2 years ago

Jordan,

Thanks for using Bgen-Reader!

I'll be able to look into this tomorrow. I wonder if you have write access to the *.bgen file's directory?

In any case, adding an option to put the *.mmm file in any directory seems like the right answer and that is likely what we'll do.

From: Jordan Rossen @.> Sent: Tuesday, June 07, 2022 7:58 AM To: limix/bgen-reader-py @.> Cc: Subscribed @.***> Subject: [limix/bgen-reader-py] Can't open bgen using symbolic links (Issue #42)

Hello,

I want to use the numpy inspired API to open an bgen using a symbolic link. I want to do this because I don't want to create the .metadata2.mmm file in the shared directory where the bgen lives. I thought I recalled being able to do this in a previous version of bgen-reader, but am running into the following problem now.

from bgen_reader import open_bgen wha = open_bgen('/n/holylfs05/LABS/liang_lab_l3/Lab/jordan/ssctpr/data/ukbb_symbolic_link/ukb_imp_chr1_v3.bgen') Traceback (most recent call last): File "", line 1, in File "/n/home_fasse/jrossen/.conda/envs/ssctpr2/lib/python3.9/site-packages/bgen_reader/_bgen2.py", line 133, in init assert_file_exist(filepath) File "/n/home_fasse/jrossen/.conda/envs/ssctpr2/lib/python3.9/site-packages/bgen_reader/_file.py", line 14, in assert_file_exist raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), filepath)

I don't have a very strong understanding of how PosixPath works, but after a little playing around, it seems like PosixPath precludes the use of symbolic links.

Either allowing for the use of symbolic links, or allowing the user to specify a directory where .metadata2.mmm should be created would solve my problem completely.

Thanks for making this software!

- Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flimix%2Fbgen-reader-py%2Fissues%2F42&data=05%7C01%7C%7Cf122f3122a244133143308da48962d84%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637902107101785825%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Csmdm8wELv3aKeKlHLljUI4eIs5fH8UKEJFh2SCk9Kc%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABR65P4NZAC4U3P7ZSZZC3TVN5PRFANCNFSM5YDIGANA&data=05%7C01%7C%7Cf122f3122a244133143308da48962d84%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637902107101785825%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZEqkxE39TrV6eWCALOmeeG4cTCjb5EY4ZbchudfhbGg%3D&reserved=0. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

CarlKCarlK commented 2 years ago

Jordan,

Here is a possible temporary fix.

Uninstall bgen-reader (for example, pip uninstall bgen-reader)

Then install my private version from GitHub: pip install @.***

Then you should be able to use a new 'metadata_filepath' in open_bgen. For example:

filepath = example_filepath("example.32bits.bgen")
with open_bgen(
    filepath, metadata_filepath=r"m:\deldir\cmk.mmm", verbose=False
) as bgen2:
    for num_threads in [1, 2]:
        for slice in [np.s_[:, :], np.s_[:, []]]:
            val = bgen2.read(index=slice, num_threads=num_threads)
            row_count = len(bgen2.samples[slice[0]])
            col_count = len(bgen2.ids[slice[1]])
            assert val.shape == (row_count, col_count, 3)

Please, let me know if this works for you. In any case, I'll have time tomorrow to work on it.

From: Carl K Sent: Tuesday, June 07, 2022 8:14 AM To: limix/bgen-reader-py @.>; limix/bgen-reader-py @.> Cc: Subscribed @.***> Subject: RE: [limix/bgen-reader-py] Can't open bgen using symbolic links (Issue #42)

Jordan,

Thanks for using Bgen-Reader!

I'll be able to look into this tomorrow. I wonder if you have write access to the *.bgen file's directory?

In any case, adding an option to put the *.mmm file in any directory seems like the right answer and that is likely what we'll do.

From: Jordan Rossen @.**@.>> Sent: Tuesday, June 07, 2022 7:58 AM To: limix/bgen-reader-py @.**@.>> Cc: Subscribed @.**@.>> Subject: [limix/bgen-reader-py] Can't open bgen using symbolic links (Issue #42)

Hello,

I want to use the numpy inspired API to open an bgen using a symbolic link. I want to do this because I don't want to create the .metadata2.mmm file in the shared directory where the bgen lives. I thought I recalled being able to do this in a previous version of bgen-reader, but am running into the following problem now.

from bgen_reader import open_bgen wha = open_bgen('/n/holylfs05/LABS/liang_lab_l3/Lab/jordan/ssctpr/data/ukbb_symbolic_link/ukb_imp_chr1_v3.bgen') Traceback (most recent call last): File "", line 1, in File "/n/home_fasse/jrossen/.conda/envs/ssctpr2/lib/python3.9/site-packages/bgen_reader/_bgen2.py", line 133, in init assert_file_exist(filepath) File "/n/home_fasse/jrossen/.conda/envs/ssctpr2/lib/python3.9/site-packages/bgen_reader/_file.py", line 14, in assert_file_exist raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), filepath)

I don't have a very strong understanding of how PosixPath works, but after a little playing around, it seems like PosixPath precludes the use of symbolic links.

Either allowing for the use of symbolic links, or allowing the user to specify a directory where .metadata2.mmm should be created would solve my problem completely.

Thanks for making this software!

- Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flimix%2Fbgen-reader-py%2Fissues%2F42&data=05%7C01%7C%7Cf122f3122a244133143308da48962d84%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637902107101785825%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Csmdm8wELv3aKeKlHLljUI4eIs5fH8UKEJFh2SCk9Kc%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABR65P4NZAC4U3P7ZSZZC3TVN5PRFANCNFSM5YDIGANA&data=05%7C01%7C%7Cf122f3122a244133143308da48962d84%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637902107101785825%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZEqkxE39TrV6eWCALOmeeG4cTCjb5EY4ZbchudfhbGg%3D&reserved=0. You are receiving this because you are subscribed to this thread.Message ID: @.**@.>>

jordanero commented 2 years ago

Was there some censoring done by github?

Your message reads:

Then install my private version from GitHub: pip install *@****

CarlKCarlK commented 2 years ago

Try to send me an email at carlk at msn dot com

From: Jordan Rossen @.> Sent: Tuesday, June 7, 2022 12:56:17 PM To: limix/bgen-reader-py @.> Cc: Carl Kadie @.>; Comment @.> Subject: Re: [limix/bgen-reader-py] Can't open bgen using symbolic links (Issue #42)

Was there some censoring done by github?

Your message reads:

Then install my private version from GitHub: pip install @.***

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flimix%2Fbgen-reader-py%2Fissues%2F42%23issuecomment-1149103523&data=05%7C01%7C%7Ca555abedcb7a432b3c6008da48bfc928%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637902285793338402%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=I6THS0WcwMla6ai9kptgosquMOYZv2mHD2gkfxoS2Rg%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABR65P2674PWFAWQLXQUKODVN6SODANCNFSM5YDIGANA&data=05%7C01%7C%7Ca555abedcb7a432b3c6008da48bfc928%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637902285793495230%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fPWoQkzIYk4LvPEiGfOzjZgpfoEMO6tsGfUVVs23dZk%3D&reserved=0. You are receiving this because you commented.Message ID: @.***>

CarlKCarlK commented 2 years ago

Addressed by https://github.com/limix/bgen-reader-py/pull/44

jordanero commented 2 years ago

Hi Carl, the update is working great! Sorry for taking so long to try it out. I'm able to open a bgen with a metadata file in a different directory, and reuse that metadata file. Thanks a lot.

CarlKCarlK commented 2 years ago

Jordan,

That's great! (We hope to have this feature in the main version of the package soon).

From: Jordan Rossen @.> Sent: Friday, August 05, 2022 2:19 PM To: limix/bgen-reader-py @.> Cc: Carl Kadie @.>; Comment @.> Subject: Re: [limix/bgen-reader-py] Can't open bgen using symbolic links (Issue #42)

Hi Carl, the update is working great! Sorry for taking so long to try it out. I'm able to open a bgen with a metadata file in a different directory, and reuse that metadata file. Thanks a lot.

- Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flimix%2Fbgen-reader-py%2Fissues%2F42%23issuecomment-1206875128&data=05%7C01%7C%7Ccd9a0494289e4895f45b08da77281172%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637953311219575762%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wFgKGGCA2hPzgiILJJUw26RcbjdkqAXbpLlOo8ydFYs%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABR65P3MFC5ORSCMVEA2A43VXWAK7ANCNFSM5YDIGANA&data=05%7C01%7C%7Ccd9a0494289e4895f45b08da77281172%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637953311219575762%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=E%2Fb5D1HBmOq1y%2BIZVz%2BVt02Ou6uUjbkud%2B513eFlpiY%3D&reserved=0. You are receiving this because you commented.Message ID: @.**@.>>