search

fair-research / bdbag

Big Data Bag Utilities
https://fair-research.org
Apache License 2.0
48 stars 23 forks source link

JSON FILE CONFIGURATION #61

Open errrna opened 3 days ago

errrna commented 3 days ago

keychain.json

We are trying to download the Velmeshev files (https://assets.nemoarchive.org/dat-3ah9h9x), using BDBag. We created a google cloud account, but are having issues with the json configuration (attached). Is the attached json file correct, and if not, can you assist with configuring the json file correctly.

mikedarcy commented 3 days ago

The attached keychain.json file just looks like the sample configuration file. You need to add an appropriate keychain entry for the GCS bucket you are trying to access. Have a look at the GCS support release notes for some more info on GCS support and how to configure the keychain. https://github.com/fair-research/bdbag/blob/master/CHANGELOG.md#added-google-cloud-storage-fetch-handler-for-handling-gs-urls-in-fetchtxt

mikedarcy commented 3 days ago

Note that you have to install the GCloud CLI https://cloud.google.com/sdk/docs/install and configure it properly to access your bucket. If you can access the files via GCloud CLI, the bdbag GCS fetch should also work if you have a matching keychain entry for your bucket.

errrna commented 2 days ago

hi Mike Thank you for taking the time to respond to my issue. We cannot seem to locate the URI for the project (https://assets.nemoarchive.org/dat-b3brzfa). Can you let us know where we can find the "gs:// ... " location? we looked through the NEMO website and cannot find it for the Velmeshev dataset. Also, is project ID something we make, or is it an ID connected to the dataset on NEMO?

{ "uri": "gs://gcs-bdbag-integration-testing/", "auth_type": "gcs-credentials", "auth_params": { "project_id": "bdbag-204999", "allow_requester_pays": false } }

mikedarcy commented 2 days ago

Apologies, but this isn't really a bdbag issue. This issue is that you need to understand how to consume the bag from the bag provider, in this case, NEMO. They should have documentation on how to go about consuming their content.

Anyway, I downloaded the bag reference by the URL at this identifier page: https://assets.nemoarchive.org/dat-3ah9h9x. It appears to be a bag containing two other bags, with each of those bags containing fetch references to GCS locations. So you need to first materialize the wrapping bag, then try to materialize each of the child bags. If you look in the fetch.txt of the nested bags, you can see the GS URLs being used. From the information therein, I created a keychain.json entry like this:

    {
        "uri": "gs://devhu-sn-10x/",
        "auth_type": "gcs-credentials",
        "auth_params": {
            "project_id": "<my project id>",
            "allow_requester_pays": false
         }
    }

You will need to install GCS, as I mentioned. The project_id is the project ID from your GCS account that you are using to access the GCS buckets. This is something you provide when you setup the GCLoud CLI. You need to install the GCloud CLI, run gcloud init, provide your project id, then run gcloud auth application-default login. Then you can try to materialize the nested bags.

Here's a transcript of what I did when trying to materialize this bag and it's nested contents. Ultimately, I could not get the files from the GCS bucket due to permissions issues, which is something that would be have to be requested of the bucket owner. I suggest you contact someone at NEMO to get further help with your issues consuming their data.

mdarcy@ThunderDome:Downloads$ bdbag --materialize https://data.nemoarchive.org/publication_release/Velmeshev_devhu_sn.tgz

2024-06-28 11:47:06,468 - INFO - Attempting GET from URL: https://data.nemoarchive.org/publication_release/Velmeshev_devhu_sn.tgz
2024-06-28 11:47:07,078 - INFO - File [/mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn.tgz] transfer complete. 1.521 KB transferred. Elapsed time: 0:00:00.065790.
2024-06-28 11:47:07,099 - INFO - Extracting TAR/GZ/BZ2/XZ archived file: /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn.tgz
2024-06-28 11:47:07,106 - INFO - Target path /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn already exists, moving it to /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn-2024-06-28_11.47.07
2024-06-28 11:47:07,200 - INFO - File /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn.tgz was successfully extracted to directory /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn
2024-06-28 11:47:07,379 - INFO - Checking payload consistency. This can take some time for large bags with many payload files...
2024-06-28 11:47:07,451 - INFO - Attempting to resolve remote file references from /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/fetch.txt.
2024-06-28 11:47:07,460 - INFO - Attempting GET from URL: https://data.nemoarchive.org/publication_release/Velmeshev_devhu_sn/Analysis_bag_1_Velmeshev_devhu_sn_Analysis.tgz
2024-06-28 11:47:08,281 - INFO - File [/mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/data/Analysis_bag_1_Velmeshev_devhu_sn_Analysis.tgz] transfer complete. 6.383 KB transferred. Elapsed time: 0:00:00.372394.
2024-06-28 11:47:08,295 - INFO - Attempting GET from URL: https://data.nemoarchive.org/publication_release/Velmeshev_devhu_sn/Raw_data_bag_1_Velmeshev_devhu_sn_Raw.tgz
2024-06-28 11:47:08,741 - INFO - File [/mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/data/Raw_data_bag_1_Velmeshev_devhu_sn_Raw.tgz] transfer complete. 39.526 KB transferred. Elapsed time: 0:00:00.366857.
2024-06-28 11:47:08,744 - INFO - Fetch complete. Elapsed time: 0:00:01.292071
2024-06-28 11:47:08,745 - INFO - Validating bag: /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn
2024-06-28 11:47:09,045 - INFO - Verifying checksum for file /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/data/Analysis_bag_1_Velmeshev_devhu_sn_Analysis.tgz
2024-06-28 11:47:09,053 - INFO - Verifying checksum for file /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/data/Raw_data_bag_1_Velmeshev_devhu_sn_Raw.tgz
2024-06-28 11:47:09,060 - INFO - Verifying checksum for file /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/bagit.txt
2024-06-28 11:47:09,062 - INFO - Verifying checksum for file /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/manifest-sha256.txt
2024-06-28 11:47:09,065 - INFO - Verifying checksum for file /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/fetch.txt
2024-06-28 11:47:09,067 - INFO - Verifying checksum for file /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/bag-info.txt
2024-06-28 11:47:09,071 - INFO - Verifying checksum for file /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/manifest-md5.txt
2024-06-28 11:47:09,074 - INFO - Bag /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn is valid

mdarcy@ThunderDome:Downloads$ ll Velmeshev_devhu_sn/data/
total 48
-rwxrwxrwx 1 mdarcy mdarcy  6536 Jun 28 11:47 Analysis_bag_1_Velmeshev_devhu_sn_Analysis.tgz
-rwxrwxrwx 1 mdarcy mdarcy 40475 Jun 28 11:47 Raw_data_bag_1_Velmeshev_devhu_sn_Raw.tgz

mdarcy@ThunderDome:Downloads$ bdbag Velmeshev_devhu_sn/data/Analysis_bag_1_Velmeshev_devhu_sn_Analysis.tgz

2024-06-28 11:48:58,589 - INFO - Extracting TAR/GZ/BZ2/XZ archived file: /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/data/Analysis_bag_1_Velmeshev_devhu_sn_Analysis.tgz
2024-06-28 11:48:58,675 - INFO - File /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/data/Analysis_bag_1_Velmeshev_devhu_sn_Analysis.tgz was successfully extracted to directory /mnt/c/Users/mdarcy/Downloads/Velmeshev_devhu_sn/data/Analysis_bag_1_Velmeshev_devhu_sn_Analysis

mdarcy@ThunderDome:Downloads$ cat Velmeshev_devhu_sn/data/Analysis_bag_1_Velmeshev_devhu_sn_Analysis/fetch.txt 
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1105_BA22.bam    22891921327     data/1105_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1158_BA22.bam    13659444779     data/1158_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1443_BA22.bam    28147405475     data/1443_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1453_BA24.bam    16946892114     data/1453_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1453_INS.bam     14704541577     data/1453_INS.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1547_BA24.bam    19881841053     data/1547_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1738_BA22.bam    9425237267      data/1738_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1790_BA22.bam    14622420199     data/1790_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1791_BA22.bam    12455282740     data/1791_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1791_BA24.bam    20181629872     data/1791_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1798_BA24.bam    15150141922     data/1798_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1798_BA9.bam     13196620031     data/1798_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1798_INS.bam     13864058592     data/1798_INS.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1823_BA24.bam    18170854531     data/1823_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1864_BA22.bam    9551408213      data/1864_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4283_BA13.bam    17689084971     data/4283_BA13.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4283_BA22.bam    16712491134     data/4283_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4283_BA9.bam     15927452383     data/4283_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4293_BA24.bam    15344319223     data/4293_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4332_BA22.bam    25769189781     data/4332_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4337_BA22.bam    21896512740     data/4337_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4341_BA24.bam    19250129958     data/4341_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4341_BA46.bam    18013357482     data/4341_BA46.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4350_BA24.bam    16289986034     data/4350_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4350_BA9.bam     30475986789     data/4350_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4369_BA22.bam    12663629830     data/4369_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4369_BA24.bam    16250111302     data/4369_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4369_BA9.bam     33724446608     data/4369_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4369_INS.bam     18321348261     data/4369_INS.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4389_BA24.bam    18108137496     data/4389_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4389_BA9.bam     30309450441     data/4389_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4396_BA22.bam    20139980930     data/4396_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4396_BA24.bam    16151676619     data/4396_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4396_BA9.bam     39551267466     data/4396_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4396_INS.bam     20179727154     data/4396_INS.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4425_BA22.bam    14617325052     data/4425_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4426_BA24.bam    16815151275     data/4426_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4426_BA9.bam     25508740575     data/4426_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4670_BA22.bam    22790960446     data/4670_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4670_BA24.bam    17750531352     data/4670_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4942_BA13.bam    13103254209     data/4942_BA13.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4942_BA22.bam    14786263981     data/4942_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/4942_BA9.bam     18781858125     data/4942_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5077_BA22.bam    8528256327      data/5077_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5163_BA24.bam    16321923152     data/5163_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5242_BA24.bam    20110701353     data/5242_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5242_FIC.bam     25943343767     data/5242_FIC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5387_BA9.bam     20077887606     data/5387_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5391_BA24.bam    33171099478     data/5391_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5408_FIC.bam     27099933562     data/5408_FIC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5408_PFC.bam     27745611129     data/5408_PFC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5538_PFC.bam     25878594203     data/5538_PFC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5546_BA9.bam     13027713077     data/5546_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5554_BA24.bam    19219568197     data/5554_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5577_BA9.bam     18282962280     data/5577_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5608_BA24.bam    16307537171     data/5608_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5608_BA9.bam     33002074954     data/5608_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5608_INS.bam     15973858595     data/5608_INS.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5609_BA9.bam     20419756840     data/5609_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5787_BA9.bam     23166013986     data/5787_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5879_PFC.bam     23123718658     data/5879_PFC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5893_BA24.bam    19865924776     data/5893_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5893_BA9.bam     18372731826     data/5893_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5893_PFC.bam     24764454299     data/5893_PFC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5900_BA13.bam    22181031141     data/5900_BA13.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5900_BA22.bam    17646117643     data/5900_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5900_BA22_2.bam  22657703404     data/5900_BA22_2.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5900_BA24.bam    20325486444     data/5900_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5900_BA9.bam     15932012763     data/5900_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5936_PFC.bam     37301127255     data/5936_PFC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5958_BA24.bam    18881026707     data/5958_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5958_BA9.bam     20704954856     data/5958_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5976_BA9.bam     16729593995     data/5976_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/5981_BA9.bam     11966207699     data/5981_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/6032_BA24.bam    21744191960     data/6032_BA24.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/6032_FIC.bam     26364294482     data/6032_FIC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/663_BA22.bam     16165865117     data/663_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/779_BA22.bam     13850154910     data/779_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/779_BA9.bam      12352285126     data/779_BA9.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/814_BA22.bam     11515737585     data/814_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/872_BA13.bam     13920013541     data/872_BA13.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/872_BA22.bam     12490592852     data/872_BA22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/A14-103_Cing.bam 18802360490     data/A14-103_Cing.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/A14-103_FC.bam   31966954535     data/A14-103_FC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/A15-44_Cing.bam  13849506448     data/A15-44_Cing.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/A15-44_FC.bam    31384724358     data/A15-44_FC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/counts/ARK_postnatal.barcodes.tsv      13567804        data/ARK_postnatal.barcodes.tsv
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/counts/ARK_postnatal.genes.tsv 1663028 data/ARK_postnatal.genes.tsv
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/counts/ARK_postnatal.matrix.mtx        10690577251     data/ARK_postnatal.matrix.mtx
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW16-2-12-20_CGE.bam     51907170041     data/GW16-2-12-20_CGE.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW16-2-12-20_CGE_2.bam   47742875437     data/GW16-2-12-20_CGE_2.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW16-2-12-20_LGE.bam     69370429789     data/GW16-2-12-20_LGE.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW16-2-12-20_MGE.bam     63479087418     data/GW16-2-12-20_MGE.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW16-2-12-20_MGE_2.bam   53689361588     data/GW16-2-12-20_MGE_2.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW16-2-2-20_PFC.bam      43983642260     data/GW16-2-2-20_PFC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW16-2-2-20_temp.bam     66779677627     data/GW16-2-2-20_temp.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-6-14-19_CGE.bam     71908639774     data/GW18-6-14-19_CGE.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-6-14-19_CGE_2.bam   57046094083     data/GW18-6-14-19_CGE_2.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-6-14-19_LGE.bam     69287259734     data/GW18-6-14-19_LGE.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-6-14-19_MGE.bam     51619404078     data/GW18-6-14-19_MGE.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-6-14-19_MGE_2.bam   59452203325     data/GW18-6-14-19_MGE_2.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-6-14-19_PFC.bam     36832638542     data/GW18-6-14-19_PFC.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-6-14-19_cing.bam    53139529795     data/GW18-6-14-19_cing.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-6-14-19_temp.bam    74522467281     data/GW18-6-14-19_temp.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-CGE-3-22-17.bam     22788191823     data/GW18-CGE-3-22-17.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-LGE-3-22-17.bam     22158914462     data/GW18-LGE-3-22-17.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW18-MGE-3-22-17.bam     22997546389     data/GW18-MGE-3-22-17.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW19-1-17-20_GE.bam      42071050792     data/GW19-1-17-20_GE.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW20-ACC-12-7-18.bam     23937674087     data/GW20-ACC-12-7-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW20-PFC-11-28-18.bam    23042122773     data/GW20-PFC-11-28-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW20-PFC-12-7-18.bam     24127338474     data/GW20-PFC-12-7-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW20-cing-11-28-18.bam   21995915552     data/GW20-cing-11-28-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW20-temp-11-28-18.bam   20740373137     data/GW20-temp-11-28-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW25-PFC-11-17-17.bam    24257764701     data/GW25-PFC-11-17-17.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW27-PFC-2-7-18.bam      23640593663     data/GW27-PFC-2-7-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW27-STG-2-7-18.bam      22731232308     data/GW27-STG-2-7-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW27-cing-2-7-18.bam     22736813267     data/GW27-cing-2-7-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW30-PFC-11-6-18.bam     23564137277     data/GW30-PFC-11-6-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW30-cing-11-6-18.bam    20434350557     data/GW30-cing-11-6-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/GW30-temp-11-6-18.bam    20683429674     data/GW30-temp-11-6-18.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/SM10-C23.bam     19993726763     data/SM10-C23.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/SM11-C22.bam     14366792047     data/SM11-C22.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/SM12-C20.bam     14379940232     data/SM12-C20.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/SM4-C20.bam      21354240427     data/SM4-C20.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/SM5-C19.bam      21900815370     data/SM5-C19.bam
gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/SM6-C19.bam      19626005226     data/SM6-C19.bam

At this point I tried to materialize one of the nested bags, but as I mentioned I got access denied errors. 

bdbag --materialize Velmeshev_devhu_sn/data/Analysis_bag_1_Velmeshev_devhu_sn_Analysis

2024-06-28 12:51:07,264 - INFO - Checking payload consistency. This can take some time for large bags with many payload files...
2024-06-28 12:51:07,367 - INFO - Attempting to resolve remote file references from C:\Users\mdarcy\Downloads\Velmeshev_devhu_sn\data\Analysis_bag_1_Velmeshev_devhu_sn_Analysis\fetch.txt.
2024-06-28 12:51:08,742 - INFO - Attempting GET from URL: gs://devhu-sn-10x/kriegstein_sn_10xv2/trancriptome/sncell/10x_v2/human/processed/align/1105_BA22.bam with project_id=bdbag-123456 and allow_requester_pays=False
2024-06-28 12:51:09,687 - ERROR - [Forbidden] 403 GET https://storage.googleapis.com/download/storage/v1/b/devhu-sn-10x/o/kriegstein_sn_10xv2%2Ftrancriptome%2Fsncell%2F10x_v2%2Fhuman%2Fprocessed%2Falign%2F1105_BA22.bam?alt=media: mdarcy.isi.usc@gmail.com does not have storage.objects.get access to the Google Cloud Storage object. Permission &#39;storage.objects.get&#39; denied on resource (or it may not exist).: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

Sorry I cannot help you any more beyond this. If you can get this far and get the same errors, please contact NEMO and request help from them.

errrna commented 2 days ago

ok thank you Mike. I really appreciate your help.