eric-czech
commented
4 years ago I tried running conversions on preemptible VMs to see if we can save some money using those, but this experiment didn't go well. Here are some of the issues I ran into:
- I intermittently get Zarr errors relating to a store already existing when it definitely does not. I wasn't able to reproduce this on another VM -- only when using the preemptible GKE nodes. About half of the jobs I ran today hit this.
ValueError: path 'variant_id' contains an array
Environment for envs/io.yaml created (location: .snakemake/conda/a0f8b672) Using shell: /bin/bash Provided cores: 32 Rules claiming more threads will be scaled down. Job counts: count jobs 1 bgen_to_zarr 1 [Fri Oct 9 15:42:47 2020] rule bgen_to_zarr: input: rs-ukb/raw-data/gt-imputation/ukb_imp_chr21_v3.bgen, rs-ukb/raw-data/gt-imputation/ukb_mfi_chr21_v3.txt, rs-ukb/raw-data/gt-imputation/ukb59384_imp_chr21_v3_s487296.sample output: rs-ukb/prep-data/gt-imputation/ukb_chr21.ckpt jobid: 0 wildcards: bgen_contig=21 threads: 30 resources: mem_mb=124072 Downloading from remote: rs-ukb/raw-data/gt-imputation/ukb59384_imp_chr21_v3_s487296.sample Finished download. Downloading from remote: rs-ukb/raw-data/gt-imputation/ukb_imp_chr21_v3.bgen Finished download. Downloading from remote: rs-ukb/raw-data/gt-imputation/ukb_mfi_chr21_v3.txt Finished download. Activating conda environment: /workdir/.snakemake/conda/a0f8b672 2020-10-09 15:50:05,248 | __main__ | INFO | Loading BGEN dataset for contig Contig(name=21, index=20) from rs-ukb/raw-data/gt-imputation/ukb_imp_chr21_v3.bgen (chunks = (250, -1)) 2020-10-09 15:50:30,683 | __main__ | INFO | Rechunking dataset for contig Contig(name=21, index=20) to rs-ukb/prep-data/gt-imputation/ukb_chr21.zarr (chunks = (5216, 5792)) Traceback (most recent call last): File "scripts/convert_genetic_data.py", line 309, in- One of the jobs failed with this error relating to the GCE metadata service not being available.
google.auth.exceptions.RefreshError: Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/452178172058-compute@developer.gserviceaccount.com/?recursive=true from the Google Compute Enginemetadata service. Compute Engine Metadata server unavailable
- About halfway through a rechunking snakemake failed a job due possibly to another issue with GCE metadata services.
Traceback (most recent call last):
File "/opt/conda/envs/snakemake/lib/python3.8/site-packages/snakemake/__init__.py", line 490, in snakemake
_default_remote_provider = rmt.RemoteProvider(
File "/opt/conda/envs/snakemake/lib/python3.8/site-packages/snakemake/remote/GS.py", line 93, in __init__
self.client = storage.Client(*args, **kwargs)
File "/opt/conda/envs/snakemake/lib/python3.8/site-packages/google/cloud/storage/client.py", line 110, in __init__
super(Client, self).__init__(
File "/opt/conda/envs/snakemake/lib/python3.8/site-packages/google/cloud/client.py", line 249, in __init__
_ClientProjectMixin.__init__(self, project=project)
File "/opt/conda/envs/snakemake/lib/python3.8/site-packages/google/cloud/client.py", line 203, in __init__
raise EnvironmentError(
OSError: Project was not passed and could not be determined from the environment.I'm not going to experiment any more with these.
Stats for conversion of chr 21 and 22:
Compute Cost
Approximate cost for conversion of chr21 and 22 = 16.5 hrs $0.38 = $6.27. Extrapolating to all chromosomes: $6.27 (97M / (1.26M + 1.25M)) = $242.
Storage Cost
Estimated storage costs (see https://cloud.google.com/storage/pricing) = $0.026 per GB 2.32 TiB raw bgen 1.2 size inflation (https://github.com/related-sciences/ukb-gwas-pipeline-nealelab/issues/7) = .026 2550.867 1.2 = $79.58 monthly.