Error: No such file or directory: 'pheno.phenotypes_and_permutations'

[viannam]

Hi Voichek, I am running kmers_gwas.py and getting an error. I would appreciate it if you could help.

My loads: module load kmc/3.1.1 module load R/4.1 module load gemma/0.98.1 module load gcc/9.3.0 module load bamtools

My command: python2.7 kmers_gwas.py --pheno GER_21.pheno --kmers_table ./kmers_table -l 31 -p 8 -k 100000 --dont_remove_intermediates --maf 0.01 --pattern_counter --outdir ./GER_21 All input files exist and are listed in the same folder.

The error: Error in read.table(file = file, header = header, sep = sep, quote = quote, : no lines available in input Calls: as.matrix -> read.csv -> read.table Execution halted Traceback (most recent call last): File "kmers_gwas.py", line 278, in main() File "kmers_gwas.py", line 107, in main phenotypes_names = file(paths["pheno_permuted_fn"] ,"r").read().split("\n")[0].split("\t")[1:] IOError: [Errno 2] No such file or directory: './GER_21/pheno.phenotypes_and_permutations'

[voichek]

Dear Mariana,

1. Are the names in the phenotype file written the same as in the k-mer table?
2. If you send me the output folder of the failed run, I might be able to understand what went wrong.

Best, Yoav

[viannam]

Dear @voichek,

Thank you for getting back to me. My ID names on accetion_ID of the phenotype file were between quotes. I fixed it and reran it. However, I got another error. Idk if you preferer that I open another issue, but here it is: Error file: /blue/mresende/share/viannam/Kmers/external_programs/gemma_0_96: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory Traceback (most recent call last): File "kmers_gwas.py", line 281, in main() File "kmers_gwas.py", line 250, in main th_5per = get_threshold_from_perm(res, "P", args.n_permutations, 0.05) File "/blue/mresende/share/viannam/Kmers/src/py/functions.py", line 110, in get_threshold_from_perm pvals.append(best_pvals[prefix + str(i)]) KeyError: 'P1'

Output file: cat kmers_16471653.out Rscript /blue/mresende/share/viannam/Kmers/src/R/transform_and_permute_phenotypes.R ./germ_21/pheno.phenotypes ./germ_21/pheno.kinship 100 ./germ_21/pheno.phenotypes_and_permutations ./germ_21/pheno.phenotypes_permuted_transformed ./germ_21/EMMA_perm.log > ./germ_21/phenotypes_transformation_permutation.log affective MAF = 0.01

Files generated on the trait outdir: kmers log_file associate_kmers.log EMMA_perm.log pheno.phenotypes_and_permutations pheno.phenotypes_permuted_transformed pheno.kinship pheno.phenotypes phenotypes_transformation_permutation.log

Log file: head log_file Namespace(fn_phenotype='GER_21.pheno', gemma_path='/blue/mresende/share/viannam/Kmers/external_programs/gemma_0_96', kmers_len=31, kmers_pattern_counter=True, kmers_table='./kmers_table', mac=5, maf=0.01, min_data_points=30, n_extra_phenotype_kmers=None, n_kmers=100000, n_permutations=100, n_snps=10001, outdir='./germ_21', parallel=8, remove_intermediate=False, run_kmers=True, run_one_step_snps=False, run_two_steps_snps=False, snps_matrix=None, use_kinship_from_kmers=True)Multiple phenotypes found per accessions -> Averaging Using kinship calculated on k-mersRUN: python2.7 /blue/mresende/share/viannam/Kmers/src/py/align_kinship_phenotype.py --pheno ./germ_21/pheno.original_phenotypes --fam_file ./germ_21/pheno_for_accessions_order.fam --kinship_file ./kmers_table.kinship --output_pheno ./germ_21/pheno.phenotypes --output_kinship ./germ_21/pheno.kinship --DBs_list ./kmers_table.names RUN: Rscript /blue/mresende/share/viannam/Kmers/src/R/transform_and_permute_phenotypes.R ./germ_21/pheno.phenotypes ./germ_21/pheno.kinship 100 ./germ_21/pheno.phenotypes_and_permutations ./germ_21/pheno.phenotypes_permuted_transformed ./germ_21/EMMA_perm.log > ./germ_21/phenotypes_transformation_permutation.log RUN: mkdir ./germ_21/kmers RUN: /blue/mresende/share/viannam/Kmers/bin/associate_kmers -p ./germ_21/pheno.phenotypes_permuted_transformed -b pheno -o ./germ_21/kmers -n 100000 --parallel 8 --kmers_table ./kmers_table --kmer_len 31 --maf 0.010000 --mac 5 --pattern_counter 2> ./germ_21/associate_kmers.log We have 101 phenotypes RUN: mv ./germ_21/kmers/pheno.0.phenotype_value.fam ./germ_21/kmers/pheno.0.phenotype_value.fam.orig RUN: cat ./germ_21/pheno.phenotypes_and_permutations | tail -n +2 | awk '{print $1 " " $1 " 0 0 0 " $2}' > ./germ_21/kmers/pheno.0.phenotype_value.fam RUNG (1/1): /blue/mresende/share/viannam/Kmers/external_programs/gemma_0_96 -bfile ./germ_21/kmers/pheno.0.phenotype_value -lmm 2 -k ./germ_21/pheno.kinship -outdir ./germ_21/kmers/output -o phenotype_value -maf 0.010000 -miss 0.5 RUN: mv ./germ_21/kmers/pheno.1.P1.fam ./germ_21/kmers/pheno.1.P1.fam.orig

It should be 693 individuals. Does kmers_table.names file show the correct order of the genotypes in the kinship?

[voichek]

Dear Mariana,

In general, it is better to open a new topic for another problem. But let's close this one here.

It seems that you cannot run the gemma that comes with the library /blue/mresende/share/viannam/Kmers/external_programs/gemma_0_96.

1. You can try running it from the command line with no arguments to make sure this is the problem.
2. I saw before that you load gemma 0.98, you can tell the program to use your gemma (by just telling it the path to it is gemma, which is how I assume you are running it from the command line. However, if you want to use gemma 0.98, you need to change the files using a patch I made in the past.

Let me know if this helps, Yoav

[viannam]

Dear Voichek,

I cloned the library again (git clone https://github.com/voichek/kmersGWAS.git).

Those are the files and folders listed on my kmersGWAS directory:

associate_kmers
emma_kinship
filter_kmers
kmers_01.sh
kmers_gwas.py
kmers_table_to_bed
manual.pdf
test_germ (outdir) associate_snps
emma_kinship_kmers
GER_21.pheno
kmers_table.kinship
LICENSE
pheno
bin
examples
histogram_KMC_kmers_counts
kmers_table.names
list_kmers_found_in_multiple_samples
README.md
tmp build_kmers_table
external_programs
include
kmers_add_strand_information
kmers_table.table
Makefile
src

However, I got this error after 11 h running: /blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory

Should I load libgfortran.so.3 too? I am using HPC; should I ask them to install it, or does this package come from the kmersGWAS library? If it comes, is any file or folder missing from my kmerGWAS folder?

My current module loads: module load kmc/3.1.1 module load R/4.1 module load gcc/9.3.0 module load bamtools

And my command line again: python2.7 kmers_gwas.py --pheno GER_21.pheno --kmers_table kmers_table -l 31 -p 8 -k 100000 --dont_remove_intermediates --maf 0.01 --pattern_counter --outdir test_germ

I am including head and tail of the log file too: head log_file Namespace(fn_phenotype='./GER_21.pheno', gemma_path='/blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96', kmers_len=31, kmers_pattern_counter=True, kmers_table='./kmers_table', mac=5, maf=0.01, min_data_points=30, n_extra_phenotype_kmers=None, n_kmers=100000, n_permutations=100, n_snps=10001, outdir='./test_germ', parallel=8, remove_intermediate=False, run_kmers=True, run_one_step_snps=False, run_two_steps_snps=False, snps_matrix=None, use_kinship_from_kmers=True)Multiple phenotypes found per accessions -> Averaging Using kinship calculated on k-mersRUN: python2.7 /blue/mresende/share/viannam/Kmers/kmersGWAS/src/py/align_kinship_phenotype.py --pheno ./test_germ/pheno.original_phenotypes --fam_file ./test_germ/pheno_for_accessions_order.fam --kinship_file ./kmers_table.kinship --output_pheno ./test_germ/pheno.phenotypes --output_kinship ./test_germ/pheno.kinship --DBs_list ./kmers_table.names RUN: Rscript /blue/mresende/share/viannam/Kmers/kmersGWAS/src/R/transform_and_permute_phenotypes.R ./test_germ/pheno.phenotypes ./test_germ/pheno.kinship 100 ./test_germ/pheno.phenotypes_and_permutations ./test_germ/pheno.phenotypes_permuted_transformed ./test_germ/EMMA_perm.log > ./test_germ/phenotypes_transformation_permutation.log RUN: mkdir ./test_germ/kmers RUN: /blue/mresende/share/viannam/Kmers/kmersGWAS/bin/associate_kmers -p ./test_germ/pheno.phenotypes_permuted_transformed -b pheno -o ./test_germ/kmers -n 100000 --parallel 8 --kmers_table ./kmers_table --kmer_len 31 --maf 0.010000 --mac 5 --pattern_counter 2> ./test_germ/associate_kmers.log We have 101 phenotypes RUN: mv ./test_germ/kmers/pheno.0.phenotype_value.fam ./test_germ/kmers/pheno.0.phenotype_value.fam.orig RUN: cat ./test_germ/pheno.phenotypes_and_permutations | tail -n +2 | awk '{print $1 " " $1 " 0 0 0 " $2}' > ./test_germ/kmers/pheno.0.phenotype_value.fam RUNG (1/1): /blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96 -bfile ./test_germ/kmers/pheno.0.phenotype_value -lmm 2 -k ./test_germ/pheno.kinship -outdir ./test_germ/kmers/output -o phenotype_value -maf 0.010000 -miss 0.5 RUN: mv ./test_germ/kmers/pheno.1.P1.fam ./test_germ/kmers/pheno.1.P1.fam.orig [viannam@login10 test_germ]$ tail log_file RUNG (1/98): /blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96 -bfile ./test_germ/kmers/pheno.97.P97 -lmm 2 -k ./test_germ/pheno.kinship -outdir ./test_germ/kmers/output -o P97 -maf 0.010000 -miss 0.5 RUN: mv ./test_germ/kmers/pheno.98.P98.fam ./test_germ/kmers/pheno.98.P98.fam.orig RUN: cat ./test_germ/pheno.phenotypes_and_permutations | tail -n +2 | awk '{print $1 " " $1 " 0 0 0 " $100}' > ./test_germ/kmers/pheno.98.P98.fam RUNG (1/99): /blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96 -bfile ./test_germ/kmers/pheno.98.P98 -lmm 2 -k ./test_germ/pheno.kinship -outdir ./test_germ/kmers/output -o P98 -maf 0.010000 -miss 0.5 RUN: mv ./test_germ/kmers/pheno.99.P99.fam ./test_germ/kmers/pheno.99.P99.fam.orig RUN: cat ./test_germ/pheno.phenotypes_and_permutations | tail -n +2 | awk '{print $1 " " $1 " 0 0 0 " $101}' > ./test_germ/kmers/pheno.99.P99.fam RUNG (1/100): /blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96 -bfile ./test_germ/kmers/pheno.99.P99 -lmm 2 -k ./test_germ/pheno.kinship -outdir ./test_germ/kmers/output -o P99 -maf 0.010000 -miss 0.5 RUN: mv ./test_germ/kmers/pheno.100.P100.fam ./test_germ/kmers/pheno.100.P100.fam.orig RUN: cat ./test_germ/pheno.phenotypes_and_permutations | tail -n +2 | awk '{print $1 " " $1 " 0 0 0 " $102}' > ./test_germ/kmers/pheno.100.P100.fam RUNG (1/101): /blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96 -bfile ./test_germ/kmers/pheno.100.P100 -lmm 2 -k ./test_germ/pheno.kinship -outdir ./test_germ/kmers/output -o P100 -maf 0.010000 -miss 0.5

Please, let me know if you need more info to be able to help. Thank you,

[voichek]

Dear Mariana,

Can you clarify, you tried running just this command: /blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96 and it gave the error: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory after running for 11 hours?

Yoav

[viannam]

To clarify,

I ran this command: python2.7 kmers_gwas.py --pheno GER_21.pheno --kmers_table kmers_table -l 31 -p 8 -k 100000 --dont_remove_intermediates --maf 0.01 --pattern_counter --outdir test_germ

After copying the kmersGWAS library to my folder again. Now, gemma_0_96 is listed on external_programs folder: [viannam@login10 kmersGWAS]$ ls external_programs/ gemma_0_96 kmc_v3

I got this error:

/blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory Traceback (most recent call last): File "./kmers_gwas.py", line 278, in main() File "./kmers_gwas.py", line 247, in main th_5per = get_threshold_from_perm(res, "P", args.n_permutations, 0.05) File "/blue/mresende/share/viannam/Kmers/kmersGWAS/src/py/functions.py", line 110, in get_threshold_from_perm pvals.append(best_pvals[prefix + str(i)]) KeyError: 'P1'

I guess this message "/blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory" was repeated through all individuals and that is why it took 11 hours. But, there should also be 693 genotypes, and not just 101 as listed in the log file.

My genotypes ID (accession_ID column on the pheno file) should be in the same order as the kmers_table.names file?

[viannam]

If I load the modules like this: module load kmc/3.1.1 module load R/4.1 module load gcc/9.3.0 module load bamtools module load libgfortran

I get this error: Lmod is automatically replacing "R/4.1" with "libgfortran/3.0.0".

sh: Rscript: command not found Traceback (most recent call last): File "kmers_gwas.py", line 278, in main() File "kmers_gwas.py", line 107, in main phenotypes_names = file(paths["pheno_permuted_fn"] ,"r").read().split("\n")[0].split("\t")[1:] IOError: [Errno 2] No such file or directory: 'test_germ_02/pheno.phenotypes_and_permutations'

Cause R needs to be load as the last one. But even so, my script will give me the same error: /blue/mresende/share/viannam/Kmers/kmersGWAS/external_programs/gemma_0_96: error while loading shared libraries: libgfortran.so.3: cannot open shared object file: No such file or directory

I think that I am missing a file or script from the library in my folder. I sincerely appreciate your help.

[voichek]

Dear Mariana,

I am sorry, but I just missed this last comment from you. Is it still an issue for you?

Sorry, Yoav

[voichek]

I will close this issue, please reopen if needed.

[viannam]

Dear Dr. Voichek, I installed gemma 0.98 and used the patch that you recommend and I was able to run the GWAS. Thank you very much. Best,