search

cumc / xqtl-protocol

Molecular QTL analysis protocol developed by ADSP Functional Genomics Consortium
https://cumc.github.io/xqtl-protocol/
MIT License
38 stars 42 forks source link

The order of the command generated by command generator is not stable #296

Open hsun3163 opened 2 years ago

hsun3163 commented 2 years ago

As demostrated below:

hs3163@node57:/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis$ cat /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/8_celltypes_script_3 | grep sos  | uniq -c
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/gene_annotation.ipynb region_list_generation \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/gene_annotation.ipynb annotate_coord \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/TensorQTL.ipynb cis \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/sample_matcher.ipynb filtered_sample_list \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/phenotype_formatting.ipynb partition_by_chrom \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb qc_no_prune \
      1 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/yml_generator.ipynb yml_list \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb king \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb qc \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PCA.ipynb flashpca \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PCA.ipynb project_samples \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb merge_pca_covariate \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb compute_residual \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PEER_factor.ipynb PEER \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb merge_factor_covariate \
      1 sos run  /home/hs3163/GIT/xqtl-pipeline/pipeline/summary_stats_merger.ipynb     \
hs3163@node57:/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis$ cat /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/8_celltypes_script | grep sos  | uniq -c
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb qc_no_prune \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/gene_annotation.ipynb annotate_coord \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/gene_annotation.ipynb region_list_generation \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/phenotype_formatting.ipynb partition_by_chrom \
      1 sos run  /home/hs3163/GIT/xqtl-pipeline/pipeline/summary_stats_merger.ipynb     \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/sample_matcher.ipynb filtered_sample_list \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb king \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb qc \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PCA.ipynb flashpca \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PCA.ipynb project_samples \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb merge_pca_covariate \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb compute_residual \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PEER_factor.ipynb PEER \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb merge_factor_covariate \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/TensorQTL.ipynb cis \
      1 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/yml_generator.ipynb yml_list \
hs3163@node57:/mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis$ cat /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis/8_celltypes_script_2 | grep sos  | uniq -c
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/gene_annotation.ipynb annotate_coord \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PCA.ipynb flashpca \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/sample_matcher.ipynb filtered_sample_list \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/phenotype_formatting.ipynb partition_by_chrom \
      1 sos run  /home/hs3163/GIT/xqtl-pipeline/pipeline/summary_stats_merger.ipynb     \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb king \
      1 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/yml_generator.ipynb yml_list \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/gene_annotation.ipynb region_list_generation \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb qc \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/GWAS_QC.ipynb qc_no_prune \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PCA.ipynb project_samples \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb merge_pca_covariate \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb compute_residual \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/PEER_factor.ipynb PEER \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/covariate_formatting.ipynb merge_factor_covariate \
      8 sos run /home/hs3163/GIT/xqtl-pipeline/pipeline/TensorQTL.ipynb cis \

All of the above scripts is generated by following commands but the orders of command are different. 

cd /mnt/vast/hpc/csg/snuc_pseudo_bulk/eight_celltypes_analysis
sos run pipeline/eQTL_analysis_commands.ipynb sumstat_merge     \
--recipe  data/recipe_8tissue_new  \
--genotype_list data/genotype_qced/plink_files_list.txt   \
--annotation_gtf data/reference_data/genes.reformatted.gene.gtf   \
--sample_participant_lookup data/reference_data/sampleSheetAfterQC.txt   \
--TARGET_list reference_data/TARGET_list   \
--Association_option "TensorQTL" --cwd output   -n > 8_celltypes_script    &
hsun3163 commented 1 year ago

Will try to produce a reproducible error to post to sos repo