search

shuzhao-li-lab / PythonCentricPipelineForMetabolomics

Python pipeline for metabolomics data preprocessing, QC, standardization and annotation
Other
9 stars 0 forks source link

Signaling completion for each step while running the command lines #50

Closed gmhhope closed 8 months ago

gmhhope commented 8 months ago

Might be great if having each step, if completed, spell out

(base) MLG-JGM357:pcpfm_run gongm$ bash HILICneg_pel_analysis.sh 
Dropping:
     Qstd_HILICneg_1_SZ_01292024_05
     Pool_QC_Monocyte_DC_Mac_Differentiation_Cellpellets_HILICneg_1_SZ_01292024_09
     Pool_QC_Monocyte_DC_Mac_Differentiation_Cellpellets_HILICneg_2_SZ_01292024_31
     Pool_QC_Monocyte_DC_Mac_Differentiation_Cellpellets_HILICneg_3_SZ_01292024_53
     Pool_QC_Monocyte_DC_Mac_Differentiation_Cellpellets_HILICneg_4_SZ_01292024_75
     Pool_QC_Park7cellpellets_1_HILICneg_SZ_01292024_07
     Pool_QC_Park7cellpellets_2_HILICneg_SZ_01292024_77
     Process_blank_Monocyte_DC_Mac_Differentiation_Cellpellets_1_HILICneg_SZ_01292024_03
     Process_blank_Monocyte_DC_Mac_Differentiation_Cellpellets_2_HILICneg_SZ_01292024_81
     Qstd_HILICneg_2_SZ_01292024_79
     Solvent_blank_MobileA2B2_1_HILICneg_SZ_01292024_01
     Solvent_blank_MobileA2B2_2_HILICneg_SZ_01292024_83
{'Type': 'feature_count_z_scores', 'Config': {'intensity_cutoff': 0}, 'Result': {'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample10_HILICneg_SZ_01292024_29': 1.1235726460337816, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample11_HILICneg_SZ_01292024_33': 1.1692834941230323, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample12_HILICneg_SZ_01292024_35': 0.07222313998101684, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample13_HILICneg_SZ_01292024_37': 1.150999154887332, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample14_HILICneg_SZ_01292024_39': -0.07405157390458521, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample15_HILICneg_SZ_01292024_41': -0.8145673129504456, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample16_HILICneg_SZ_01292024_43': 0.5384737904913734, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample17_HILICneg_SZ_01292024_45': -1.7470686139711589, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample18_HILICneg_SZ_01292024_47': 0.44705209431287213, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample19_HILICneg_SZ_01292024_49': -0.0009142169617841814, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample1_HILICneg_SZ_01292024_11': 0.9681557625303294, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample20_HILICneg_SZ_01292024_51': -1.4179505077285541, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample21_HILICneg_SZ_01292024_55': -1.765352953206859, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample22_HILICneg_SZ_01292024_57': -0.28432147511513817, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample23_HILICneg_SZ_01292024_59': 0.22764002348446905, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample24_HILICneg_SZ_01292024_61': -1.7104999354997583, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample25_HILICneg_SZ_01292024_63': -0.08319374352243535, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample26_HILICneg_SZ_01292024_65': -1.4270926773464043, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample27_HILICneg_SZ_01292024_67': -0.17461543970093663, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample28_HILICneg_SZ_01292024_69': -1.5185143735249056, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample29_HILICneg_SZ_01292024_71': -0.2203262877901873, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample2_HILICneg_SZ_01292024_13': 0.26420870195586954, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample30_HILICneg_SZ_01292024_73': -0.9516998572181976, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample3_HILICneg_SZ_01292024_15': 0.9864401017660297, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample4_HILICneg_SZ_01292024_17': 0.721317182848376, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample5_HILICneg_SZ_01292024_19': 0.8218810486447274, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample6_HILICneg_SZ_01292024_21': 1.2332786814479832, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample7_HILICneg_SZ_01292024_23': 0.22764002348446905, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample8_HILICneg_SZ_01292024_25': 0.39219907660577136, 'Monocyte_DC_Mac_Differentiation_Cellpellets_Sample9_HILICneg_SZ_01292024_27': 1.8458040458439418}}
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample10_HILICneg_SZ_01292024_29 1.1235726460337816
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample11_HILICneg_SZ_01292024_33 1.1692834941230323
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample12_HILICneg_SZ_01292024_35 0.07222313998101684
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample13_HILICneg_SZ_01292024_37 1.150999154887332
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample14_HILICneg_SZ_01292024_39 -0.07405157390458521
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample15_HILICneg_SZ_01292024_41 -0.8145673129504456
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample16_HILICneg_SZ_01292024_43 0.5384737904913734
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample17_HILICneg_SZ_01292024_45 -1.7470686139711589
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample18_HILICneg_SZ_01292024_47 0.44705209431287213
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample19_HILICneg_SZ_01292024_49 -0.0009142169617841814
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample1_HILICneg_SZ_01292024_11 0.9681557625303294
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample20_HILICneg_SZ_01292024_51 -1.4179505077285541
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample21_HILICneg_SZ_01292024_55 -1.765352953206859
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample22_HILICneg_SZ_01292024_57 -0.28432147511513817
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample23_HILICneg_SZ_01292024_59 0.22764002348446905
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample24_HILICneg_SZ_01292024_61 -1.7104999354997583
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample25_HILICneg_SZ_01292024_63 -0.08319374352243535
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample26_HILICneg_SZ_01292024_65 -1.4270926773464043
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample27_HILICneg_SZ_01292024_67 -0.17461543970093663
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample28_HILICneg_SZ_01292024_69 -1.5185143735249056
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample29_HILICneg_SZ_01292024_71 -0.2203262877901873
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample2_HILICneg_SZ_01292024_13 0.26420870195586954
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample30_HILICneg_SZ_01292024_73 -0.9516998572181976
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample3_HILICneg_SZ_01292024_15 0.9864401017660297
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample4_HILICneg_SZ_01292024_17 0.721317182848376
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample5_HILICneg_SZ_01292024_19 0.8218810486447274
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample6_HILICneg_SZ_01292024_21 1.2332786814479832
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample7_HILICneg_SZ_01292024_23 0.22764002348446905
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample8_HILICneg_SZ_01292024_25 0.39219907660577136
Monocyte_DC_Mac_Differentiation_Cellpellets_Sample9_HILICneg_SZ_01292024_27 1.8458040458439418
Dropping:

Multiple charges considered: [1, 2, 3]

Khipu search grid: 
               M-H-       Na/H        HCl        K/H        ACN
M0        -1.007276  20.974724  34.969424  36.948606  40.019273
13C/12C   -0.003921  21.978079  35.972779  37.951961  41.022628
13C/12C*2  0.999434  22.981434  36.976134  38.955316  42.025983
13C/12C*3  2.002789  23.984789  37.979489  39.958671  43.029338
Downsized input network with 32 features, highest peak at F6505 
Downsized input network with 30 features, highest peak at F4566 
Downsized input network with 23 features, highest peak at F3746 
Downsized input network with 45 features, highest peak at F7257 
Downsized input network with 21 features, highest peak at F8403 
Downsized input network with 37 features, highest peak at F9226 
Downsized input network with 39 features, highest peak at F50827 
Downsized input network with 25 features, highest peak at F50828 
Downsized input network with 23 features, highest peak at F46580 
Downsized input network with 29 features, highest peak at F60485 
Downsized input network with 36 features, highest peak at F70734 
Downsized input network with 24 features, highest peak at F71041 
Downsized input network with 27 features, highest peak at F51382 
Constructed 6272 khipus in this round.

Khipu search grid: 
                       M-H-, 2x charged  Na/H, 2x charged  HCl, 2x charged  K/H, 2x charged  ACN, 2x charged
M0                            -0.503638          9.983724        16.981074        17.970665        19.505998
13C/12C, 2x charged           -0.001961         10.485401        17.482751        18.472342        20.007676
13C/12C*2, 2x charged          0.499717         10.987079        17.984429        18.974020        20.509353
13C/12C*3, 2x charged          1.001394         11.488756        18.486106        19.475697        21.011031
Downsized input network with 33 features, highest peak at F71463 
Constructed 1990 khipus in this round.

Khipu search grid: 
                       M-H-, 3x charged  Na/H, 3x charged  HCl, 3x charged  K/H, 3x charged  ACN, 3x charged
M0                            -0.335759          6.320057        10.984957        11.644684        12.668240
13C/12C, 3x charged           -0.001307          6.654509        11.319409        11.979136        13.002692
13C/12C*2, 3x charged          0.333145          6.988960        11.653860        12.313588        13.337143
13C/12C*3, 3x charged          0.667596          7.323412        11.988312        12.648039        13.671595
Constructed 433 khipus in this round.

 ~~~~~~ Got 8695 khipus, with 24920 features ~~~~~~~ 

No such table:  log_transformed_for_analysis
Unable to processes section: 
 {'section': 'figure', 'table': 'pref_blank_masked', 'name': 'pca'}
Unable to processes section: 
 {'section': 'figure', 'table': 'masked_pref_unknowns', 'name': 'pca'}
Unable to processes section: 
 {'section': 'figure', 'table': 'qaqc_filtered_masked_pref_unknowns', 'name': 'pca'}
Unable to processes section: 
 {'section': 'figure', 'table': 'pref_normalized', 'name': 'pca'}
...

Since there are multiple steps involved, would be great to have each steps completed spelled out.

Best, MG

jmmitc06 commented 8 months ago

will add to version 1.0.10

jmmitc06 commented 8 months ago

added to newest commit, will close.