[RUNNING] Do a run on the many-SNPs data with for all combinations of autoencoder and phenotypic models

AJResearchGroup / nsphs_ml_qt

R package for nsphs_ml_qt

GNU General Public License v3.0

0 stars 1 forks source link

[RUNNING] Do a run on the many-SNPs data with for all combinations of autoencoder and phenotypic models #42

Closed richelbilderbeek closed 2 years ago

richelbilderbeek commented 2 years ago

Use the data like #29 (IL-17RA), which as multiple SNPs

Todo:

[x] Find best model combination: the M1 and p1 model combination gives the best results
[ ] Analyse runtimes

richelbilderbeek commented 2 years ago

Started all 3024 jobs, this alone takes 6 mins :-)

richelbilderbeek commented 2 years ago

This one failed:

[richel@sens2021565-bianca ~]$ cat 22_create_issue_42_M0_1n_p1_1_data.log
Parameters: /home/richel/data_issue_42_M0_1n_p1_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /home/richel/data_issue_42_M0_1n_p1_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-16T12:18:12+0200
Running on computer with HOSTNAME: sens2021565-b9
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_bianca/22_create_issue_29_data.R /home/richel/data_issue_42_M0_1n_p1_1/experiment_params.csv'
gcae_experiment_params_filename: /home/richel/data_issue_42_M0_1n_p1_1/experiment_params.csv
Parameters are valid
matches: 
 * NA
 * NA
 * NA
 * NA
 * NA
Error: no matches found for 'gcae_experiment_params_filename': /home/richel/data_issue_42_M0_1n_p1_1/experiment_params.csv
Execution halted
End time: 2022-05-16T12:18:14+0200
Duration: 2 seconds

This happened to all the 22s:

[richel@sens2021565-bianca ~]$ egrep -iR "error" --include=*.log
22_create_issue_42_M0_1n_p0_10_data.log:Error: no matches found for 'gcae_experiment_params_filename': /home/richel/data_issue_42_M0_1n_p0_10/experiment_params.csv
22_create_issue_42_M0_1n_p0_100_data.log:Error: no matches found for 'gcae_experiment_params_filename': /home/richel/data_issue_42_M0_1n_p0_100/experiment_params.csv
22_create_issue_42_M0_1n_p0_1000_data.log:Error: no matches found for 'gcae_experiment_params_filename': /home/richel/data_issue_42_M0_1n_p0_1000/experiment_params.csv
22_create_issue_42_M0_1n_p0_1_data.log:Error: no matches found for 'gcae_experiment_params_filename': /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
22_create_issue_42_M0_1n_p1_1_data.log:Error: no matches found for 'gcae_experiment_params_filename': /home/richel/data_issue_42_M0_1n_p1_1/experiment_params.csv

richelbilderbeek commented 2 years ago

Ah, I fixed the wrong one. This one is more fixed than ever:

[richel@sens2021565-bianca ~]$ cat 21_create_issue_42_M0_1n_p0_1_params.log 
Parameters: /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-16T12:48:33+0200
Running on computer with HOSTNAME: sens2021565-b9
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_bianca/21_create_issue_42_params.R /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv'
gcae_experiment_params_filename: /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
Matches in 'gcae_experiment_params_filename': 
 * /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
 * /home/richel/data_issue_42_M0_1n_p0_1/
 * /home/richel
 * issue_42_M0_1n_p0_1
 * M0_1n
 * p0
 * 1
unique_id: issue_42_M0_1n_p0_1
datadir: /home/richel/data_issue_42_M0_1n_p0_1/
data: data_issue_42_M0_1n_p0_1
trainedmodeldir: /home/richel/data_issue_42_M0_1n_p0_1_ae/
model_id: M0_1n
pheno_model_id: p0
window_kb: 1
base_input_filename: /home/richel/data_issue_42_M0_1n_p0_1/data_issue_42_M0_1n_p0_1
Saved 'gcae_experiment_params' to /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
Really saved 'gcae_experiment_params' at /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
End time: 2022-05-16T12:48:37+0200
Duration: 4 seconds

It is the next one that fails:

[richel@sens2021565-bianca ~]$ cat 22_create_issue_42_M0_1n_p0_1_data.log
Parameters: /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-16T12:48:41+0200
Running on computer with HOSTNAME: sens2021565-b9
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_bianca/22_create_issue_29_data.R /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv'
gcae_experiment_params_filename: /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
Parameters are valid
matches: 
 * NA
 * NA
 * NA
 * NA
 * NA
Error: no matches found for 'gcae_experiment_params_filename': /home/richel/data_issue_42_M0_1n_p0_1/experiment_params.csv
Execution halted
End time: 2022-05-16T12:48:43+0200
Duration: 2 seconds

richelbilderbeek commented 2 years ago

Fixed by using its own 22 script, with its own regex

richelbilderbeek commented 2 years ago

Running!

[richel@sens2021565-bianca ~]$ tail 22_create_issue_42_M0_1n_p0_1_data.log

$n_individuals_in_fam_table
[1] 870

$n_individuals_in_phe_table
[1] 870

Done resizing the data
End time: 2022-05-16T13:20:55+0200
Duration: 18 seconds

richelbilderbeek commented 2 years ago

[richel@sens2021565-bianca ~]$ while true; do echo "$(date): $(squeue -u $USER | wc --lines)" ; sleep 60; done
ma mei 16 13:25:17 CEST 2022: 2991

richelbilderbeek commented 2 years ago

This seems to go excellent!

[richel@sens2021565-bianca ~]$ cd /proj/sens2021565/nobackup/
NSPHS_data/          nsphs_ml_qt_results/ wharf/               
[richel@sens2021565-bianca ~]$ cd /proj/sens2021565/nobackup/nsphs_ml_qt_results/
[richel@sens2021565-bianca nsphs_ml_qt_results]$ ls
02_start_richel_issue_129.log     data_issue_42_M0_4n_p2_10    data_issue_42_M1_4n_p0_10
02_start_richel_issue_136.log     data_issue_42_M0_4n_p2_100   data_issue_42_M1_4n_p0_100
10_create_richel_issue_129.log    data_issue_42_M0_4n_p2_1000  data_issue_42_M1_4n_p0_1000
10_create_richel_issue_136.log    data_issue_42_M0_5n_p0_1     data_issue_42_M1_4n_p1_1
11_train_richel_issue_129.log     data_issue_42_M0_5n_p0_10    data_issue_42_M1_4n_p1_10
11_train_richel_issue_136.log     data_issue_42_M0_5n_p0_100   data_issue_42_M1_4n_p1_100
12_project_richel_issue_129.log   data_issue_42_M0_5n_p0_1000  data_issue_42_M1_4n_p1_1000
12_project_richel_issue_136.log   data_issue_42_M0_5n_p1_1     data_issue_42_M1_4n_p2_1
13_plot_richel_issue_129.log      data_issue_42_M0_5n_p1_10    data_issue_42_M1_4n_p2_10
13_plot_richel_issue_136.log      data_issue_42_M0_5n_p1_100   data_issue_42_M1_4n_p2_100
14_animate_richel_issue_129.log   data_issue_42_M0_5n_p1_1000  data_issue_42_M1_4n_p2_1000
14_animate_richel_issue_136.log   data_issue_42_M0_5n_p2_1     data_issue_42_M1_5n_p0_1
15_evaluate_richel_issue_129.log  data_issue_42_M0_5n_p2_10    data_issue_42_M1_5n_p0_10
15_evaluate_richel_issue_136.log  data_issue_42_M0_5n_p2_100   data_issue_42_M1_5n_p0_100
16_analyse_richel_issue_129.log   data_issue_42_M0_5n_p2_1000  data_issue_42_M1_5n_p0_1000
16_analyse_richel_issue_136.log   data_issue_42_M0_p0_1        data_issue_42_M1_5n_p1_1
17_zip_richel_issue_129.log       data_issue_42_M0_p0_10       data_issue_42_M1_5n_p1_10
17_zip_richel_issue_136.log       data_issue_42_M0_p0_100      data_issue_42_M1_5n_p1_100
20220304                          data_issue_42_M0_p0_1000     data_issue_42_M1_5n_p1_1000
data_issue_42_M0_1n_p0_1          data_issue_42_M0_p1_1        data_issue_42_M1_5n_p2_1
data_issue_42_M0_1n_p0_10         data_issue_42_M0_p1_10       data_issue_42_M1_5n_p2_10
data_issue_42_M0_1n_p0_100        data_issue_42_M0_p1_100      data_issue_42_M1_5n_p2_100
data_issue_42_M0_1n_p0_1000       data_issue_42_M0_p1_1000     data_issue_42_M1_5n_p2_1000
data_issue_42_M0_1n_p1_1          data_issue_42_M0_p2_1        data_issue_42_M1_p0_1
data_issue_42_M0_1n_p1_10         data_issue_42_M0_p2_10       data_issue_42_M1_p0_10
data_issue_42_M0_1n_p1_100        data_issue_42_M0_p2_100      data_issue_42_M1_p0_100
data_issue_42_M0_1n_p1_1000       data_issue_42_M0_p2_1000     data_issue_42_M1_p0_1000
data_issue_42_M0_1n_p2_1          data_issue_42_M1_1n_p0_1     data_issue_42_M1_p1_1
data_issue_42_M0_1n_p2_10         data_issue_42_M1_1n_p0_10    data_issue_42_M1_p1_10
data_issue_42_M0_1n_p2_100        data_issue_42_M1_1n_p0_100   data_issue_42_M1_p1_100
data_issue_42_M0_1n_p2_1000       data_issue_42_M1_1n_p0_1000  data_issue_42_M1_p1_1000
data_issue_42_M0_2n_p0_1          data_issue_42_M1_1n_p1_1     data_issue_42_M1_p2_1
data_issue_42_M0_2n_p0_10         data_issue_42_M1_1n_p1_10    data_issue_42_M1_p2_10
data_issue_42_M0_2n_p0_100        data_issue_42_M1_1n_p1_100   data_issue_42_M1_p2_100
data_issue_42_M0_2n_p0_1000       data_issue_42_M1_1n_p1_1000  data_issue_42_M1_p2_1000
data_issue_42_M0_2n_p1_1          data_issue_42_M1_1n_p2_1     data_issue_42_M3d_1n_p0_1
data_issue_42_M0_2n_p1_10         data_issue_42_M1_1n_p2_10    data_issue_42_M3d_1n_p0_10
data_issue_42_M0_2n_p1_100        data_issue_42_M1_1n_p2_100   data_issue_42_M3d_1n_p0_100
data_issue_42_M0_2n_p1_1000       data_issue_42_M1_1n_p2_1000  data_issue_42_M3d_1n_p0_1000
data_issue_42_M0_2n_p2_1          data_issue_42_M1_2n_p0_1     data_issue_42_M3d_1n_p1_1
data_issue_42_M0_2n_p2_10         data_issue_42_M1_2n_p0_10    data_issue_42_M3d_1n_p1_10
data_issue_42_M0_2n_p2_100        data_issue_42_M1_2n_p0_100   data_issue_42_M3d_1n_p1_100
data_issue_42_M0_2n_p2_1000       data_issue_42_M1_2n_p0_1000  data_issue_42_M3d_1n_p1_1000
data_issue_42_M0_3n_p0_1          data_issue_42_M1_2n_p1_1     data_issue_42_M3d_1n_p2_1
data_issue_42_M0_3n_p0_10         data_issue_42_M1_2n_p1_10    data_issue_42_M3d_1n_p2_10
data_issue_42_M0_3n_p0_100        data_issue_42_M1_2n_p1_100   data_issue_42_M3d_1n_p2_100
data_issue_42_M0_3n_p0_1000       data_issue_42_M1_2n_p1_1000  data_richel_issue_129
data_issue_42_M0_3n_p1_1          data_issue_42_M1_2n_p2_1     data_richel_issue_129_ae
data_issue_42_M0_3n_p1_10         data_issue_42_M1_2n_p2_10    data_richel_issue_130
data_issue_42_M0_3n_p1_100        data_issue_42_M1_2n_p2_100   data_richel_issue_130_ae
data_issue_42_M0_3n_p1_1000       data_issue_42_M1_2n_p2_1000  data_richel_issue_136
data_issue_42_M0_3n_p2_1          data_issue_42_M1_3n_p0_1     data_richel_issue_136_ae
data_issue_42_M0_3n_p2_10         data_issue_42_M1_3n_p0_10    issue_28_sensitive_1000_epochs_p0.zip
data_issue_42_M0_3n_p2_100        data_issue_42_M1_3n_p0_100   issue_28_sensitive_1000_epochs_p1.zip
data_issue_42_M0_3n_p2_1000       data_issue_42_M1_3n_p0_1000  issue_29_sensitive_1000_epochs_p0.zip
data_issue_42_M0_4n_p0_1          data_issue_42_M1_3n_p1_1     issue_29_sensitive_1000_epochs_p1.zip
data_issue_42_M0_4n_p0_10         data_issue_42_M1_3n_p1_10    issue_5_100_epochs_20220422.zip
data_issue_42_M0_4n_p0_100        data_issue_42_M1_3n_p1_100   issue_5_sensitive_20220425.zip
data_issue_42_M0_4n_p0_1000       data_issue_42_M1_3n_p1_1000  issue_5_sensitive.zip
data_issue_42_M0_4n_p1_1          data_issue_42_M1_3n_p2_1     richel_issue_129_1
data_issue_42_M0_4n_p1_10         data_issue_42_M1_3n_p2_10    richel_issue_129_2
data_issue_42_M0_4n_p1_100        data_issue_42_M1_3n_p2_100   richel_issue_129.zip
data_issue_42_M0_4n_p1_1000       data_issue_42_M1_3n_p2_1000  richel_issue_136.zip
data_issue_42_M0_4n_p2_1          data_issue_42_M1_4n_p0_1
[richel@sens2021565-bianca nsphs_ml_qt_results]$ cd data_issue_42_M0_2n_p0_1000
[richel@sens2021565-bianca data_issue_42_M0_2n_p0_1000]$ ls
data_issue_42_M0_2n_p0_1000.bed  data_issue_42_M0_2n_p0_1000.fam  experiment_params.csv
data_issue_42_M0_2n_p0_1000.bim  data_issue_42_M0_2n_p0_1000.phe
[richel@sens2021565-bianca data_issue_42_M0_2n_p0_1000]$ cat experiment_params.csv
parameter,value
datadir,/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_2n_p0_1000/
data,data_issue_42_M0_2n_p0_1000
superpops,
model_id,M0_2n
train_opts_id,ex3
data_opts_id,b_0_4
trainedmodeldir,/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_2n_p0_1000_ae/
pheno_model_id,p0
analyse_epochs,"10,20,30,40,50,60,70,80,90,100,110,120,130,140,150,160,170,180,190,200,210,220,230,240,250,260,270,280,290,300,310,320,330,340,350,360,370,380,390,400,410,420,430,440,450,460,470,480,490,500,510,520,530,540,550,560,570,580,590,600,610,620,630,640,650,660,670,680,690,700,710,720,730,740,750,760,770,780,790,800,810,820,830,840,850,860,870,880,890,900,910,920,930,940,950,960,970,980,990,1000"
metrics,
gcae_folder,/opt/gcae_richel
ormr_folder_name,python3
gcae_version,1.0
python_version,3.6

richelbilderbeek commented 2 years ago

No, there are problems:

The logs are saved in the home folder
The working folder has zips and log files

richelbilderbeek commented 2 years ago

Fixed and restarted.

richelbilderbeek commented 2 years ago

[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/92_poll_n_jobs.sh 
ma mei 16 14:54:45 CEST 2022: 2972 jobs

richelbilderbeek commented 2 years ago

[richel@sens2021565-bianca nsphs_ml_qt_results]$ cat 25_run_issue_42_M0_1n_p0_1000.log
Parameters: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-16T15:25:34+0200
Running on computer with HOSTNAME: sens2021565-b11
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_rackham/25_run.R /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/experiment_params.csv'
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/experiment_params.csv
Running the GCAE experiment
Error in gcae_train_more(gcae_setup = gcae_experiment_params$gcae_setup,  : 
  'ae_out_subfolder' not found at path '/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000_ae/ae.M0_1n.ex3.b_0_4.data_issue_42_M0_1n_p0_1000.p0' 
gcae_setup$datadir: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/
gcae_setup$data: data_issue_42_M0_1n_p0_1000
gcae_setup$superpops: 
gcae_setup$model_id: M0_1n
gcae_setup$train_opts_id: ex3
gcae_setup$data_opts_id: b_0_4
gcae_setup$trainedmodeldir: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000_ae/
gcae_setup$pheno_model_id: p0
gcae_options$gcae_folder: /opt/gcae_richel
gcae_options$ormr_folder_name: python3
gcae_options$gcae_version: 1.0
gcae_options$python_version: 3.6
'args': 'train --datadir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/ --data data_issue_42_M0_1n_p0_1000 --model_id M0_1n --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /proj/sens2021565/nobackup/ns
Calls: <Anonymous> -> gcae_train_more
In addition: Warning message:
In system2(command = run_args[1], args = run_args[-1], stdout = TRUE,  :
  running command ''python3' /opt/gcae_richel/run_gcae.py train --datadir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/ --data data_issue_42_M0_1n_p0_1000 --model_id M0_1n --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000_ae/ --pheno_model_id p0 2>&1' had status 1
Execution halted
End time: 2022-05-16T15:26:08+0200
Duration: 34 seconds

richelbilderbeek commented 2 years ago

Running it by hand:

[richel@sens2021565-bianca nsphs_ml_qt_results]$ singularity run ~/nsphs_ml_qt/nsphs_ml_qt.sif python3 /opt/gcae_richel/run_gcae.py train --datadir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/ --data data_issue_42_M0_1n_p0_1000 --model_id M0_1n --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000_ae/ --pheno_model_id p0
'nsphs_ml_qt.sif' running with arguments 'python3 /opt/gcae_richel/run_gcae.py train --datadir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/ --data data_issue_42_M0_1n_p0_1000 --model_id M0_1n --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000_ae/ --pheno_model_id p0'
2022-05-16 15:43:40.500770: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2022-05-16 15:43:40.500874: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2022-05-16 15:43:40.500918: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (sens2021565-bianca.uppmax.uu.se): /proc/driver/nvidia/version does not exist
2022-05-16 15:43:40.501483: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-05-16 15:43:40.511710: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2394450000 Hz
2022-05-16 15:43:40.512234: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2b5048000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-05-16 15:43:40.512250: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
tensorflow version 2.2.0

______________________________ arguments ______________________________
train : True
datadir : /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/
data : data_issue_42_M0_1n_p0_1000
model_id : M0_1n
train_opts_id : ex3
data_opts_id : b_0_4
save_interval : 10
epochs : 10
resume_from : 0
trainedmodeldir : /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000_ae/
pheno_model_id : p0
project : False
superpops : None
epoch : None
pdata : None
trainedmodelname : None
plot : False
animate : False
evaluate : False
metrics : None

______________________________ data opts ______________________________
sparsifies : [0.0, 0.1, 0.2, 0.3, 0.4]
norm_opts : {'flip': False, 'missing_val': -1.0}
norm_mode : genotypewise01
impute_missing : True
validation_split : 0.2

______________________________ train opts ______________________________
learning_rate : 0.00032
batch_size : 10
noise_std : 0.0032
n_samples : -1
loss : {'module': 'tf.keras.losses', 'class': 'CategoricalCrossentropy', 'args': {'from_logits': False}}
regularizer : {'reg_factor': 1e-07, 'module': 'tf.keras.regularizers', 'class': 'l2'}
lr_scheme : {'module': 'tf.keras.optimizers.schedules', 'class': 'ExponentialDecay', 'args': {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}}
______________________________
Imputing originally missing genotypes to most common value.
Reading ind pop list from /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/data_issue_42_M0_1n_p0_1000.fam
Reading ind pop list from /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M0_1n_p0_1000/data_issue_42_M0_1n_p0_1000.fam
Mapping files: 100%|████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 82.67it/s]
Using learning rate schedule tf.keras.optimizers.schedules.ExponentialDecay with {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}

______________________________ Data ______________________________
N unique train samples: 696
--- training on : 696
N valid samples: 174
N markers: 5269

______________________________ Building model ______________________________
Traceback (most recent call last):
  File "/opt/gcae_richel/run_gcae.py", line 1619, in <module>
    main()
  File "/opt/gcae_richel/run_gcae.py", line 1004, in main
    autoencoder = Autoencoder(model_architecture, n_markers, noise_std, regularizer)
  File "/opt/gcae_richel/run_gcae.py", line 87, in __init__
    layer_module = getattr(eval(first_layer_def["module"]), first_layer_def["class"])
TypeError: eval() arg 1 must be a string, bytes or code object

richelbilderbeek commented 2 years ago

Hmmm, all models seem to be there:

[richel@sens2021565-bianca nsphs_ml_qt_results]$ singularity shell ~/nsphs_ml_qt/nsphs_ml_qt.sif 
Singularity> cd /opt/
Singularity> ls
gcae  gcae_richel  pandoc  plinkr
Singularity> cd gcae
Singularity> cd ../gcae_richel
Singularity> cd models
Singularity> ls
M0.json     M1.json M3d.json     M3e.json     M3f.json     M3j10U.json     M3j10X.json     p0.json
M0_1n.json  M1_1n.json  M3d_1n.json  M3e_1n.json  M3f_1n.json  M3j10U_1n.json  M3j10X_1n.json  p1.json
M0_2n.json  M1_2n.json  M3d_2n.json  M3e_2n.json  M3f_2n.json  M3j10U_2n.json  M3j10X_2n.json  p2.json
M0_3n.json  M1_3n.json  M3d_3n.json  M3e_3n.json  M3f_3n.json  M3j10U_3n.json  M3j10X_3n.json
M0_4n.json  M1_4n.json  M3d_4n.json  M3e_4n.json  M3f_4n.json  M3j10U_4n.json  M3j10X_4n.json
M0_5n.json  M1_5n.json  M3d_5n.json  M3e_5n.json  M3f_5n.json  M3j10U_5n.json  M3j10X_5n.json

richelbilderbeek commented 2 years ago

Not all combinations work, but some do. Just run all and see how many pass: as GCAE fails at the start of the training, not much computing time is lost.

richelbilderbeek commented 2 years ago

[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/92_poll_n_jobs.sh
di mei 17 12:12:17 CEST 2022: 3002 jobs

richelbilderbeek commented 2 years ago

60 jobs that work:

[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll_jobs.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             18866      core 29_zip.s   richel PD       0:00      1 (Dependency)
[...]          
             17072      core 29_zip.s   richel PD       0:00      1 (Dependency)
             18810      core 25_run.s   richel  R      36:50      1 sens2021565-b16
             18852      core 25_run.s   richel  R      34:18      1 sens2021565-b40
             18846      core 25_run.s   richel  R      34:19      1 sens2021565-b36
             18840      core 25_run.s   richel  R      34:24      1 sens2021565-b36
             18834      core 25_run.s   richel  R      34:26      1 sens2021565-b26
             18828      core 25_run.s   richel  R      31:39      1 sens2021565-b17
             18816      core 25_run.s   richel  R      31:42      1 sens2021565-b32
             18822      core 25_run.s   richel  R      39:02      1 sens2021565-b42
             18432      core 25_run.s   richel  R      37:52      1 sens2021565-b16
             18426      core 25_run.s   richel  R      37:56      1 sens2021565-b16
             18414      core 25_run.s   richel  R      37:59      1 sens2021565-b11
             18420      core 25_run.s   richel  R      37:59      1 sens2021565-b9
             18372      core 25_run.s   richel  R      38:03      1 sens2021565-b12
             18408      core 25_run.s   richel  R      38:03      1 sens2021565-b12
             18366      core 25_run.s   richel  R      38:07      1 sens2021565-b11
             18858      core 25_run.s   richel  R      36:07      1 sens2021565-b17
             18864      core 25_run.s   richel  R      36:07      1 sens2021565-b17
             18798      core 25_run.s   richel  R      36:14      1 sens2021565-b20
             18804      core 25_run.s   richel  R      36:14      1 sens2021565-b20
             18402      core 25_run.s   richel  R      42:53      1 sens2021565-b11
             18396      core 25_run.s   richel  R      42:55      1 sens2021565-b36
             18390      core 25_run.s   richel  R      42:58      1 sens2021565-b38
             18384      core 25_run.s   richel  R      43:00      1 sens2021565-b32
             18378      core 25_run.s   richel  R      43:01      1 sens2021565-b42
             17568      core 25_run.s   richel  R      49:37      1 sens2021565-b18
             17562      core 25_run.s   richel  R      49:39      1 sens2021565-b38
             17994      core 25_run.s   richel  R      45:39      1 sens2021565-b28
             17988      core 25_run.s   richel  R      45:46      1 sens2021565-b43
             17976      core 25_run.s   richel  R      45:51      1 sens2021565-b21
             17982      core 25_run.s   richel  R      45:51      1 sens2021565-b21
             17970      core 25_run.s   richel  R      45:55      1 sens2021565-b41
             17958      core 25_run.s   richel  R      45:58      1 sens2021565-b16
             17964      core 25_run.s   richel  R      45:58      1 sens2021565-b16
             17952      core 25_run.s   richel  R      46:01      1 sens2021565-b43
             17940      core 25_run.s   richel  R      46:04      1 sens2021565-b36
             17946      core 25_run.s   richel  R      46:04      1 sens2021565-b36
             17934      core 25_run.s   richel  R      46:12      1 sens2021565-b43
             18000      core 25_run.s   richel  R      45:01      1 sens2021565-b33
             17550      core 25_run.s   richel  R      45:22      1 sens2021565-b42
             17544      core 25_run.s   richel  R      45:26      1 sens2021565-b28
             17538      core 25_run.s   richel  R      45:28      1 sens2021565-b29
             17526      core 25_run.s   richel  R      45:29      1 sens2021565-b30
             17532      core 25_run.s   richel  R      45:29      1 sens2021565-b30
             17130      core 25_run.s   richel  R      54:10      1 sens2021565-b43
             17136      core 25_run.s   richel  R      54:10      1 sens2021565-b43
             17112      core 25_run.s   richel  R      54:12      1 sens2021565-b35
             17118      core 25_run.s   richel  R      54:12      1 sens2021565-b42
             17124      core 25_run.s   richel  R      54:12      1 sens2021565-b42
             17100      core 25_run.s   richel  R      54:15      1 sens2021565-b41
             17106      core 25_run.s   richel  R      54:15      1 sens2021565-b41
             17088      core 25_run.s   richel  R      54:16      1 sens2021565-b40
             17094      core 25_run.s   richel  R      54:16      1 sens2021565-b40
             17076      core 25_run.s   richel  R      54:18      1 sens2021565-b28
             17082      core 25_run.s   richel  R      54:18      1 sens2021565-b28
             17070      core 25_run.s   richel  R      54:20      1 sens2021565-b30
             17520      core 25_run.s   richel  R      49:49      1 sens2021565-b22
             17514      core 25_run.s   richel  R      49:52      1 sens2021565-b28
             17502      core 25_run.s   richel  R      50:06      1 sens2021565-b11
             17508      core 25_run.s   richel  R      50:06      1 sens2021565-b11
             17556      core 25_run.s   richel  R      49:40      1 sens2021565-b33

richelbilderbeek commented 2 years ago

[richel@sens2021565-bianca nsphs_ml_qt_results]$ cat succeeded.txt
./data_issue_42_M1_p2_1000_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p2_1000.p2/pheno_weights
./data_issue_42_M3d_p0_1000_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p0_1000.p0/pheno_weights
./data_issue_42_M1_p1_100_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p1_100.p1/pheno_weights
./data_issue_42_M3e_p0_1000_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p0_1000.p0/pheno_weights
./data_issue_42_M3e_p2_100_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p2_100.p2/pheno_weights
./data_issue_42_M3f_p0_100_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p0_100.p0/pheno_weights
./data_issue_42_M3f_p1_100_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p1_100.p1/pheno_weights
./data_issue_42_M1_p0_1000_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p0_1000.p0/pheno_weights
./data_issue_42_M0_p0_10_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p0_10.p0/pheno_weights
./data_issue_42_M0_p1_1_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p1_1.p1/pheno_weights
./data_issue_42_M3e_p2_1_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p2_1.p2/pheno_weights
./data_issue_42_M3f_p0_10_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p0_10.p0/pheno_weights
./data_issue_42_M3f_p0_1000_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p0_1000.p0/pheno_weights
./data_issue_42_M1_p1_10_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p1_10.p1/pheno_weights
./data_issue_42_M3f_p1_1_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p1_1.p1/pheno_weights
./data_issue_42_M0_p1_10_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p1_10.p1/pheno_weights
./data_issue_42_M3d_p1_10_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p1_10.p1/pheno_weights
./data_issue_42_M3d_p1_1000_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p1_1000.p1/pheno_weights
./data_issue_42_M3e_p0_10_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p0_10.p0/pheno_weights
./data_issue_42_M3e_p1_1000_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p1_1000.p1/pheno_weights
./data_issue_42_M3f_p1_10_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p1_10.p1/pheno_weights
./data_issue_42_M1_p0_100_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p0_100.p0/pheno_weights
./data_issue_42_M0_p2_1_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p2_1.p2/pheno_weights
./data_issue_42_M3d_p2_100_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p2_100.p2/pheno_weights
./data_issue_42_M3e_p0_100_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p0_100.p0/pheno_weights
./data_issue_42_M3e_p1_10_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p1_10.p1/pheno_weights
./data_issue_42_M0_p2_1000_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p2_1000.p2/pheno_weights
./data_issue_42_M0_p1_1000_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p1_1000.p1/pheno_weights
./data_issue_42_M3d_p0_1_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p0_1.p0/pheno_weights
./data_issue_42_M3d_p1_1_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p1_1.p1/pheno_weights
./data_issue_42_M1_p1_1000_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p1_1000.p1/pheno_weights
./data_issue_42_M3d_p0_10_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p0_10.p0/pheno_weights
./data_issue_42_M3d_p2_1_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p2_1.p2/pheno_weights
./data_issue_42_M3e_p1_100_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p1_100.p1/pheno_weights
./data_issue_42_M3f_p2_100_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p2_100.p2/pheno_weights
./data_issue_42_M3f_p2_10_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p2_10.p2/pheno_weights
./data_issue_42_M1_p0_10_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p0_10.p0/pheno_weights
./data_issue_42_M1_p2_100_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p2_100.p2/pheno_weights
./data_issue_42_M3d_p1_100_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p1_100.p1/pheno_weights
./data_issue_42_M3e_p2_1000_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p2_1000.p2/pheno_weights
./data_issue_42_M3f_p0_1_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p0_1.p0/pheno_weights
./data_issue_42_M0_p0_100_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p0_100.p0/pheno_weights
./data_issue_42_M0_p0_1000_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p0_1000.p0/pheno_weights
./data_issue_42_M1_p0_1_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p0_1.p0/pheno_weights
./data_issue_42_M3f_p2_1000_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p2_1000.p2/pheno_weights
./data_issue_42_M3f_p1_1000_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p1_1000.p1/pheno_weights
./data_issue_42_M0_p0_1_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p0_1.p0/pheno_weights
./data_issue_42_M3e_p0_1_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p0_1.p0/pheno_weights
./data_issue_42_M3f_p2_1_ae/ae.M3f.ex3.b_0_4.data_issue_42_M3f_p2_1.p2/pheno_weights
./data_issue_42_M1_p1_1_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p1_1.p1/pheno_weights
./data_issue_42_M1_p2_10_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p2_10.p2/pheno_weights
./data_issue_42_M3d_p2_10_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p2_10.p2/pheno_weights
./data_issue_42_M0_p2_100_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p2_100.p2/pheno_weights
./data_issue_42_M1_p2_1_ae/ae.M1.ex3.b_0_4.data_issue_42_M1_p2_1.p2/pheno_weights
./data_issue_42_M3d_p2_1000_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p2_1000.p2/pheno_weights
./data_issue_42_M3e_p1_1_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p1_1.p1/pheno_weights
./data_issue_42_M3e_p2_10_ae/ae.M3e.ex3.b_0_4.data_issue_42_M3e_p2_10.p2/pheno_weights
./data_issue_42_M0_p1_100_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p1_100.p1/pheno_weights
./data_issue_42_M0_p2_10_ae/ae.M0.ex3.b_0_4.data_issue_42_M0_p2_10.p2/pheno_weights
./data_issue_42_M3d_p0_100_ae/ae.M3d.ex3.b_0_4.data_issue_42_M3d_p0_100.p0/pheno_weights

richelbilderbeek commented 2 years ago

Removed duplicates, now in table:

`model_id`	`pheno_model_id`
M0	p0
M0	p1
M0	p2
M1	p0
M1	p1
M1	p2
M3d	p0
M3d	p1
M3d	p2
M3e	p0
M3e	p1
M3e	p2
M3f	p0
M3f	p1
M3f	p2

So there are 5 autoencoder models and 3 phenotypic models

richelbilderbeek commented 2 years ago

This is an example error:

[richel@sens2021565-bianca nsphs_ml_qt_results]$ cat 25_run_issue_42_M3f_5n_p2_1.log
Parameters: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-17T13:18:38+0200
Running on computer with HOSTNAME: sens2021565-b38
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_rackham/25_run.R /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1/experiment_params.csv'
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1/experiment_params.csv
Running the GCAE experiment
Error in gcae_train_more(gcae_setup = gcae_experiment_params$gcae_setup,  : 
  'ae_out_subfolder' not found at path '/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1_ae/ae.M3f_5n.ex3.b_0_4.data_issue_42_M3f_5n_p2_1.p2' 
gcae_setup$datadir: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1/
gcae_setup$data: data_issue_42_M3f_5n_p2_1
gcae_setup$superpops: 
gcae_setup$model_id: M3f_5n
gcae_setup$train_opts_id: ex3
gcae_setup$data_opts_id: b_0_4
gcae_setup$trainedmodeldir: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1_ae/
gcae_setup$pheno_model_id: p2
gcae_options$gcae_folder: /opt/gcae_richel
gcae_options$ormr_folder_name: python3
gcae_options$gcae_version: 1.0
gcae_options$python_version: 3.6
'args': 'train --datadir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1/ --data data_issue_42_M3f_5n_p2_1 --model_id M3f_5n --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /proj/sens2021565/nobackup/nsphs_ml_qt_r
Calls: <Anonymous> -> gcae_train_more
In addition: Warning message:
In system2(command = run_args[1], args = run_args[-1], stdout = TRUE,  :
  running command ''python3' /opt/gcae_richel/run_gcae.py train --datadir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1/ --data data_issue_42_M3f_5n_p2_1 --model_id M3f_5n --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3f_5n_p2_1_ae/ --pheno_model_id p2 2>&1' had status 1
Execution halted
End time: 2022-05-17T13:19:12+0200
Duration: 34 seconds

richelbilderbeek commented 2 years ago

All models have finished, now zipping the results.

richelbilderbeek commented 2 years ago

This takes at least 1 hour. The first zip failed, setting it to 100 hours now :-/ ...?

richelbilderbeek commented 2 years ago

It takes 46 mins:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             19737      core 29_zip_i   richel  R      45:38      1 sens2021565-b9
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

richelbilderbeek commented 2 years ago

This is an outlier, it is the only one that failed:

richel@N141CU:~/GitHubs/nsphs_ml_qt_results/issue_42$ cat 25_run_issue_42_M3d_p2_100.log
Parameters: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_100/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_100/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-17T13:12:16+0200
Running on computer with HOSTNAME: sens2021565-b28
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_rackham/25_run.R /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_100/experiment_params.csv'
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_100/experiment_params.csv
Running the GCAE experiment
Error: length(results_phe_filename) not equal to 1.
1/1 mismatches
[1] 0 - 1 == -1
In addition: Warning message:
In system2(command = run_args[1], args = run_args[-1], stdout = TRUE,  :
  running command ''python3' /opt/gcae_richel/run_gcae.py train --datadir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_100/ --data data_issue_42_M3d_p2_100 --model_id M3d --resume_from 70 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_100_ae/ --pheno_model_id p2 2>&1' had status 1
Execution halted
End time: 2022-05-17T14:07:51+0200
Duration: 3335 seconds

I've checked an the 21, 22 and 24 files all work.

Also the files that have a shorter and longer window work:

richel@N141CU:~/GitHubs/nsphs_ml_qt_results/issue_42$ cat 25_run_issue_42_M3d_p2_10.log
Parameters: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-17T13:12:09+0200
Running on computer with HOSTNAME: sens2021565-b43
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_rackham/25_run.R /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10/experiment_params.csv'
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10/experiment_params.csv
Running the GCAE experiment
Save the GCAE experiment results
$genotype_concordances_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//genotype_concordances.csv"

$losses_from_train_t_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//losses_from_train_t.csv"

$losses_from_train_v_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//losses_from_train_v.csv"

$nmse_in_time_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//nmse_in_time.csv"

$phenotype_predictions_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//phenotype_predictions.csv"

$scores_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//scores.csv"

$score_per_pop_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//score_per_pop.csv"

$train_times_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//train_times.csv"

Create the GCAE experiment results' plots
$genotype_concordances_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//genotype_concordances.png"

$losses_from_train_t_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//losses_from_train_t.png"

$losses_from_train_v_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//losses_from_train_v.png"

$nmse_in_time_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//nmse_in_time.png"

$phenotype_predictions_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//phenotype_predictions.png"

$scores_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//scores.png"

$score_per_pop_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//score_per_pop.png"

$train_times_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_10_ae//train_times.png"

Warning message:
In grSoftVersion() :
  unable to load shared object '/usr/local/lib/R/modules//R_X11.so':
  libXt.so.6: cannot open shared object file: No such file or directory
End time: 2022-05-18T03:43:08+0200
Duration: 52259 seconds
richel@N141CU:~/GitHubs/nsphs_ml_qt_results/issue_42$ cat 25_run_issue_42_M3d_p2_1000.log
Parameters: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-17T13:12:54+0200
Running on computer with HOSTNAME: sens2021565-b33
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_rackham/25_run.R /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000/experiment_params.csv'
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000/experiment_params.csv
Running the GCAE experiment
Save the GCAE experiment results
$genotype_concordances_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//genotype_concordances.csv"

$losses_from_train_t_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//losses_from_train_t.csv"

$losses_from_train_v_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//losses_from_train_v.csv"

$nmse_in_time_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//nmse_in_time.csv"

$phenotype_predictions_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//phenotype_predictions.csv"

$scores_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//scores.csv"

$score_per_pop_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//score_per_pop.csv"

$train_times_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//train_times.csv"

Create the GCAE experiment results' plots
$genotype_concordances_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//genotype_concordances.png"

$losses_from_train_t_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//losses_from_train_t.png"

$losses_from_train_v_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//losses_from_train_v.png"

$nmse_in_time_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//nmse_in_time.png"

$phenotype_predictions_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//phenotype_predictions.png"

$scores_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//scores.png"

$score_per_pop_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//score_per_pop.png"

$train_times_filename
[1] "/proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_42_M3d_p2_1000_ae//train_times.png"

Warning message:
In grSoftVersion() :
  unable to load shared object '/usr/local/lib/R/modules//R_X11.so':
  libXt.so.6: cannot open shared object file: No such file or directory
End time: 2022-05-18T17:58:53+0200
Duration: 103559 seconds

richelbilderbeek commented 2 years ago

The genotype concordance is usually low:

Screenshot from 2022-05-19 15-11-10

richelbilderbeek commented 2 years ago

I expected the NMSE to go down (i.e. get better), when the GC goes up (i.e. get better). What I find it is that there is trade-off. I don't believe in that trade-off at all, I'd say that the SNPs have gone missing in Generalization Land.

Screenshot from 2022-05-19 15-19-42

richelbilderbeek commented 2 years ago

Here, with super big points:

Screenshot from 2022-05-19 15-26-00

richelbilderbeek commented 2 years ago

M1 gives the best (highest) genotype concordance:

Screenshot from 2022-05-19 15-27-30

richelbilderbeek commented 2 years ago

For ~~M1~~ M3d, p1 gives the best/lowest NMSE:

Screenshot from 2022-05-19 15-29-23

richelbilderbeek commented 2 years ago

TODO:

re-create this figure, using r-squareds
re-create the last phenotype predictions (i.e. not with a phenotype prediction in time)

richelbilderbeek commented 2 years ago

[richel@sens2021565-bianca ~]$ sbatch nsphs_ml_qt/scripts_bianca/20_start_issue_42.sh 
Submitted batch job 19747
[richel@sens2021565-bianca ~]$ sbatch nsphs_ml_qt/scripts_bianca/20_start_issue_28.sh 
Submitted batch job 19748
[richel@sens2021565-bianca ~]$ sbatch nsphs_ml_qt/scripts_bianca/20_start_issue_29.sh 
Submitted batch job 19749
[richel@sens2021565-bianca ~]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             19749      core 20_start   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
             19748      core 20_start   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
             19747      core 20_start   richel PD       0:00      1 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)

richelbilderbeek commented 2 years ago

Running again:

Screenshot from 2022-06-15 11-19-07

richelbilderbeek commented 2 years ago

From notes of 2022-06-22 and from https://github.com/richelbilderbeek/nsphs_ml_qt_results/commit/1ebc49f996e4d983f3ffeebf3cf49a1a17ba0c8c, before splitting up per window_kb:

r_squared_per_model_combination

genotype_concordance_per_model_combination

genotype_concordance_to_nmse_per_model_combination

genotype_concordance_to_r_squared_per_model_combination

nmse_per_model_combination

model_id	pheno_model_id	window_kb	genotype_concordance	nmse	r_squared
M1	p1	1	0.4231527	1.6039901	0.4987305
M3d	p2	10	0.0433862	1.0016810	0.4000778
M3e	p2	1	0.2022988	2.2897811	0.2752620
M3f	p2	1	0.1909688	0.9991235	0.2358860
M3d	p1	1	0.2336617	0.9995123	0.1820836
M3e	p2	10	0.0455209	1.1819971	0.1376100
M3f	p2	10	0.0462324	1.0016713	0.1329283

richelbilderbeek commented 2 years ago

From notes of 2022-06-20:

Split per window_kb, with this commit:

genotype_concordance_to_r_squared_per_model_combination

genotype_concordance_per_model_combination

genotype_concordance_to_nmse_per_model_combination

r_squared_per_model_combination

nmse_per_model_combination

richelbilderbeek commented 2 years ago

The highest r_squared GCAE gave is 0.4987305 (for autoencoder M1 and phenotypic prediction architecture p1 and a window_size of 1 kb). Max r_squared that omicspred could do was 0.754:

Screenshot from 2022-06-20 06-24-56

`model_id`	`pheno_model_id`
M0	p0
M0	p1
M0	p2
M1	p0
M1	p1
M1	p2
M3d	p0
M3d	p1
M3d	p2
M3e	p0
M3e	p1
M3e	p2
M3f	p0
M3f	p1
M3f	p2

`model_id`	`pheno_model_id`
M0	p0
M0	p1
M0	p2
M1	p0
M1	p1
M1	p2
M3d	p0
M3d	p1
M3d	p2
M3e	p0
M3e	p1
M3e	p2
M3f	p0
M3f	p1
M3f	p2

`model_id`	`pheno_model_id`
M0	p0
M0	p1
M0	p2
M1	p0
M1	p1
M1	p2
M3d	p0
M3d	p1
M3d	p2
M3e	p0
M3e	p1
M3e	p2
M3f	p0
M3f	p1
M3f	p2