search

AJResearchGroup / nsphs_ml_qt

R package for nsphs_ml_qt
GNU General Public License v3.0
0 stars 1 forks source link

[RUNNING] Use window of 10k, 100k and 1Mb around SNP from #5 #28

Closed richelbilderbeek closed 2 years ago

richelbilderbeek commented 2 years ago

This Issue continues from #5.

As discussed with Asa.

richelbilderbeek commented 2 years ago 
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/20_start_issue_
20_start_issue_28.sh  20_start_issue_5.sh   
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/20_start_issue_28.sh 
Starting time: 2022-05-02T10:35:08+0200
Running on computer with HOSTNAME: sens2021565-bianca.uppmax.uu.se
Running at location /home/richel
window_kb: 1
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
unique_id: issue_28_1
jobid_21: 771
jobid_22: 772
jobid_25: 773
jobid_29: 774
window_kb: 10
gcae_experiment_params_filename: /home/richel/data_issue_28_10/experiment_params.csv
unique_id: issue_28_10
jobid_21: 775
jobid_22: 776
jobid_25: 777
jobid_29: 778
window_kb: 100
gcae_experiment_params_filename: /home/richel/data_issue_28_100/experiment_params.csv
unique_id: issue_28_100
jobid_21: 779
jobid_22: 780
jobid_25: 781
jobid_29: 782
window_kb: 1000
gcae_experiment_params_filename: /home/richel/data_issue_28_1000/experiment_params.csv
unique_id: issue_28_1000
jobid_21: 783
jobid_22: 784
jobid_25: 785
jobid_29: 786
End time: 2022-05-02T10:35:11+0200
Duration: 3 seconds

And:

[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               775      core 21_creat   richel CG       0:05      1 sens2021565-b10
               779      core 21_creat   richel CG       0:04      1 sens2021565-b10
               783      core 21_creat   richel CG       0:05      1 sens2021565-b10
               772      core 22_creat   richel PD       0:00      1 (Resources)
               773      core 25_run.s   richel PD       0:00      1 (Dependency)
               774      core 29_zip.s   richel PD       0:00      1 (Dependency)
               776      core 22_creat   richel PD       0:00      1 (Dependency)
               777      core 25_run.s   richel PD       0:00      1 (Dependency)
               778      core 29_zip.s   richel PD       0:00      1 (Dependency)
               780      core 22_creat   richel PD       0:00      1 (Dependency)
               781      core 25_run.s   richel PD       0:00      1 (Dependency)
               782      core 29_zip.s   richel PD       0:00      1 (Dependency)
               784      core 22_creat   richel PD       0:00      1 (Dependency)
               785      core 25_run.s   richel PD       0:00      1 (Dependency)
               786      core 29_zip.s   richel PD       0:00      1 (Dependency)
richelbilderbeek commented 2 years ago 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               773      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               777      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               781      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               785      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               774      core 29_zip.s   richel PD       0:00      1 (Dependency)
               778      core 29_zip.s   richel PD       0:00      1 (Dependency)
               782      core 29_zip.s   richel PD       0:00      1 (Dependency)
               786      core 29_zip.s   richel PD       0:00      1 (Dependency)
richelbilderbeek commented 2 years ago 
column_index: 1
snp: rs12126142
Error: is.numeric(window_kb) is not TRUE

`actual`:   FALSE
`expected`: TRUE 
Execution halted
End time: 2022-05-02T10:39:07+0200
Duration: 3 seconds
richelbilderbeek commented 2 years ago

28 and #29 are running again:

[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               803      core 21_creat   richel CG       0:02      1 sens2021565-b9
               807      core 21_creat   richel CG       0:02      1 sens2021565-b9
               788      core 22_creat   richel CG       0:02      1 sens2021565-b10
               791      core 21_creat   richel CG       0:02      1 sens2021565-b10
               795      core 21_creat   richel CG       0:02      1 sens2021565-b10
               799      core 21_creat   richel CG       0:02      1 sens2021565-b10
               789      core 25_run.s   richel PD       0:00      1 (Dependency)
               790      core 29_zip.s   richel PD       0:00      1 (Dependency)
               792      core 22_creat   richel PD       0:00      1 (Dependency)
               793      core 25_run.s   richel PD       0:00      1 (Dependency)
               794      core 29_zip.s   richel PD       0:00      1 (Dependency)
               796      core 22_creat   richel PD       0:00      1 (Dependency)
               797      core 25_run.s   richel PD       0:00      1 (Dependency)
               798      core 29_zip.s   richel PD       0:00      1 (Dependency)
               800      core 22_creat   richel PD       0:00      1 (Dependency)
               801      core 25_run.s   richel PD       0:00      1 (Dependency)
               802      core 29_zip.s   richel PD       0:00      1 (Dependency)
               804      core 22_creat   richel PD       0:00      1 (Dependency)
               805      core 25_run.s   richel PD       0:00      1 (Dependency)
               806      core 29_zip.s   richel PD       0:00      1 (Dependency)
               808      core 22_creat   richel PD       0:00      1 (Dependency)
               809      core 25_run.s   richel PD       0:00      1 (Dependency)
               810      core 29_zip.s   richel PD       0:00      1 (Dependency)
               811      core 21_creat   richel PD       0:00      1 (Resources)
               812      core 22_creat   richel PD       0:00      1 (Dependency)
               813      core 25_run.s   richel PD       0:00      1 (Dependency)
               814      core 29_zip.s   richel PD       0:00      1 (Dependency)
               815      core 21_creat   richel PD       0:00      1 (Priority)
               816      core 22_creat   richel PD       0:00      1 (Dependency)
               817      core 25_run.s   richel PD       0:00      1 (Dependency)
               818      core 29_zip.s   richel PD       0:00      1 (Dependency)
richelbilderbeek commented 2 years ago

There we go again:

gcae_experiment_params_filename: /home/richel/data_issue_29_1000/experiment_params.csv
unique_id: issue_29_1000
jobid_21: 847
jobid_22: 848
jobid_25: 849
jobid_29: 850
End time: 2022-05-02T14:24:03+0200
Duration: 3 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               835      core 21_creat   richel CG       0:05      1 sens2021565-b10
               839      core 21_creat   richel CG       0:05      1 sens2021565-b10
               823      core 21_creat   richel CG       0:04      1 sens2021565-b9
               827      core 21_creat   richel CG       0:05      1 sens2021565-b9
               831      core 21_creat   richel CG       0:04      1 sens2021565-b9
               821      core 25_run.s   richel PD       0:00      1 (Dependency)
               822      core 29_zip.s   richel PD       0:00      1 (Dependency)
               824      core 22_creat   richel PD       0:00      1 (Dependency)
               825      core 25_run.s   richel PD       0:00      1 (Dependency)
               826      core 29_zip.s   richel PD       0:00      1 (Dependency)
               828      core 22_creat   richel PD       0:00      1 (Dependency)
               829      core 25_run.s   richel PD       0:00      1 (Dependency)
               830      core 29_zip.s   richel PD       0:00      1 (Dependency)
               832      core 22_creat   richel PD       0:00      1 (Dependency)
               833      core 25_run.s   richel PD       0:00      1 (Dependency)
               834      core 29_zip.s   richel PD       0:00      1 (Dependency)
               836      core 22_creat   richel PD       0:00      1 (Dependency)
               837      core 25_run.s   richel PD       0:00      1 (Dependency)
               838      core 29_zip.s   richel PD       0:00      1 (Dependency)
               840      core 22_creat   richel PD       0:00      1 (Dependency)
               841      core 25_run.s   richel PD       0:00      1 (Dependency)
               842      core 29_zip.s   richel PD       0:00      1 (Dependency)
               844      core 22_creat   richel PD       0:00      1 (Dependency)
               845      core 25_run.s   richel PD       0:00      1 (Dependency)
               846      core 29_zip.s   richel PD       0:00      1 (Dependency)
               848      core 22_creat   richel PD       0:00      1 (Dependency)
               849      core 25_run.s   richel PD       0:00      1 (Dependency)
               850      core 29_zip.s   richel PD       0:00      1 (Dependency)
               820      core 22_creat   richel  R       0:04      1 sens2021565-b10
               843      core 21_creat   richel  R       0:04      1 sens2021565-b10
               847      core 21_creat   richel  R       0:04      1 sens2021565-b10
richelbilderbeek commented 2 years ago

Now it is waiting until maintenance is over.

End time: 2022-05-02T14:28:10+0200
Duration: 15 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               821      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               845      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               849      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               825      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               829      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               833      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               837      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               841      core 25_run.s   richel PD       0:00      1 (ReqNodeNotAvail, Reserved for maintenance)
               822      core 29_zip.s   richel PD       0:00      1 (Dependency)
               826      core 29_zip.s   richel PD       0:00      1 (Dependency)
               830      core 29_zip.s   richel PD       0:00      1 (Dependency)
               834      core 29_zip.s   richel PD       0:00      1 (Dependency)
               838      core 29_zip.s   richel PD       0:00      1 (Dependency)
               842      core 29_zip.s   richel PD       0:00      1 (Dependency)
               846      core 29_zip.s   richel PD       0:00      1 (Dependency)
               850      core 29_zip.s   richel PD       0:00      1 (Dependency)
richelbilderbeek commented 2 years ago

Now with #30:

End time: 2022-05-02T15:19:40+0200
Duration: 2 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               886      core 21_creat   richel CG       0:04      1 sens2021565-b10
               876      core 21_creat   richel CG       0:04      1 sens2021565-b10
               881      core 21_creat   richel CG       0:04      1 sens2021565-b10
               856      core 21_creat   richel CG       0:04      1 sens2021565-b9
               861      core 21_creat   richel CG       0:04      1 sens2021565-b9
               866      core 21_creat   richel CG       0:05      1 sens2021565-b9
               853      core 25_run.s   richel PD       0:00      1 (Dependency)
               854      core 26_assoc   richel PD       0:00      1 (Dependency)
               855      core 29_zip.s   richel PD       0:00      1 (Dependency)
               857      core 22_creat   richel PD       0:00      1 (Dependency)
               858      core 25_run.s   richel PD       0:00      1 (Dependency)
               859      core 26_assoc   richel PD       0:00      1 (Dependency)
               860      core 29_zip.s   richel PD       0:00      1 (Dependency)
               862      core 22_creat   richel PD       0:00      1 (Dependency)
               863      core 25_run.s   richel PD       0:00      1 (Dependency)
               864      core 26_assoc   richel PD       0:00      1 (Dependency)
               865      core 29_zip.s   richel PD       0:00      1 (Dependency)
               867      core 22_creat   richel PD       0:00      1 (Dependency)
               868      core 25_run.s   richel PD       0:00      1 (Dependency)
               869      core 26_assoc   richel PD       0:00      1 (Dependency)
               870      core 29_zip.s   richel PD       0:00      1 (Dependency)
               873      core 25_run.s   richel PD       0:00      1 (Dependency)
               874      core 26_assoc   richel PD       0:00      1 (Dependency)
               875      core 29_zip.s   richel PD       0:00      1 (Dependency)
               877      core 22_creat   richel PD       0:00      1 (Dependency)
               878      core 25_run.s   richel PD       0:00      1 (Dependency)
               879      core 26_assoc   richel PD       0:00      1 (Dependency)
               880      core 29_zip.s   richel PD       0:00      1 (Dependency)
               882      core 22_creat   richel PD       0:00      1 (Dependency)
               883      core 25_run.s   richel PD       0:00      1 (Dependency)
               884      core 26_assoc   richel PD       0:00      1 (Dependency)
               885      core 29_zip.s   richel PD       0:00      1 (Dependency)
               887      core 22_creat   richel PD       0:00      1 (Dependency)
               888      core 25_run.s   richel PD       0:00      1 (Dependency)
               889      core 26_assoc   richel PD       0:00      1 (Dependency)
               890      core 29_zip.s   richel PD       0:00      1 (Dependency)
               872      core 22_creat   richel  R       0:01      1 sens2021565-b11
               852      core 22_creat   richel  R       0:06      1 sens2021565-b10
richelbilderbeek commented 2 years ago

Running again after maintenance:

End time: 2022-05-06T13:30:13+0200
Duration: 4 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1098      core 22_creat   richel PD       0:00      1 (Dependency)
              1097      core 21_creat   richel PD       0:00      1 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)
              1144      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1143      core 26_assoc   richel PD       0:00      1 (Dependency)
              1142      core 25_run.s   richel PD       0:00      1 (Dependency)
              1141      core 24_creat   richel PD       0:00      1 (Dependency)
              1140      core 22_creat   richel PD       0:00      1 (Dependency)
              1139      core 21_creat   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
              1138      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1137      core 26_assoc   richel PD       0:00      1 (Dependency)
              1136      core 25_run.s   richel PD       0:00      1 (Dependency)
              1135      core 24_creat   richel PD       0:00      1 (Dependency)
              1134      core 22_creat   richel PD       0:00      1 (Dependency)
              1133      core 21_creat   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
              1132      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1131      core 26_assoc   richel PD       0:00      1 (Dependency)
              1130      core 25_run.s   richel PD       0:00      1 (Dependency)
              1129      core 24_creat   richel PD       0:00      1 (Dependency)
              1128      core 22_creat   richel PD       0:00      1 (Dependency)
              1127      core 21_creat   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
              1126      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1125      core 26_assoc   richel PD       0:00      1 (Dependency)
              1124      core 25_run.s   richel PD       0:00      1 (Dependency)
              1123      core 24_creat   richel PD       0:00      1 (Dependency)
              1122      core 22_creat   richel PD       0:00      1 (Dependency)
              1121      core 21_creat   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
              1120      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1119      core 26_assoc   richel PD       0:00      1 (Dependency)
              1118      core 25_run.s   richel PD       0:00      1 (Dependency)
              1117      core 24_creat   richel PD       0:00      1 (Dependency)
              1116      core 22_creat   richel PD       0:00      1 (Dependency)
              1115      core 21_creat   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
              1114      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1113      core 26_assoc   richel PD       0:00      1 (Dependency)
              1112      core 25_run.s   richel PD       0:00      1 (Dependency)
              1111      core 24_creat   richel PD       0:00      1 (Dependency)
              1110      core 22_creat   richel PD       0:00      1 (Dependency)
              1109      core 21_creat   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
              1108      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1107      core 26_assoc   richel PD       0:00      1 (Dependency)
              1106      core 25_run.s   richel PD       0:00      1 (Dependency)
              1105      core 24_creat   richel PD       0:00      1 (Dependency)
              1104      core 22_creat   richel PD       0:00      1 (Dependency)
              1103      core 21_creat   richel PD       0:00      1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
              1102      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1101      core 26_assoc   richel PD       0:00      1 (Dependency)
              1100      core 25_run.s   richel PD       0:00      1 (Dependency)
              1099      core 24_creat   richel PD       0:00      1 (Dependency)
richelbilderbeek commented 2 years ago 
[richel@sens2021565-bianca ~]$ cat 25_run_issue_28_1.log 
Parameters: /home/richel/data_issue_28_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-06T13:53:22+0200
Running on computer with HOSTNAME: sens2021565-b9
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_rackham/25_run.R /home/richel/data_issue_28_1/experiment_params.csv'
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
Running the GCAE experiment
Error in gcae_train_more(gcae_setup = gcae_experiment_params$gcae_setup,  : 
  'ae_out_subfolder' not found at path '/home/richel/data_issue_28_1_ae/ae.M1.ex3.b_0_4.data_issue_28_1.p0' 
gcae_setup$datadir: /home/richel/data_issue_28_1/
gcae_setup$data: data_issue_28_1
gcae_setup$superpops: 
gcae_setup$model_id: M1
gcae_setup$train_opts_id: ex3
gcae_setup$data_opts_id: b_0_4
gcae_setup$trainedmodeldir: /home/richel/data_issue_28_1_ae/
gcae_setup$pheno_model_id: p0
gcae_options$gcae_folder: /opt/gcae_richel
gcae_options$ormr_folder_name: python3
gcae_options$gcae_version: 1.0
gcae_options$python_version: 3.6
'args': 'train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0' 
Tip: you should be able to copy-paste the args :-)
Calls: <Anonymous> -> gcae_train_more
In addition: Warning message:
In system2(command = run_args[1], args = run_args[-1], stdout = TRUE,  :
  running command ''python3' /opt/gcae_richel/run_gcae.py train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0 2>&1' had status 1
Execution halted
End time: 2022-05-06T13:53:58+0200
Duration: 36 seconds
richelbilderbeek commented 2 years ago

Running the failed command:

[richel@sens2021565-bianca ~]$ singularity run nsphs_ml_qt/nsphs_ml_qt.sif  python3 /opt/gcae_richel/run_gcae.py train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0 2>&1
'nsphs_ml_qt.sif' running with arguments 'python3 /opt/gcae_richel/run_gcae.py train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0'
2022-05-09 08:32:52.598760: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2022-05-09 08:32:52.598825: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2022-05-09 08:32:52.598869: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (sens2021565-bianca.uppmax.uu.se): /proc/driver/nvidia/version does not exist
2022-05-09 08:32:52.703878: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-05-09 08:32:52.926539: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2394450000 Hz
2022-05-09 08:32:52.926799: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2b4728000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-05-09 08:32:52.926816: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
tensorflow version 2.2.0

______________________________ arguments ______________________________
train : True
datadir : /home/richel/data_issue_28_1/
data : data_issue_28_1
model_id : M1
train_opts_id : ex3
data_opts_id : b_0_4
save_interval : 10
epochs : 10
resume_from : 0
trainedmodeldir : /home/richel/data_issue_28_1_ae/
pheno_model_id : p0
project : False
superpops : None
epoch : None
pdata : None
trainedmodelname : None
plot : False
animate : False
evaluate : False
metrics : None

______________________________ data opts ______________________________
sparsifies : [0.0, 0.1, 0.2, 0.3, 0.4]
norm_opts : {'flip': False, 'missing_val': -1.0}
norm_mode : genotypewise01
impute_missing : True
validation_split : 0.2

______________________________ train opts ______________________________
learning_rate : 0.00032
batch_size : 10
noise_std : 0.0032
n_samples : -1
loss : {'module': 'tf.keras.losses', 'class': 'CategoricalCrossentropy', 'args': {'from_logits': False}}
regularizer : {'reg_factor': 1e-07, 'module': 'tf.keras.regularizers', 'class': 'l2'}
lr_scheme : {'module': 'tf.keras.optimizers.schedules', 'class': 'ExponentialDecay', 'args': {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}}
______________________________
Imputing originally missing genotypes to most common value.
Reading ind pop list from /home/richel/data_issue_28_1/data_issue_28_1.fam
Reading ind pop list from /home/richel/data_issue_28_1/data_issue_28_1.fam
Mapping files: 100%|████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 72.09it/s]
Using learning rate schedule tf.keras.optimizers.schedules.ExponentialDecay with {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}

______________________________ Data ______________________________
N unique train samples: 816
--- training on : 816
N valid samples: 205
N markers: 10

______________________________ Building model ______________________________
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'strides': 1}
Adding layer: BatchNormalization: {}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: MaxPooling1D: {'pool_size': 5, 'strides': 2, 'padding': 'same'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu'}
Adding layer: BatchNormalization: {}
Adding layer: Flatten: {}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dense: {'units': 2, 'name': 'encoded'}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 40}
Adding layer: Reshape: {'target_shape': (5, 8), 'name': 'i_msvar'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu'}
Adding layer: BatchNormalization: {}
Adding layer: Reshape: {'target_shape': (5, 1, 8)}
Adding layer: UpSampling2D: {'size': (2, 1)}
Adding layer: Reshape: {'target_shape': (10, 8)}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu', 'name': 'nms'}
Adding layer: BatchNormalization: {}
Adding layer: Conv1D: {'filters': 1, 'kernel_size': 1, 'padding': 'same'}
Adding layer: Flatten: {'name': 'logits'}

______________________________ Building model ______________________________
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'strides': 1}
Adding layer: BatchNormalization: {}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: MaxPooling1D: {'pool_size': 5, 'strides': 2, 'padding': 'same'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same'}
Adding layer: BatchNormalization: {}
Adding layer: Flatten: {}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dense: {'units': 2, 'name': 'encoded'}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 40}
Adding layer: Reshape: {'target_shape': (5, 8), 'name': 'i_msvar'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same'}
Adding layer: BatchNormalization: {}
Adding layer: Reshape: {'target_shape': (5, 1, 8)}
Adding layer: UpSampling2D: {'size': (2, 1)}
Adding layer: Reshape: {'target_shape': (10, 8)}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'name': 'nms'}
Adding layer: BatchNormalization: {}
Adding layer: Conv1D: {'filters': 1, 'kernel_size': 1, 'padding': 'same'}
Adding layer: Flatten: {'name': 'logits'}

______________________________ Building model ______________________________
Adding layer: Dense: {'units': 1}
No marker specific variable.
ALLVARS [<tf.Variable 'autoencoder/conv1d/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/dense/kernel:0' shape=(40, 75) dtype=float32>, <tf.Variable 'autoencoder/dense/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_1/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_1/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder/dense_2/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_2/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_3/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_3/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_4/kernel:0' shape=(75, 40) dtype=float32>, <tf.Variable 'autoencoder/dense_4/bias:0' shape=(40,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/kernel:0' shape=(40, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/kernel:0' shape=(75, 40) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/bias:0' shape=(40,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/kernel:0' shape=(2, 1) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/bias:0' shape=(1,) dtype=float32>] ###
ALLVARS [<tf.Variable 'autoencoder/conv1d/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/dense/kernel:0' shape=(40, 75) dtype=float32>, <tf.Variable 'autoencoder/dense/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_1/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_1/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder/dense_2/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_2/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_3/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_3/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_4/kernel:0' shape=(75, 40) dtype=float32>, <tf.Variable 'autoencoder/dense_4/bias:0' shape=(40,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/kernel:0' shape=(40, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/kernel:0' shape=(75, 40) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/bias:0' shape=(40,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/kernel:0' shape=(2, 1) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/bias:0' shape=(1,) dtype=float32>] ###
0.130096659 0 -0.00702388072 0 0.0251532793 0.0212626532 True 0 0.0945547298 0 1.00443554 0.263480335

______________________________ Train ______________________________
Model layers and dimensions:
-----------------------------
In DG.get_train_set: number of -1.0 genotypes in train: 0
In DG.get_train_set: number of -9 genotypes in train: 0
In DG.get_train_set: number of 0 values in train mask: 0
WARNING:tensorflow:Layer autoencoder is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

inputs shape (2, 10, 3)
layer 1
--- type: <class 'tensorflow.python.keras.layers.convolutional.Conv1D'>
--- shape: (2, 10, 8)
layer 2: batch_normalization (<class 'tensorflow.python.keras.layers.normalization_v2.BatchNormalization'>) 
--- shape: (2, 10, 8)
layer 3: res_block1 (<class 'utils.layers.ResidualBlock2'>) 
--- shape: (2, 10, 8)
layer 4: max_pooling1d (<class 'tensorflow.python.keras.layers.pooling.MaxPooling1D'>) 
--- shape: (2, 5, 8)
layer 5: conv1d_3 (<class 'tensorflow.python.keras.layers.convolutional.Conv1D'>) 
--- shape: (2, 5, 8)
layer 6: batch_normalization_3 (<class 'tensorflow.python.keras.layers.normalization_v2.BatchNormalization'>) 
--- shape: (2, 5, 8)
layer 7: flatten (<class 'tensorflow.python.keras.layers.core.Flatten'>) 
--- shape: (2, 40)
layer 8: dropout (<class 'tensorflow.python.keras.layers.core.Dropout'>) 
--- shape: (2, 40)
layer 9: dense (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 75)
layer 10: dropout_1 (<class 'tensorflow.python.keras.layers.core.Dropout'>) 
--- shape: (2, 75)
layer 11: dense_1 (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 75)
layer 12: encoded (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 2)
layer 13: dense_2 (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 75)
layer 14: dropout_2 (<class 'tensorflow.python.keras.layers.core.Dropout'>) 
--- shape: (2, 75)
layer 15: dense_3 (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 75)
layer 16: dropout_3 (<class 'tensorflow.python.keras.layers.core.Dropout'>) 
--- shape: (2, 75)
layer 17: dense_4 (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 40)
layer 18: i_msvar (<class 'tensorflow.python.keras.layers.core.Reshape'>) 
----- injecting marker-specific variable
ms var (1, 10)
ms tiled (2, 5, 2)
concatting: (2, 5, 8) (2, 5, 2)
--- shape: (2, 5, 10)
layer 19: conv1d_4 (<class 'tensorflow.python.keras.layers.convolutional.Conv1D'>) 
--- shape: (2, 5, 8)
layer 20: batch_normalization_4 (<class 'tensorflow.python.keras.layers.normalization_v2.BatchNormalization'>) 
--- shape: (2, 5, 8)
layer 21: reshape (<class 'tensorflow.python.keras.layers.core.Reshape'>) 
--- shape: (2, 5, 1, 8)
layer 22: up_sampling2d (<class 'tensorflow.python.keras.layers.convolutional.UpSampling2D'>) 
--- shape: (2, 10, 1, 8)
layer 23: reshape_1 (<class 'tensorflow.python.keras.layers.core.Reshape'>) 
--- shape: (2, 10, 8)
layer 24: res_block1 (<class 'utils.layers.ResidualBlock2'>) 
--- shape: (2, 10, 8)
layer 25: nms (<class 'tensorflow.python.keras.layers.convolutional.Conv1D'>) 
----- injecting marker-specific variable
ms var (1, 10)
ms tiled (2, 10, 1)
concatting: (2, 10, 8) (2, 10, 1)
--- shape: (2, 10, 9)
layer 26: batch_normalization_7 (<class 'tensorflow.python.keras.layers.normalization_v2.BatchNormalization'>) 
--- shape: (2, 10, 9)
layer 27: conv1d_7 (<class 'tensorflow.python.keras.layers.convolutional.Conv1D'>) 
--- shape: (2, 10, 1)
layer 28: logits (<class 'tensorflow.python.keras.layers.core.Flatten'>) 
--- shape: (2, 10)
Traceback (most recent call last):
  File "/opt/gcae_richel/run_gcae.py", line 1619, in <module>
    main()
  File "/opt/gcae_richel/run_gcae.py", line 1072, in main
    phenotargets = generatepheno(phenodata, poplist)
  File "/opt/gcae_richel/run_gcae.py", line 621, in generatepheno
    return tf.expand_dims(tf.convert_to_tensor([data.get((fam, name), None) for name, fam in poplist]), axis=-1)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1278, in convert_to_tensor_v2
    return convert_to_tensor(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1341, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 321, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 261, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Can't convert Python sequence with mixed types to Tensor.
richelbilderbeek commented 2 years ago

Huh, I don't use those labels anymore, yet they are used ...?

File "/opt/gcae_richel/run_gcae.py", line 621, in generatepheno
    return tf.expand_dims(tf.convert_to_tensor([data.get((fam, name), None) for name, fam in poplist]), axis=-1)
richelbilderbeek commented 2 years ago

This is the function:

def generatepheno(data, poplist):
    if data is None:
        return None
    return tf.expand_dims(tf.convert_to_tensor([data.get((fam, name), None) for name, fam in poplist]), axis=-1)

I guess it should have been ...

def generatepheno(data, poplist):
    if poplist is None:
        return None
    return tf.expand_dims(tf.convert_to_tensor([data.get((fam, name), None) for name, fam in poplist]), axis=-1)

... but well, the code had never been tested anyways.

Instead of fixing the code, I just going to put in the labels back again.

richelbilderbeek commented 2 years ago

Aha:

[richel@sens2021565-bianca ~]$ cat 22_create_issue_28_1_data.log 
Parameters: /home/richel/data_issue_28_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-06T13:49:05+0200
Running on computer with HOSTNAME: sens2021565-b9
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_bianca/22_create_issue_28_data.R /home/richel/data_issue_28_1/experiment_params.csv'
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
Parameters are valid
matches: 
 * /home/richel/data_issue_28_1/experiment_params.csv
 * /home/richel/data_issue_28_1/
 * /home/richel/data_
 * issue_28
 * 1
unique_id: issue_28
datadir: /home/richel/data_issue_28_1/
window_kb: 1
data: data_issue_28_1
base_input_filename: /home/richel/data_issue_28_1/data_issue_28_1
column_index: 1
snp: rs12126142
protein_name: CVD3_142_IL-6RA
experiment_base_filename: /home/richel/data_issue_28_1/data_issue_28_1
labels_filename: /home/richel/data_issue_28_1/data_issue_28_1_labels.csv
experiment_phe_filename: /home/richel/data_issue_28_1/data_issue_28_1.phe
#####################################################################
1. Select the SNPs
#####################################################################
input_data_basename: /proj/sens2021565/nobackup/NSPHS_data/NSPHS.WGS.hg38.plink1
input_plink_bin_filenames: 
 * /proj/sens2021565/nobackup/NSPHS_data/NSPHS.WGS.hg38.plink1.bed
 * /proj/sens2021565/nobackup/NSPHS_data/NSPHS.WGS.hg38.plink1.bim
 * /proj/sens2021565/nobackup/NSPHS_data/NSPHS.WGS.hg38.plink1.fam
Number of SNPs in .bed table: 10
Number of SNPs in .bim table: 10
Number of samples in .bed table: 1021
Number of samples in .fam table: 1021
Save data to 'experiment_base_filename'': /home/richel/data_issue_28_1/data_issue_28_1
$bed_filename
[1] "/home/richel/data_issue_28_1/data_issue_28_1.bed"

$bim_filename
[1] "/home/richel/data_issue_28_1/data_issue_28_1.bim"

$fam_filename
[1] "/home/richel/data_issue_28_1/data_issue_28_1.fam"

Done saving PLINK binary data to /home/richel/data_issue_28_1/data_issue_28_1
#####################################################################
2. Add FIDs to .fam table
#####################################################################
Set the FID to the first characters of the IID
Saving 'fam_table' to /home/richel/data_issue_28_1/data_issue_28_1.fam
Done saving 'fam_table' to /home/richel/data_issue_28_1/data_issue_28_1.fam
#####################################################################
3. Select the phenotypes
#####################################################################
Picking the table to use
Protein name 'CVD3_142_IL-6RA' must be present in the table
Creating unsorted 'phe_table' with NAs
Removing the NAs
Creating sorted 'phe_table'
Set the FID to the first characters of the IID
Saving 'phe_table' to /home/richel/data_issue_28_1/data_issue_28_1.phe
Done saving 'phe_table' to /home/richel/data_issue_28_1/data_issue_28_1.phe
#####################################################################
4. Resize the data
#####################################################################
Parameters are valid
Summary before resize
Error: all(file.exists((as.character(unlist(gcae_input_filenames))))) is not TRUE

`actual`:   FALSE
`expected`: TRUE 
Execution halted
End time: 2022-05-06T13:49:17+0200
Duration: 12 seconds
richelbilderbeek commented 2 years ago

Running again!

richelbilderbeek commented 2 years ago 
[richel@sens2021565-bianca ~]$ cat 25_run_issue_28_1.log 
Parameters: /home/richel/data_issue_28_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-09T11:44:00+0200
Running on computer with HOSTNAME: sens2021565-b9
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_rackham/25_run.R /home/richel/data_issue_28_1/experiment_params.csv'
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
Running the GCAE experiment
Error: file.exists(losses_from_project_filename) is not TRUE

`actual`:       
`expected`: TRUE
In addition: Warning messages:
1: In system2(command = run_args[1], args = run_args[-1], stdout = TRUE,  :
  running command ''python3' /opt/gcae_richel/run_gcae.py train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0 2>&1' had status 1
2: In system2(command = run_args[1], args = run_args[-1], stdout = TRUE,  :
  running command ''python3' /opt/gcae_richel/run_gcae.py project --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0 2>&1' had status 1
Execution halted
End time: 2022-05-09T11:45:26+0200
Duration: 86 seconds
richelbilderbeek commented 2 years ago

Running that Python command again:

[richel@sens2021565-bianca ~]$ ls
21_create_issue_28_1000_params.log  22_create_issue_29_10_data.log              data_issue_28_1000
21_create_issue_28_100_params.log   22_create_issue_29_1_data.log               data_issue_28_10_ae
21_create_issue_28_10_params.log    24_create_input_data_plots_issue_28_10.log  data_issue_28_1_ae
21_create_issue_28_1_params.log     24_create_input_data_plots_issue_28_1.log   data_issue_29_1
21_create_issue_29_1000_params.log  25_run_issue_28_10.log                      data_issue_29_10
21_create_issue_29_100_params.log   25_run_issue_28_1.log                       data_issue_29_100
21_create_issue_29_10_params.log    26_assoc_qt_issue_28_10.log                 data_issue_29_1000
21_create_issue_29_1_params.log     26_assoc_qt_issue_28_1.log                  nsphs_ml_qt
22_create_issue_28_1000_data.log    98_clean_bianca.sh                          README.md
22_create_issue_28_100_data.log     bin                                         richel-sens2021565
22_create_issue_28_10_data.log      data_issue_28_1                             script.R
22_create_issue_28_1_data.log       data_issue_28_10
22_create_issue_29_100_data.log     data_issue_28_100
[richel@sens2021565-bianca ~]$ cd data_issue_28_1_ae
[richel@sens2021565-bianca data_issue_28_1_ae]$ ls
ae.M1.ex3.b_0_4.data_issue_28_1.p0  assoc_qt.nosex      trait_value_box_plot.png      trait_value_histogram.png
assoc_qt.log                        assoc_qt.P1.qassoc  trait_value_density_plot.png
[richel@sens2021565-bianca data_issue_28_1_ae]$ cd ..
[richel@sens2021565-bianca ~]$ singularity run nsphs_ml_qt/nsphs_ml_qt.sif python3 /opt/gcae_richel/run_gcae.py train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0
'nsphs_ml_qt.sif' running with arguments 'python3 /opt/gcae_richel/run_gcae.py train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0'
2022-05-09 12:09:37.170860: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.singularity.d/libs
2022-05-09 12:09:37.170933: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2022-05-09 12:09:37.170974: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (sens2021565-bianca.uppmax.uu.se): /proc/driver/nvidia/version does not exist
2022-05-09 12:09:37.171580: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-05-09 12:09:37.182355: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2394450000 Hz
2022-05-09 12:09:37.182689: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2b7044000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-05-09 12:09:37.182707: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
tensorflow version 2.2.0

______________________________ arguments ______________________________
train : True
datadir : /home/richel/data_issue_28_1/
data : data_issue_28_1
model_id : M1
train_opts_id : ex3
data_opts_id : b_0_4
save_interval : 10
epochs : 10
resume_from : 0
trainedmodeldir : /home/richel/data_issue_28_1_ae/
pheno_model_id : p0
project : False
superpops : None
epoch : None
pdata : None
trainedmodelname : None
plot : False
animate : False
evaluate : False
metrics : None

______________________________ data opts ______________________________
sparsifies : [0.0, 0.1, 0.2, 0.3, 0.4]
norm_opts : {'flip': False, 'missing_val': -1.0}
norm_mode : genotypewise01
impute_missing : True
validation_split : 0.2

______________________________ train opts ______________________________
learning_rate : 0.00032
batch_size : 10
noise_std : 0.0032
n_samples : -1
loss : {'module': 'tf.keras.losses', 'class': 'CategoricalCrossentropy', 'args': {'from_logits': False}}
regularizer : {'reg_factor': 1e-07, 'module': 'tf.keras.regularizers', 'class': 'l2'}
lr_scheme : {'module': 'tf.keras.optimizers.schedules', 'class': 'ExponentialDecay', 'args': {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}}
______________________________
Imputing originally missing genotypes to most common value.
Reading ind pop list from /home/richel/data_issue_28_1/data_issue_28_1.fam
Reading ind pop list from /home/richel/data_issue_28_1/data_issue_28_1.fam
Mapping files: 100%|███████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 127.83it/s]
Using learning rate schedule tf.keras.optimizers.schedules.ExponentialDecay with {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}

______________________________ Data ______________________________
N unique train samples: 816
--- training on : 816
N valid samples: 205
N markers: 10

______________________________ Building model ______________________________
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'strides': 1}
Adding layer: BatchNormalization: {}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: MaxPooling1D: {'pool_size': 5, 'strides': 2, 'padding': 'same'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu'}
Adding layer: BatchNormalization: {}
Adding layer: Flatten: {}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dense: {'units': 2, 'name': 'encoded'}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 40}
Adding layer: Reshape: {'target_shape': (5, 8), 'name': 'i_msvar'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu'}
Adding layer: BatchNormalization: {}
Adding layer: Reshape: {'target_shape': (5, 1, 8)}
Adding layer: UpSampling2D: {'size': (2, 1)}
Adding layer: Reshape: {'target_shape': (10, 8)}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu', 'name': 'nms'}
Adding layer: BatchNormalization: {}
Adding layer: Conv1D: {'filters': 1, 'kernel_size': 1, 'padding': 'same'}
Adding layer: Flatten: {'name': 'logits'}

______________________________ Building model ______________________________
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'strides': 1}
Adding layer: BatchNormalization: {}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: MaxPooling1D: {'pool_size': 5, 'strides': 2, 'padding': 'same'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same'}
Adding layer: BatchNormalization: {}
Adding layer: Flatten: {}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dense: {'units': 2, 'name': 'encoded'}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 40}
Adding layer: Reshape: {'target_shape': (5, 8), 'name': 'i_msvar'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same'}
Adding layer: BatchNormalization: {}
Adding layer: Reshape: {'target_shape': (5, 1, 8)}
Adding layer: UpSampling2D: {'size': (2, 1)}
Adding layer: Reshape: {'target_shape': (10, 8)}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'name': 'nms'}
Adding layer: BatchNormalization: {}
Adding layer: Conv1D: {'filters': 1, 'kernel_size': 1, 'padding': 'same'}
Adding layer: Flatten: {'name': 'logits'}

______________________________ Building model ______________________________
Adding layer: Dense: {'units': 1}
No marker specific variable.
ALLVARS [<tf.Variable 'autoencoder/conv1d/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/dense/kernel:0' shape=(40, 75) dtype=float32>, <tf.Variable 'autoencoder/dense/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_1/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_1/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder/dense_2/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_2/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_3/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_3/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_4/kernel:0' shape=(75, 40) dtype=float32>, <tf.Variable 'autoencoder/dense_4/bias:0' shape=(40,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/kernel:0' shape=(40, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/kernel:0' shape=(75, 40) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/bias:0' shape=(40,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/kernel:0' shape=(2, 1) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/bias:0' shape=(1,) dtype=float32>] ###
ALLVARS [<tf.Variable 'autoencoder/conv1d/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/dense/kernel:0' shape=(40, 75) dtype=float32>, <tf.Variable 'autoencoder/dense/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_1/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_1/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder/dense_2/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_2/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_3/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_3/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_4/kernel:0' shape=(75, 40) dtype=float32>, <tf.Variable 'autoencoder/dense_4/bias:0' shape=(40,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/kernel:0' shape=(40, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/kernel:0' shape=(75, 40) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/bias:0' shape=(40,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 10) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/kernel:0' shape=(2, 1) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/bias:0' shape=(1,) dtype=float32>] ###
0.157929033 0 -0.00437371852 0 0.042737 0.0652555674 True 0 -0.0327524543 0 1.00254738 0.138199553

______________________________ Train ______________________________
Model layers and dimensions:
-----------------------------
In DG.get_train_set: number of -1.0 genotypes in train: 0
In DG.get_train_set: number of -9 genotypes in train: 0
In DG.get_train_set: number of 0 values in train mask: 0
WARNING:tensorflow:Layer autoencoder is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

inputs shape (2, 10, 3)
layer 1
--- type: <class 'tensorflow.python.keras.layers.convolutional.Conv1D'>
--- shape: (2, 10, 8)
layer 2: batch_normalization (<class 'tensorflow.python.keras.layers.normalization_v2.BatchNormalization'>) 
--- shape: (2, 10, 8)
layer 3: res_block1 (<class 'utils.layers.ResidualBlock2'>) 
--- shape: (2, 10, 8)
layer 4: max_pooling1d (<class 'tensorflow.python.keras.layers.pooling.MaxPooling1D'>) 
--- shape: (2, 5, 8)
layer 5: conv1d_3 (<class 'tensorflow.python.keras.layers.convolutional.Conv1D'>) 
--- shape: (2, 5, 8)
layer 6: batch_normalization_3 (<class 'tensorflow.python.keras.layers.normalization_v2.BatchNormalization'>) 
--- shape: (2, 5, 8)
layer 7: flatten (<class 'tensorflow.python.keras.layers.core.Flatten'>) 
--- shape: (2, 40)
layer 8: dropout (<class 'tensorflow.python.keras.layers.core.Dropout'>) 
--- shape: (2, 40)
layer 9: dense (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 75)
layer 10: dropout_1 (<class 'tensorflow.python.keras.layers.core.Dropout'>) 
--- shape: (2, 75)
layer 11: dense_1 (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 75)
layer 12: encoded (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 2)
layer 13: dense_2 (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 75)
layer 14: dropout_2 (<class 'tensorflow.python.keras.layers.core.Dropout'>) 
--- shape: (2, 75)
layer 15: dense_3 (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 75)
layer 16: dropout_3 (<class 'tensorflow.python.keras.layers.core.Dropout'>) 
--- shape: (2, 75)
layer 17: dense_4 (<class 'tensorflow.python.keras.layers.core.Dense'>) 
--- shape: (2, 40)
layer 18: i_msvar (<class 'tensorflow.python.keras.layers.core.Reshape'>) 
----- injecting marker-specific variable
ms var (1, 10)
ms tiled (2, 5, 2)
concatting: (2, 5, 8) (2, 5, 2)
--- shape: (2, 5, 10)
layer 19: conv1d_4 (<class 'tensorflow.python.keras.layers.convolutional.Conv1D'>) 
--- shape: (2, 5, 8)
layer 20: batch_normalization_4 (<class 'tensorflow.python.keras.layers.normalization_v2.BatchNormalization'>) 
--- shape: (2, 5, 8)
layer 21: reshape (<class 'tensorflow.python.keras.layers.core.Reshape'>) 
--- shape: (2, 5, 1, 8)
layer 22: up_sampling2d (<class 'tensorflow.python.keras.layers.convolutional.UpSampling2D'>) 
--- shape: (2, 10, 1, 8)
layer 23: reshape_1 (<class 'tensorflow.python.keras.layers.core.Reshape'>) 
--- shape: (2, 10, 8)
layer 24: res_block1 (<class 'utils.layers.ResidualBlock2'>) 
--- shape: (2, 10, 8)
layer 25: nms (<class 'tensorflow.python.keras.layers.convolutional.Conv1D'>) 
----- injecting marker-specific variable
ms var (1, 10)
ms tiled (2, 10, 1)
concatting: (2, 10, 8) (2, 10, 1)
--- shape: (2, 10, 9)
layer 26: batch_normalization_7 (<class 'tensorflow.python.keras.layers.normalization_v2.BatchNormalization'>) 
--- shape: (2, 10, 9)
layer 27: conv1d_7 (<class 'tensorflow.python.keras.layers.convolutional.Conv1D'>) 
--- shape: (2, 10, 1)
layer 28: logits (<class 'tensorflow.python.keras.layers.core.Flatten'>) 
--- shape: (2, 10)
Traceback (most recent call last):
  File "/opt/gcae_richel/run_gcae.py", line 1619, in <module>
    main()
  File "/opt/gcae_richel/run_gcae.py", line 1072, in main
    phenotargets = generatepheno(phenodata, poplist)
  File "/opt/gcae_richel/run_gcae.py", line 621, in generatepheno
    return tf.expand_dims(tf.convert_to_tensor([data.get((fam, name), None) for name, fam in poplist]), axis=-1)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1278, in convert_to_tensor_v2
    return convert_to_tensor(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1341, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 321, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 261, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Can't convert Python sequence with mixed types to Tensor.
richelbilderbeek commented 2 years ago

When running without --resume-from 0, there is the same error:

Traceback (most recent call last):
  File "/opt/gcae_richel/run_gcae.py", line 1619, in <module>
    main()
  File "/opt/gcae_richel/run_gcae.py", line 970, in main
    phenotargets_init = generatepheno(phenodata, poplist)
  File "/opt/gcae_richel/run_gcae.py", line 621, in generatepheno
    return tf.expand_dims(tf.convert_to_tensor([data.get((fam, name), None) for name, fam in poplist]), axis=-1)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1278, in convert_to_tensor_v2
    return convert_to_tensor(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1341, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 321, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 261, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Attempt to convert a value (None) with an unsupported type (<class 'NoneType'>) to a Tensor.
richelbilderbeek commented 2 years ago

Added verbosity and re-run.

richelbilderbeek commented 2 years ago

Could p0 be a problem?

[richel@sens2021565-bianca ~]$ cat 25_run_issue_28_1.log
Parameters: /home/richel/data_issue_28_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-09T12:33:54+0200
Running on computer with HOSTNAME: sens2021565-b9
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_rackham/25_run.R /home/richel/data_issue_28_1/experiment_params.csv'
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
Running the GCAE experiment
1/100
Running GCAE with arguments: 'train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0 
Tip: you should be able to copy-paste this :-)
Running: 'python3 /opt/gcae_richel/run_gcae.py train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0'. 
Tip: you should be able to copy-paste this :-)
GCAE output: 
2022-05-09 12:34:24.126959: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/lib/R/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server:/.singularity.d/libs
2022-05-09 12:34:24.126997: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2022-05-09 12:34:24.127096: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (sens2021565-b9.uppmax.uu.se): /proc/driver/nvidia/version does not exist
2022-05-09 12:34:24.127573: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2022-05-09 12:34:24.137727: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2394450000 Hz
2022-05-09 12:34:24.138072: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2b38d0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-05-09 12:34:24.138090: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
tensorflow version 2.2.0

______________________________ arguments ______________________________
train : True
datadir : /home/richel/data_issue_28_1/
data : data_issue_28_1
model_id : M1
train_opts_id : ex3
data_opts_id : b_0_4
save_interval : 10
epochs : 10
resume_from : 0
trainedmodeldir : /home/richel/data_issue_28_1_ae/
pheno_model_id : p0
project : False
superpops : None
epoch : None
pdata : None
trainedmodelname : None
plot : False
animate : False
evaluate : False
metrics : None

______________________________ data opts ______________________________
sparsifies : [0.0, 0.1, 0.2, 0.3, 0.4]
norm_opts : {'flip': False, 'missing_val': -1.0}
norm_mode : genotypewise01
impute_missing : True
validation_split : 0.2

______________________________ train opts ______________________________
learning_rate : 0.00032
batch_size : 10
noise_std : 0.0032
n_samples : -1
loss : {'module': 'tf.keras.losses', 'class': 'CategoricalCrossentropy', 'args': {'from_logits': False}}
regularizer : {'reg_factor': 1e-07, 'module': 'tf.keras.regularizers', 'class': 'l2'}
lr_scheme : {'module': 'tf.keras.optimizers.schedules', 'class': 'ExponentialDecay', 'args': {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}}
______________________________
Imputing originally missing genotypes to most common value.
Reading ind pop list from /home/richel/data_issue_28_1/data_issue_28_1.fam
Reading ind pop list from /home/richel/data_issue_28_1/data_issue_28_1.fam
Mapping files: 100%|██████████| 3/3 [00:00<00:00, 140.16it/s]
Traceback (most recent call last):
  File "/opt/gcae_richel/run_gcae.py", line 1619, in <module>
    main()
  File "/opt/gcae_richel/run_gcae.py", line 970, in main
    phenotargets_init = generatepheno(phenodata, poplist)
  File "/opt/gcae_richel/run_gcae.py", line 621, in generatepheno
    return tf.expand_dims(tf.convert_to_tensor([data.get((fam, name), None) for name, fam in poplist]), axis=-1)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1278, in convert_to_tensor_v2
    return convert_to_tensor(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1341, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 321, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 261, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 270, in _constant_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 96, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Attempt to convert a value (None) with an unsupported type (<class 'NoneType'>) to a Tensor.
Error in gcae_train_more(gcae_setup = gcae_experiment_params$gcae_setup,  : 
  'ae_out_subfolder' not found at path '/home/richel/data_issue_28_1_ae/ae.M1.ex3.b_0_4.data_issue_28_1.p0' 
gcae_setup$datadir: /home/richel/data_issue_28_1/
gcae_setup$data: data_issue_28_1
gcae_setup$superpops: 
gcae_setup$model_id: M1
gcae_setup$train_opts_id: ex3
gcae_setup$data_opts_id: b_0_4
gcae_setup$trainedmodeldir: /home/richel/data_issue_28_1_ae/
gcae_setup$pheno_model_id: p0
gcae_options$gcae_folder: /opt/gcae_richel
gcae_options$ormr_folder_name: python3
gcae_options$gcae_version: 1.0
gcae_options$python_version: 3.6
'args': 'train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0' 
Tip: you should be able to copy-paste the args :-)
Calls: <Anonymous> -> gcae_train_more
In addition: Warning message:
In system2(command = run_args[1], args = run_args[-1], stdout = TRUE,  :
  running command ''python3' /opt/gcae_richel/run_gcae.py train --datadir /home/richel/data_issue_28_1/ --data data_issue_28_1 --model_id M1 --resume_from 0 --epochs 10 --save_interval 10 --train_opts_id ex3 --data_opts_id b_0_4 --trainedmodeldir /home/richel/data_issue_28_1_ae/ --pheno_model_id p0 2>&1' had status 1
Execution halted
End time: 2022-05-09T12:34:27+0200
Duration: 33 seconds
richelbilderbeek commented 2 years ago

Hmm, p0 is there:

[richel@sens2021565-bianca ~]$ singularity shell nsphs_ml_qt/nsphs_ml_qt.sif 
Singularity> cd /opt
Singularity> ls
gcae  gcae_richel  pandoc  plinkr
Singularity> cd gcae_richel/
Singularity> ls
LICENSE.txt         data_opts       project_ae_alvis.sh        train_ae_alvis_inner.sh
README.md           docker          project_ae_alvis_inner.sh  train_opts
Singularity         example_tiny        requirements.txt           upload_singularity_container.sh
build_docker_container.sh   images          run_gcae.py            utils
build_docker_image.sh       launch_ae_alvis.sh  tips.md
build_singularity_container.sh  models          train_ae_alvis.sh
Singularity> cd models/
Singularity> ls
M0.json     M1.json M3d.json     M3e.json     M3f.json     M3j10U.json     M3j10X.json     p0.json
M0_1n.json  M1_1n.json  M3d_1n.json  M3e_1n.json  M3f_1n.json  M3j10U_1n.json  M3j10X_1n.json  p1.json
M0_2n.json  M1_2n.json  M3d_2n.json  M3e_2n.json  M3f_2n.json  M3j10U_2n.json  M3j10X_2n.json  p2.json
M0_3n.json  M1_3n.json  M3d_3n.json  M3e_3n.json  M3f_3n.json  M3j10U_3n.json  M3j10X_3n.json
M0_4n.json  M1_4n.json  M3d_4n.json  M3e_4n.json  M3f_4n.json  M3j10U_4n.json  M3j10X_4n.json
M0_5n.json  M1_5n.json  M3d_5n.json  M3e_5n.json  M3f_5n.json  M3j10U_5n.json  M3j10X_5n.json
richelbilderbeek commented 2 years ago

There is the problem, in 22_:

#####################################################################
4. Resize the data
#####################################################################
Parameters are valid
Summary before resize
Error in gcaer::read_gcae_input_files(gcae_input_filenames = gcae_input_filenames,  : 
  'read_gcae_input_files' cannot find file at path '' 
Calls: <Anonymous> -> <Anonymous>
Execution halted
End time: 2022-05-09T12:29:43+0200
Duration: 12 seconds
richelbilderbeek commented 2 years ago

Fix gcaer, re-run:

window_kb: 1000
gcae_experiment_params_filename: /home/richel/data_issue_28_1000/experiment_params.csv
unique_id: issue_28_1000
jobid_21: 1241
jobid_22: 1242
jobid_24: 1243
jobid_25: 1244
jobid_26: 1245
jobid_29: 1246
End time: 2022-05-09T13:25:52+0200
Duration: 4 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1229      core 21_creat   richel CG       0:05      1 sens2021565-b10
              1235      core 21_creat   richel CG       0:04      1 sens2021565-b10
              1241      core 21_creat   richel CG       0:04      1 sens2021565-b10
              1246      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1245      core 26_assoc   richel PD       0:00      1 (Dependency)
              1244      core 25_run.s   richel PD       0:00      1 (Dependency)
              1243      core 24_creat   richel PD       0:00      1 (Dependency)
              1242      core 22_creat   richel PD       0:00      1 (Dependency)
              1240      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1239      core 26_assoc   richel PD       0:00      1 (Dependency)
              1238      core 25_run.s   richel PD       0:00      1 (Dependency)
              1237      core 24_creat   richel PD       0:00      1 (Dependency)
              1236      core 22_creat   richel PD       0:00      1 (Dependency)
              1234      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1233      core 26_assoc   richel PD       0:00      1 (Dependency)
              1232      core 25_run.s   richel PD       0:00      1 (Dependency)
              1231      core 24_creat   richel PD       0:00      1 (Dependency)
              1230      core 22_creat   richel PD       0:00      1 (Dependency)
              1228      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1227      core 26_assoc   richel PD       0:00      1 (Dependency)
              1226      core 25_run.s   richel PD       0:00      1 (Dependency)
              1225      core 24_creat   richel PD       0:00      1 (Dependency)
              1224      core 22_creat   richel PD       0:00      1 (Resources)
richelbilderbeek commented 2 years ago 
#####################################################################
4. Resize the data
#####################################################################
Parameters are valid
Summary before resize
Error: file.exists(labels_filename) is not TRUE

`actual`:   FALSE
`expected`: TRUE 
Execution halted
End time: 2022-05-09T15:09:31+0200
Duration: 13 seconds
richelbilderbeek commented 2 years ago 
#####################################################################
4. Resize the data
#####################################################################
Parameters are valid
Summary before resize
Reading PLINK binary data, with basename /home/richel/data_issue_28_1/data_issue_28_1
Reading the labels table, with filename 
Error: file.exists(labels_filename) is not TRUE

`actual`:   FALSE
`expected`: TRUE 
Execution halted
End time: 2022-05-09T15:55:43+0200
Duration: 12 seconds
richelbilderbeek commented 2 years ago 
#####################################################################
4. Resize the data
#####################################################################
Parameters are valid
Summary before resize
$n_individuals_in_bed_table
[1] 1021

$n_snps_in_bed_table
[1] 10

$n_snps_in_bim_table
[1] 10

$n_individuals_in_fam_table
[1] 1021

$n_individuals_in_phe_table
[1] 890

Start resizing
Error: tibble::is_tibble(labels_table) is not TRUE

`actual`:   FALSE
`expected`: TRUE 
Execution halted
End time: 2022-05-10T09:18:24+0200
Duration: 16 seconds
richelbilderbeek commented 2 years ago

This works now:

[richel@sens2021565-bianca ~]$ cat 22_create_issue_28_1_data.log 
Parameters: /home/richel/data_issue_28_1/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
Starting time: 2022-05-10T09:51:54+0200
Running on computer with HOSTNAME: sens2021565-b9
Running at location /home/richel
'nsphs_ml_qt.sif' running with arguments 'Rscript nsphs_ml_qt/scripts_bianca/22_create_issue_28_data.R /home/richel/data_issue_28_1/experiment_params.csv'
gcae_experiment_params_filename: /home/richel/data_issue_28_1/experiment_params.csv
Parameters are valid
matches: 
 * /home/richel/data_issue_28_1/experiment_params.csv
 * /home/richel/data_issue_28_1/
 * /home/richel/data_
 * issue_28
 * 1
unique_id: issue_28
datadir: /home/richel/data_issue_28_1/
window_kb: 1
data: data_issue_28_1
base_input_filename: /home/richel/data_issue_28_1/data_issue_28_1
column_index: 1
snp: rs12126142
protein_name: CVD3_142_IL-6RA
experiment_base_filename: /home/richel/data_issue_28_1/data_issue_28_1
experiment_phe_filename: /home/richel/data_issue_28_1/data_issue_28_1.phe
#####################################################################
1. Select the SNPs
#####################################################################
input_data_basename: /proj/sens2021565/nobackup/NSPHS_data/NSPHS.WGS.hg38.plink1
input_plink_bin_filenames: 
 * /proj/sens2021565/nobackup/NSPHS_data/NSPHS.WGS.hg38.plink1.bed
 * /proj/sens2021565/nobackup/NSPHS_data/NSPHS.WGS.hg38.plink1.bim
 * /proj/sens2021565/nobackup/NSPHS_data/NSPHS.WGS.hg38.plink1.fam
Number of SNPs in .bed table: 10
Number of SNPs in .bim table: 10
Number of samples in .bed table: 1021
Number of samples in .fam table: 1021
Save data to 'experiment_base_filename'': /home/richel/data_issue_28_1/data_issue_28_1
$bed_filename
[1] "/home/richel/data_issue_28_1/data_issue_28_1.bed"

$bim_filename
[1] "/home/richel/data_issue_28_1/data_issue_28_1.bim"

$fam_filename
[1] "/home/richel/data_issue_28_1/data_issue_28_1.fam"

Done saving PLINK binary data to /home/richel/data_issue_28_1/data_issue_28_1
#####################################################################
2. Add FIDs to .fam table
#####################################################################
Set the FID to the first characters of the IID
Saving 'fam_table' to /home/richel/data_issue_28_1/data_issue_28_1.fam
Done saving 'fam_table' to /home/richel/data_issue_28_1/data_issue_28_1.fam
#####################################################################
3. Select the phenotypes
#####################################################################
Picking the table to use
Protein name 'CVD3_142_IL-6RA' must be present in the table
Creating unsorted 'phe_table' with NAs
Removing the NAs
Creating sorted 'phe_table'
Set the FID to the first characters of the IID
Saving 'phe_table' to /home/richel/data_issue_28_1/data_issue_28_1.phe
Done saving 'phe_table' to /home/richel/data_issue_28_1/data_issue_28_1.phe
#####################################################################
4. Resize the data
#####################################################################
Parameters are valid
Summary before resize
$n_individuals_in_bed_table
[1] 1021

$n_snps_in_bed_table
[1] 10

$n_snps_in_bim_table
[1] 10

$n_individuals_in_fam_table
[1] 1021

$n_individuals_in_phe_table
[1] 890

Start resizing
Summary after resize
$n_individuals_in_bed_table
[1] 870

$n_snps_in_bed_table
[1] 10

$n_snps_in_bim_table
[1] 10

$n_individuals_in_fam_table
[1] 870

$n_individuals_in_phe_table
[1] 870

Done resizing the data
End time: 2022-05-10T09:52:10+0200
Duration: 16 seconds
richelbilderbeek commented 2 years ago 
[richel@sens2021565-bianca ~]$ du -h *.zip
1,4G    issue_28_sensitive.zip
8,4M    issue_28.zip
1,6G    issue_29_sensitive.zip
8,4M    issue_29.zip
richelbilderbeek commented 2 years ago

I would say equilibrium has not been reached yet:

genotype_concordance_28_and_29_facet_grid

nmse_28_and_29_facet_grid

runtime_hours_28_and_29_1_plot

richelbilderbeek commented 2 years ago

runtime_hours_28_and_29_1_plot

richelbilderbeek commented 2 years ago

As the longest training time (for 25_) took 19 hours (67388 seconds to be exact), I can simply make the runs 10x as long. Do do.

richelbilderbeek commented 2 years ago

Re-run with 10k epochs:

[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              1521      core 21_creat   richel PD       0:00      1 (Priority)
              1515      core 21_creat   richel PD       0:00      1 (Priority)
              1509      core 21_creat   richel PD       0:00      1 (Priority)
              1503      core 21_creat   richel PD       0:00      1 (Priority)
              1497      core 21_creat   richel PD       0:00      1 (Priority)
              1491      core 21_creat   richel PD       0:00      1 (Priority)
              1485      core 21_creat   richel PD       0:00      1 (Priority)
              1479      core 21_creat   richel PD       0:00      1 (Priority)
              1473      core 21_creat   richel PD       0:00      1 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)
              1526      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1525      core 26_assoc   richel PD       0:00      1 (Dependency)
              1524      core 25_run.s   richel PD       0:00      1 (Dependency)
              1523      core 24_creat   richel PD       0:00      1 (Dependency)
              1522      core 22_creat   richel PD       0:00      1 (Dependency)
              1520      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1519      core 26_assoc   richel PD       0:00      1 (Dependency)
              1518      core 25_run.s   richel PD       0:00      1 (Dependency)
              1517      core 24_creat   richel PD       0:00      1 (Dependency)
              1516      core 22_creat   richel PD       0:00      1 (Dependency)
              1514      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1513      core 26_assoc   richel PD       0:00      1 (Dependency)
              1512      core 25_run.s   richel PD       0:00      1 (Dependency)
              1511      core 24_creat   richel PD       0:00      1 (Dependency)
              1510      core 22_creat   richel PD       0:00      1 (Dependency)
              1508      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1507      core 26_assoc   richel PD       0:00      1 (Dependency)
              1506      core 25_run.s   richel PD       0:00      1 (Dependency)
              1505      core 24_creat   richel PD       0:00      1 (Dependency)
              1504      core 22_creat   richel PD       0:00      1 (Dependency)
              1502      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1501      core 26_assoc   richel PD       0:00      1 (Dependency)
              1500      core 25_run.s   richel PD       0:00      1 (Dependency)
              1499      core 24_creat   richel PD       0:00      1 (Dependency)
              1498      core 22_creat   richel PD       0:00      1 (Dependency)
              1496      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1495      core 26_assoc   richel PD       0:00      1 (Dependency)
              1494      core 25_run.s   richel PD       0:00      1 (Dependency)
              1493      core 24_creat   richel PD       0:00      1 (Dependency)
              1492      core 22_creat   richel PD       0:00      1 (Dependency)
              1490      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1489      core 26_assoc   richel PD       0:00      1 (Dependency)
              1488      core 25_run.s   richel PD       0:00      1 (Dependency)
              1487      core 24_creat   richel PD       0:00      1 (Dependency)
              1486      core 22_creat   richel PD       0:00      1 (Dependency)
              1484      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1483      core 26_assoc   richel PD       0:00      1 (Dependency)
              1482      core 25_run.s   richel PD       0:00      1 (Dependency)
              1481      core 24_creat   richel PD       0:00      1 (Dependency)
              1480      core 22_creat   richel PD       0:00      1 (Dependency)
              1478      core 29_zip.s   richel PD       0:00      1 (Dependency)
              1477      core 26_assoc   richel PD       0:00      1 (Dependency)
              1476      core 25_run.s   richel PD       0:00      1 (Dependency)
              1475      core 24_creat   richel PD       0:00      1 (Dependency)
              1474      core 22_creat   richel PD       0:00      1 (Dependency)
richelbilderbeek commented 2 years ago

For p0:

genotype_concordance_28_and_29_facet_grid_p0

nmse_28_and_29_facet_grid_p0

For p1:

genotype_concordance_28_and_29_facet_grid_p1 nmse_28_and_29_facet_grid_p1

richelbilderbeek commented 2 years ago

Runtimes, both p0 and p1, hence the two dots of the same color per x coordinat:

runtime_hours_28_and_29_1_plot

richelbilderbeek commented 2 years ago

Re-run with r_squareds and only plot last phenotype prediction.

richelbilderbeek commented 2 years ago

Running again:

Screenshot from 2022-06-15 11-19-07

Now with the M3d autoencoder architecture.

richelbilderbeek commented 2 years ago 
@sens2021565-bianca ~]$ ls
20_start_issue_28.log                         25_run_issue_29_1000.log       zi26gfIH  ziKk3rr3
20_start_issue_29.log                         25_run_issue_29_100.log        zi3azOjo  ziKNgUop
20_start_issue_42.log                         25_run_issue_29_10.log         zi3n8Lp7  ziL19gBn
21_create_issue_28_1000_params.log            25_run_issue_29_1.log          zi3pawZb  zil9rlbU
21_create_issue_28_100_params.log             25_run_issue_50_1000.log       zi3XzGdp  ziLDd9RW
21_create_issue_28_10_params.log              25_run_issue_50_100.log        zi594ePz  ziLgOLJ7
21_create_issue_28_1_params.log               25_run_issue_50_10.log         zi5bzypp  zill4vPG
21_create_issue_29_1000_params.log            25_run_issue_50_1.log          zi5eCOCZ  ziLmc89g
21_create_issue_29_100_params.log             26_assoc_qt_issue_28_1000.log  zi72pfaQ  zilpMTaQ
21_create_issue_29_10_params.log              26_assoc_qt_issue_28_100.log   zi7R63ON  zilRXIV0
21_create_issue_29_1_params.log               26_assoc_qt_issue_28_10.log    zi7tncoj  zimKSdVO
21_create_issue_50_1000_params.log            26_assoc_qt_issue_28_1.log     zi8MLeFU  ziMrZF7D
21_create_issue_50_100_params.log             26_assoc_qt_issue_29_1000.log  zi8tsv2e  ziOF5dMM
21_create_issue_50_10_params.log              26_assoc_qt_issue_29_100.log   zi9vmG95  zioLJc9y
21_create_issue_50_1_params.log               26_assoc_qt_issue_29_10.log    ziaHpHew  ziOQ4fQG
22_create_issue_28_1000_data.log              26_assoc_qt_issue_29_1.log     ziaJnXtb  ziOTYHCz
22_create_issue_28_100_data.log               26_assoc_qt_issue_50_1000.log  zibbgNg1  ziOXHtdp
22_create_issue_28_10_data.log                26_assoc_qt_issue_50_100.log   ziBiieQs  ziPqKFT0
22_create_issue_28_1_data.log                 26_assoc_qt_issue_50_10.log    zibj03fg  ziqqz0Ak
22_create_issue_29_1000_data.log              26_assoc_qt_issue_50_1.log     zibVuSme  ziR79y7J
22_create_issue_29_100_data.log               29_zip_issue_28_100.log        zic3W5YX  ziRCNqCC
22_create_issue_29_10_data.log                29_zip_issue_28_10.log         ziCav7gK  zirkmBq9
22_create_issue_29_1_data.log                 29_zip_issue_28_1.log          zicmh8nb  ziSteqC2
22_create_issue_50_1000_data.log              29_zip_issue_29_100.log        ziD4bZdY  zit2GFVl
22_create_issue_50_100_data.log               29_zip_issue_29_10.log         ziDs37kr  zit3vBjQ
22_create_issue_50_10_data.log                29_zip_issue_29_1.log          zidxN12A  ziTeN1ji
22_create_issue_50_1_data.log                 98_clean_bianca.sh             zie2llWs  ziTgfh8l
24_create_input_data_plots_issue_28_1000.log  bin                            zifRxBe8  ziTZpHhn
24_create_input_data_plots_issue_28_100.log   issue_28_sensitive.zip         zifTlqOk  ziU0IIad
24_create_input_data_plots_issue_28_10.log    issue_28.zip                   zigf5gv4  ziwGXVF5
24_create_input_data_plots_issue_28_1.log     issue_29_sensitive.zip         ziH5BcvI  ziwoxZtG
24_create_input_data_plots_issue_29_1000.log  issue_29.zip                   ziHdefUt  ziwq8gFS
24_create_input_data_plots_issue_29_100.log   issue_42_sensitive.zip         zihGtu35  ziWYJW6T
24_create_input_data_plots_issue_29_10.log    issue_42.zip                   ziHq8Yv7  zixuQnod
24_create_input_data_plots_issue_29_1.log     n_jobs.txt                     ziIAwdF9  zixxubFi
24_create_input_data_plots_issue_50_1000.log  nsphs_ml_qt                    ziilEKBp  ziY2Y8Iw
24_create_input_data_plots_issue_50_100.log   nsphs_ml_qt_results            ziio5X5g  ziy9Yikp
24_create_input_data_plots_issue_50_10.log    README.md                      ziJ2QYHL  ziYLtAcI
24_create_input_data_plots_issue_50_1.log     richel-sens2021565             zijFKgPs  ziyQLQCR
25_run_issue_28_1000.log                      script.R                       ziJPYadI  ziZivQiE
25_run_issue_28_100.log                       zi006oLR                       ziK9zdyK
25_run_issue_28_10.log                        zi0PUhKY                       zikFs27x
25_run_issue_28_1.log                         zi1jZQbl                       ziKifm9a
richelbilderbeek commented 2 years ago

Something is wrong with the zipping:

richel@N141CU:~/GitHubs/nsphs_ml_qt_results/issue_28_1000_epochs_p1_m3d$ unzip issue_28.zip 
Archive:  issue_28.zip
  inflating: 21_create_issue_28_10_params.log  
  inflating: 29_zip_issue_28_1.log   
  inflating: 26_assoc_qt_issue_28_100.log  
  inflating: 25_run_issue_28_100.log  
  inflating: 24_create_input_data_plots_issue_28_100.log  
  inflating: 26_assoc_qt_issue_28_1000.log  
  inflating: 25_run_issue_28_1000.log  
  inflating: 24_create_input_data_plots_issue_28_1000.log  
  inflating: 26_assoc_qt_issue_28_10.log  
  inflating: 25_run_issue_28_10.log  
  inflating: 24_create_input_data_plots_issue_28_10.log  
  inflating: 22_create_issue_28_1000_data.log  
  inflating: 22_create_issue_28_100_data.log  
  inflating: 21_create_issue_28_100_params.log  
  inflating: 21_create_issue_28_1000_params.log  
  inflating: 24_create_input_data_plots_issue_28_1.log  
  inflating: 25_run_issue_28_1.log   
  inflating: 26_assoc_qt_issue_28_1.log  
  inflating: 21_create_issue_28_1_params.log  
  inflating: 20_start_issue_28.log   
  inflating: 22_create_issue_28_10_data.log  
  inflating: 22_create_issue_28_1_data.log  
  inflating: 29_zip_issue_28_10.log  
  inflating: 29_zip_issue_28_100.log  

richel@N141CU:~/GitHubs/nsphs_ml_qt_results/issue_28_1000_epochs_p1_m3d$ ls
20_start_issue_28.log               24_create_input_data_plots_issue_28_1000.log  26_assoc_qt_issue_28_100.log
21_create_issue_28_1000_params.log  24_create_input_data_plots_issue_28_100.log   26_assoc_qt_issue_28_10.log
21_create_issue_28_100_params.log   24_create_input_data_plots_issue_28_10.log    26_assoc_qt_issue_28_1.log
21_create_issue_28_10_params.log    24_create_input_data_plots_issue_28_1.log     29_zip_issue_28_100.log
21_create_issue_28_1_params.log     25_run_issue_28_1000.log                      29_zip_issue_28_10.log
22_create_issue_28_1000_data.log    25_run_issue_28_100.log                       29_zip_issue_28_1.log
22_create_issue_28_100_data.log     25_run_issue_28_10.log                        issue_28.zip
22_create_issue_28_10_data.log      25_run_issue_28_1.log
22_create_issue_28_1_data.log       26_assoc_qt_issue_28_1000.log
richelbilderbeek commented 2 years ago 
richel@N141CU:~/GitHubs/nsphs_ml_qt_results/issue_28_1000_epochs_p1_m3d$ cat 29_zip_issue_28_100.log

Parameters: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_28_100/experiment_params.csv
Number of parameters: 1
Correct number of arguments: 1
gcae_experiment_params_filename: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_28_100/experiment_params.csv
singularity_filename: nsphs_ml_qt/nsphs_ml_qt.sif
unique_id: issue_28
datadir: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_28_100/
trainedmodeldir: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_28_100_ae/
zip_filename: /home/richel/issue_28.zip
sensitive_zip_filename: /home/richel/issue_28_sensitive.zip
Starting time: 2022-06-16T01:48:57+0200
Running on computer with HOSTNAME: sens2021565-b35
Running at location /home/richel
datadir: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_28_100/
trainedmodeldir: /proj/sens2021565/nobackup/nsphs_ml_qt_results/data_issue_28_100_ae/
unique_id: issue_28
zip_filename: /home/richel/issue_28.zip
log_filenames: 21_create_issue_28_10_params.log
[...]
22_create_issue_28_10_data.log
22_create_issue_28_1_data.log
    zip warning: name not matched: data_issue_28_100
    zip warning: name not matched: data_issue_28_100_ae
updating: 21_create_issue_28_10_params.log (deflated 74%)
richelbilderbeek commented 2 years ago

190 mins remained:

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             25749      core 29_zip.s   richel PD       0:00      1 (Dependency)
             25747      core 25_run.s   richel  R    3:10:55      1 sens2021565-b9
richelbilderbeek commented 2 years ago 
[richel@sens2021565-bianca ~]$ sbatch nsphs_ml_qt/scripts_bianca/20_start_issue_28.sh 
Submitted batch job 25759
[richel@sens2021565-bianca ~]$ sbatch nsphs_ml_qt/scripts_bianca/20_start_issue_29.sh 
Submitted batch job 25773
[richel@sens2021565-bianca ~]$ sbatch nsphs_ml_qt/scripts_bianca/20_start_issue_42.sh 
Submitted batch job 25787
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll_jobs.sh 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             25760      core 21_creat   richel CG       0:06      1 sens2021565-b9
             25766      core 21_creat   richel CG       0:06      1 sens2021565-b9
             25772      core 21_creat   richel CG       0:06      1 sens2021565-b9

[...]

[richel@sens2021565-bianca ~]$ date
do jun 16 15:15:56 CEST 2022
richelbilderbeek commented 2 years ago

Still running :-)

richelbilderbeek commented 2 years ago

R-squareds, from this commit:

r_squared_28_and_29_facet_grid_p1