Closed richelbilderbeek closed 2 years ago
richelbilderbeek
commented
2 years ago
> t <- nsphsmlqt::get_kierczak_et_al_2022_table_s2_xlsx()
> stringr::str_subset(t$Protein, "17")
[1] "IL-17D" "CCL17" "CCL17" "CCL17" "CCL17" "IL-17RA" "IL-17RA"
[8] "IL-17RA" "IL-17RA" "IL-17RA" "IL-17RA" "IL-17RA" "IL-17RA" "IL-17RA"
> stringr::str_subset(t$Protein, "IL-17RA")
[1] "IL-17RA" "IL-17RA" "IL-17RA" "IL-17RA" "IL-17RA" "IL-17RA" "IL-17RA"
[8] "IL-17RA" "IL-17RA"No idea why it is Position. instead of Position
> dplyr::select(t[ stringr::str_which(t$Protein, "IL-17RA"), ], X.Primary.Conditional.hit, Protein, SNP, Chr, Position.)
# A tibble: 9 × 5
X.Primary.Conditional.hit Protein SNP Chr Position.
<chr> <chr> <chr> <dbl> <dbl>
1 "Conditional 5" IL-17RA 22:17477147 22 17477147
2 "Conditional 4" IL-17RA rs144322196 22 17580117
3 "Conditional 1" IL-17RA rs2241044 22 17581838
4 "Conditional 2" IL-17RA rs2241046 22 17586471
5 " Primary" IL-17RA rs4819959 22 17586631
6 "Conditional 3" IL-17RA rs879575 22 17589567
7 "Conditional 6" IL-17RA rs142591902 22 20274011
8 " Primary" IL-17RA 2:37984514 2 37984514
9 " Primary" IL-17RA rs73246078 4 49586851I will just focus on the primary signal on chromosome 22, conditional 5 and 6 are too far away in any case:
richelbilderbeek
commented
2 years ago [richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
803 core 21_creat richel CG 0:02 1 sens2021565-b9
807 core 21_creat richel CG 0:02 1 sens2021565-b9
788 core 22_creat richel CG 0:02 1 sens2021565-b10
791 core 21_creat richel CG 0:02 1 sens2021565-b10
795 core 21_creat richel CG 0:02 1 sens2021565-b10
799 core 21_creat richel CG 0:02 1 sens2021565-b10
789 core 25_run.s richel PD 0:00 1 (Dependency)
790 core 29_zip.s richel PD 0:00 1 (Dependency)
792 core 22_creat richel PD 0:00 1 (Dependency)
793 core 25_run.s richel PD 0:00 1 (Dependency)
794 core 29_zip.s richel PD 0:00 1 (Dependency)
796 core 22_creat richel PD 0:00 1 (Dependency)
797 core 25_run.s richel PD 0:00 1 (Dependency)
798 core 29_zip.s richel PD 0:00 1 (Dependency)
800 core 22_creat richel PD 0:00 1 (Dependency)
801 core 25_run.s richel PD 0:00 1 (Dependency)
802 core 29_zip.s richel PD 0:00 1 (Dependency)
804 core 22_creat richel PD 0:00 1 (Dependency)
805 core 25_run.s richel PD 0:00 1 (Dependency)
806 core 29_zip.s richel PD 0:00 1 (Dependency)
808 core 22_creat richel PD 0:00 1 (Dependency)
809 core 25_run.s richel PD 0:00 1 (Dependency)
810 core 29_zip.s richel PD 0:00 1 (Dependency)
811 core 21_creat richel PD 0:00 1 (Resources)
812 core 22_creat richel PD 0:00 1 (Dependency)
813 core 25_run.s richel PD 0:00 1 (Dependency)
814 core 29_zip.s richel PD 0:00 1 (Dependency)
815 core 21_creat richel PD 0:00 1 (Priority)
816 core 22_creat richel PD 0:00 1 (Dependency)
817 core 25_run.s richel PD 0:00 1 (Dependency)
818 core 29_zip.s richel PD 0:00 1 (Dependency)There we go again:
gcae_experiment_params_filename: /home/richel/data_issue_29_1000/experiment_params.csv
unique_id: issue_29_1000
jobid_21: 847
jobid_22: 848
jobid_25: 849
jobid_29: 850
End time: 2022-05-02T14:24:03+0200
Duration: 3 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
835 core 21_creat richel CG 0:05 1 sens2021565-b10
839 core 21_creat richel CG 0:05 1 sens2021565-b10
823 core 21_creat richel CG 0:04 1 sens2021565-b9
827 core 21_creat richel CG 0:05 1 sens2021565-b9
831 core 21_creat richel CG 0:04 1 sens2021565-b9
821 core 25_run.s richel PD 0:00 1 (Dependency)
822 core 29_zip.s richel PD 0:00 1 (Dependency)
824 core 22_creat richel PD 0:00 1 (Dependency)
825 core 25_run.s richel PD 0:00 1 (Dependency)
826 core 29_zip.s richel PD 0:00 1 (Dependency)
828 core 22_creat richel PD 0:00 1 (Dependency)
829 core 25_run.s richel PD 0:00 1 (Dependency)
830 core 29_zip.s richel PD 0:00 1 (Dependency)
832 core 22_creat richel PD 0:00 1 (Dependency)
833 core 25_run.s richel PD 0:00 1 (Dependency)
834 core 29_zip.s richel PD 0:00 1 (Dependency)
836 core 22_creat richel PD 0:00 1 (Dependency)
837 core 25_run.s richel PD 0:00 1 (Dependency)
838 core 29_zip.s richel PD 0:00 1 (Dependency)
840 core 22_creat richel PD 0:00 1 (Dependency)
841 core 25_run.s richel PD 0:00 1 (Dependency)
842 core 29_zip.s richel PD 0:00 1 (Dependency)
844 core 22_creat richel PD 0:00 1 (Dependency)
845 core 25_run.s richel PD 0:00 1 (Dependency)
846 core 29_zip.s richel PD 0:00 1 (Dependency)
848 core 22_creat richel PD 0:00 1 (Dependency)
849 core 25_run.s richel PD 0:00 1 (Dependency)
850 core 29_zip.s richel PD 0:00 1 (Dependency)
820 core 22_creat richel R 0:04 1 sens2021565-b10
843 core 21_creat richel R 0:04 1 sens2021565-b10
847 core 21_creat richel R 0:04 1 sens2021565-b10Now it is waiting until maintenance is over.
End time: 2022-05-02T14:28:10+0200
Duration: 15 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
821 core 25_run.s richel PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
845 core 25_run.s richel PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
849 core 25_run.s richel PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
825 core 25_run.s richel PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
829 core 25_run.s richel PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
833 core 25_run.s richel PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
837 core 25_run.s richel PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
841 core 25_run.s richel PD 0:00 1 (ReqNodeNotAvail, Reserved for maintenance)
822 core 29_zip.s richel PD 0:00 1 (Dependency)
826 core 29_zip.s richel PD 0:00 1 (Dependency)
830 core 29_zip.s richel PD 0:00 1 (Dependency)
834 core 29_zip.s richel PD 0:00 1 (Dependency)
838 core 29_zip.s richel PD 0:00 1 (Dependency)
842 core 29_zip.s richel PD 0:00 1 (Dependency)
846 core 29_zip.s richel PD 0:00 1 (Dependency)
850 core 29_zip.s richel PD 0:00 1 (Dependency)Now with #30:
End time: 2022-05-02T15:19:40+0200
Duration: 2 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
886 core 21_creat richel CG 0:04 1 sens2021565-b10
876 core 21_creat richel CG 0:04 1 sens2021565-b10
881 core 21_creat richel CG 0:04 1 sens2021565-b10
856 core 21_creat richel CG 0:04 1 sens2021565-b9
861 core 21_creat richel CG 0:04 1 sens2021565-b9
866 core 21_creat richel CG 0:05 1 sens2021565-b9
853 core 25_run.s richel PD 0:00 1 (Dependency)
854 core 26_assoc richel PD 0:00 1 (Dependency)
855 core 29_zip.s richel PD 0:00 1 (Dependency)
857 core 22_creat richel PD 0:00 1 (Dependency)
858 core 25_run.s richel PD 0:00 1 (Dependency)
859 core 26_assoc richel PD 0:00 1 (Dependency)
860 core 29_zip.s richel PD 0:00 1 (Dependency)
862 core 22_creat richel PD 0:00 1 (Dependency)
863 core 25_run.s richel PD 0:00 1 (Dependency)
864 core 26_assoc richel PD 0:00 1 (Dependency)
865 core 29_zip.s richel PD 0:00 1 (Dependency)
867 core 22_creat richel PD 0:00 1 (Dependency)
868 core 25_run.s richel PD 0:00 1 (Dependency)
869 core 26_assoc richel PD 0:00 1 (Dependency)
870 core 29_zip.s richel PD 0:00 1 (Dependency)
873 core 25_run.s richel PD 0:00 1 (Dependency)
874 core 26_assoc richel PD 0:00 1 (Dependency)
875 core 29_zip.s richel PD 0:00 1 (Dependency)
877 core 22_creat richel PD 0:00 1 (Dependency)
878 core 25_run.s richel PD 0:00 1 (Dependency)
879 core 26_assoc richel PD 0:00 1 (Dependency)
880 core 29_zip.s richel PD 0:00 1 (Dependency)
882 core 22_creat richel PD 0:00 1 (Dependency)
883 core 25_run.s richel PD 0:00 1 (Dependency)
884 core 26_assoc richel PD 0:00 1 (Dependency)
885 core 29_zip.s richel PD 0:00 1 (Dependency)
887 core 22_creat richel PD 0:00 1 (Dependency)
888 core 25_run.s richel PD 0:00 1 (Dependency)
889 core 26_assoc richel PD 0:00 1 (Dependency)
890 core 29_zip.s richel PD 0:00 1 (Dependency)
872 core 22_creat richel R 0:01 1 sens2021565-b11
852 core 22_creat richel R 0:06 1 sens2021565-b10Running again after maintenance:
End time: 2022-05-06T13:30:13+0200
Duration: 4 seconds
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1098 core 22_creat richel PD 0:00 1 (Dependency)
1097 core 21_creat richel PD 0:00 1 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)
1144 core 29_zip.s richel PD 0:00 1 (Dependency)
1143 core 26_assoc richel PD 0:00 1 (Dependency)
1142 core 25_run.s richel PD 0:00 1 (Dependency)
1141 core 24_creat richel PD 0:00 1 (Dependency)
1140 core 22_creat richel PD 0:00 1 (Dependency)
1139 core 21_creat richel PD 0:00 1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
1138 core 29_zip.s richel PD 0:00 1 (Dependency)
1137 core 26_assoc richel PD 0:00 1 (Dependency)
1136 core 25_run.s richel PD 0:00 1 (Dependency)
1135 core 24_creat richel PD 0:00 1 (Dependency)
1134 core 22_creat richel PD 0:00 1 (Dependency)
1133 core 21_creat richel PD 0:00 1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
1132 core 29_zip.s richel PD 0:00 1 (Dependency)
1131 core 26_assoc richel PD 0:00 1 (Dependency)
1130 core 25_run.s richel PD 0:00 1 (Dependency)
1129 core 24_creat richel PD 0:00 1 (Dependency)
1128 core 22_creat richel PD 0:00 1 (Dependency)
1127 core 21_creat richel PD 0:00 1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
1126 core 29_zip.s richel PD 0:00 1 (Dependency)
1125 core 26_assoc richel PD 0:00 1 (Dependency)
1124 core 25_run.s richel PD 0:00 1 (Dependency)
1123 core 24_creat richel PD 0:00 1 (Dependency)
1122 core 22_creat richel PD 0:00 1 (Dependency)
1121 core 21_creat richel PD 0:00 1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
1120 core 29_zip.s richel PD 0:00 1 (Dependency)
1119 core 26_assoc richel PD 0:00 1 (Dependency)
1118 core 25_run.s richel PD 0:00 1 (Dependency)
1117 core 24_creat richel PD 0:00 1 (Dependency)
1116 core 22_creat richel PD 0:00 1 (Dependency)
1115 core 21_creat richel PD 0:00 1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
1114 core 29_zip.s richel PD 0:00 1 (Dependency)
1113 core 26_assoc richel PD 0:00 1 (Dependency)
1112 core 25_run.s richel PD 0:00 1 (Dependency)
1111 core 24_creat richel PD 0:00 1 (Dependency)
1110 core 22_creat richel PD 0:00 1 (Dependency)
1109 core 21_creat richel PD 0:00 1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
1108 core 29_zip.s richel PD 0:00 1 (Dependency)
1107 core 26_assoc richel PD 0:00 1 (Dependency)
1106 core 25_run.s richel PD 0:00 1 (Dependency)
1105 core 24_creat richel PD 0:00 1 (Dependency)
1104 core 22_creat richel PD 0:00 1 (Dependency)
1103 core 21_creat richel PD 0:00 1 (ReqNodeNotAvail, UnavailableNodes:sens2021565-b[1-204])
1102 core 29_zip.s richel PD 0:00 1 (Dependency)
1101 core 26_assoc richel PD 0:00 1 (Dependency)
1100 core 25_run.s richel PD 0:00 1 (Dependency)
1099 core 24_creat richel PD 0:00 1 (Dependency)Running again!
richelbilderbeek
commented
2 years ago [richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/20_start_issue_29.sh
Starting time: 2022-05-10T09:54:04+0200
Running on computer with HOSTNAME: sens2021565-bianca.uppmax.uu.se
Running at location /home/richel
window_kb: 1
gcae_experiment_params_filename: /home/richel/data_issue_29_1/experiment_params.csv
unique_id: issue_29_1
jobid_21: 1391
jobid_22: 1392
jobid_24: 1393
jobid_25: 1394
jobid_26: 1395
jobid_29: 1396
window_kb: 10
gcae_experiment_params_filename: /home/richel/data_issue_29_10/experiment_params.csv
unique_id: issue_29_10
jobid_21: 1397
jobid_22: 1398
jobid_24: 1399
jobid_25: 1400
jobid_26: 1401
jobid_29: 1402
window_kb: 100
gcae_experiment_params_filename: /home/richel/data_issue_29_100/experiment_params.csv
unique_id: issue_29_100
jobid_21: 1403
jobid_22: 1404
jobid_24: 1405
jobid_25: 1406
jobid_26: 1407
jobid_29: 1408
window_kb: 1000
gcae_experiment_params_filename: /home/richel/data_issue_29_1000/experiment_params.csv
unique_id: issue_29_1000
jobid_21: 1409
jobid_22: 1410
jobid_24: 1411
jobid_25: 1412
jobid_26: 1413
jobid_29: 1414
End time: 2022-05-10T09:54:08+0200
Duration: 4 seconds[richel@sens2021565-bianca ~]$ du -h *.zip
1,4G issue_28_sensitive.zip
8,4M issue_28.zip
1,6G issue_29_sensitive.zip
8,4M issue_29.zipI would say equilibrium has not been reached yet:
richelbilderbeek
commented
2 years ago As the longest training time (for 25_) took 19 hours (67388 seconds to be exact), I can simply make the runs 10x as long. Do do.
richelbilderbeek
commented
2 years ago Re-run with 10k epochs:
[richel@sens2021565-bianca ~]$ ./nsphs_ml_qt/scripts_bianca/91_poll.sh
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1521 core 21_creat richel PD 0:00 1 (Priority)
1515 core 21_creat richel PD 0:00 1 (Priority)
1509 core 21_creat richel PD 0:00 1 (Priority)
1503 core 21_creat richel PD 0:00 1 (Priority)
1497 core 21_creat richel PD 0:00 1 (Priority)
1491 core 21_creat richel PD 0:00 1 (Priority)
1485 core 21_creat richel PD 0:00 1 (Priority)
1479 core 21_creat richel PD 0:00 1 (Priority)
1473 core 21_creat richel PD 0:00 1 (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)
1526 core 29_zip.s richel PD 0:00 1 (Dependency)
1525 core 26_assoc richel PD 0:00 1 (Dependency)
1524 core 25_run.s richel PD 0:00 1 (Dependency)
1523 core 24_creat richel PD 0:00 1 (Dependency)
1522 core 22_creat richel PD 0:00 1 (Dependency)
1520 core 29_zip.s richel PD 0:00 1 (Dependency)
1519 core 26_assoc richel PD 0:00 1 (Dependency)
1518 core 25_run.s richel PD 0:00 1 (Dependency)
1517 core 24_creat richel PD 0:00 1 (Dependency)
1516 core 22_creat richel PD 0:00 1 (Dependency)
1514 core 29_zip.s richel PD 0:00 1 (Dependency)
1513 core 26_assoc richel PD 0:00 1 (Dependency)
1512 core 25_run.s richel PD 0:00 1 (Dependency)
1511 core 24_creat richel PD 0:00 1 (Dependency)
1510 core 22_creat richel PD 0:00 1 (Dependency)
1508 core 29_zip.s richel PD 0:00 1 (Dependency)
1507 core 26_assoc richel PD 0:00 1 (Dependency)
1506 core 25_run.s richel PD 0:00 1 (Dependency)
1505 core 24_creat richel PD 0:00 1 (Dependency)
1504 core 22_creat richel PD 0:00 1 (Dependency)
1502 core 29_zip.s richel PD 0:00 1 (Dependency)
1501 core 26_assoc richel PD 0:00 1 (Dependency)
1500 core 25_run.s richel PD 0:00 1 (Dependency)
1499 core 24_creat richel PD 0:00 1 (Dependency)
1498 core 22_creat richel PD 0:00 1 (Dependency)
1496 core 29_zip.s richel PD 0:00 1 (Dependency)
1495 core 26_assoc richel PD 0:00 1 (Dependency)
1494 core 25_run.s richel PD 0:00 1 (Dependency)
1493 core 24_creat richel PD 0:00 1 (Dependency)
1492 core 22_creat richel PD 0:00 1 (Dependency)
1490 core 29_zip.s richel PD 0:00 1 (Dependency)
1489 core 26_assoc richel PD 0:00 1 (Dependency)
1488 core 25_run.s richel PD 0:00 1 (Dependency)
1487 core 24_creat richel PD 0:00 1 (Dependency)
1486 core 22_creat richel PD 0:00 1 (Dependency)
1484 core 29_zip.s richel PD 0:00 1 (Dependency)
1483 core 26_assoc richel PD 0:00 1 (Dependency)
1482 core 25_run.s richel PD 0:00 1 (Dependency)
1481 core 24_creat richel PD 0:00 1 (Dependency)
1480 core 22_creat richel PD 0:00 1 (Dependency)
1478 core 29_zip.s richel PD 0:00 1 (Dependency)
1477 core 26_assoc richel PD 0:00 1 (Dependency)
1476 core 25_run.s richel PD 0:00 1 (Dependency)
1475 core 24_creat richel PD 0:00 1 (Dependency)
1474 core 22_creat richel PD 0:00 1 (Dependency)For p0:
For p1:
Runtimes, both p0 and p1, hence the two dots of the same color per x coordinat:
richelbilderbeek
commented
2 years ago Rerun for (1) r_squareds and (2) re-create the last phenotype predictions (i.e. not with a phenotype prediction in time)
https://github.com/AJResearchGroup/richel/issues/169#issuecomment-1156010253
richelbilderbeek
commented
2 years ago Running again:
Now with the M3d autoencoder architecture.
richelbilderbeek
commented
2 years ago R-squareds, from this commit:
As discussed with Asa.