search

kausmees / GenoCAE

Convolutional autoencoder for genotype data
BSD 3-Clause "New" or "Revised" License
15 stars 10 forks source link

Use 'evaluate' without the `HO_superpopulation` file #24

Closed richelbilderbeek closed 2 years ago

richelbilderbeek commented 2 years ago

This is a note to self, as I cannot assign myself as I am not a Collaborator. Hence I assign myself in text :-)

richelbilderbeek commented 2 years ago

Now this fails with error (from this GHA log):

Traceback (most recent call last):
  File "/home/runner/.local/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1777, in genfromtxt
    fhd = iter(fid)
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_gcae.py", line 1616, in <module>
    main()
  File "run_gcae.py", line 1561, in main
    write_f1_scores_to_csv(results_directory, "epoch_{0}".format(epoch), superpopulations_file, f1_scores_by_pop, coords_by_pop)
  File "/home/runner/work/GenoCAE/GenoCAE/utils/visualization.py", line 603, in write_f1_scores_to_csv
    superpop_pop_dict = get_superpop_pop_dict(superpopulations_file)
  File "/home/runner/work/GenoCAE/GenoCAE/utils/data_handler.py", line 869, in get_superpop_pop_dict
    pop_superpop_list = get_pop_superpop_list(pop_superpop_file)
  File "/home/runner/work/GenoCAE/GenoCAE/utils/data_handler.py", line 626, in get_pop_superpop_list
    pop_superpop_list = np.genfromtxt(file, usecols=(0,1), dtype=str, delimiter=",")
  File "/home/runner/.local/lib/python3.8/site-packages/numpy/lib/npyio.py", line 1779, in genfromtxt
    raise TypeError(
TypeError: fname must be a string, filehandle, list of strings, or generator. Got <class 'NoneType'> instead.

I can imagine the problem is that I trained the CNN too short, all shapes were skipped and hence a division by zero (successfull shapes) occurred:

Imputing originally missing genotypes to most common value.
Evaluating epochs [1, 2, 3]
########################### epoch 1 ###########################
Too few for hull: 1
--  shape: (1, 2): skipping
Too few for hull: 10
--  shape: (1, 2): skipping
Too few for hull: 100
--  shape: (1, 2): skipping
Too few for hull: 101
# [more]
--  shape: (1, 2): skipping
Too few for hull: 97
--  shape: (1, 2): skipping
Too few for hull: 98
--  shape: (1, 2): skipping
Too few for hull: 99
--  shape: (1, 2): skipping
------ hull error : 0.0
------ f1 score with 3NN :0.1807228915662651
richelbilderbeek commented 2 years ago

Also 100 epochs is not sufficient. Trying 1000.

python3 run_gcae.py train --datadir example_tiny --data issue_6_bin --model_id M1  --epochs 1000 --save_interval 1  --train_opts_id ex3  --data_opts_id b_0_4 --pheno_model_id=p1 ; python3 run_gcae.py project --datadir example_tiny --data issue_6_bin --model_id M1 --train_opts_id ex3 --data_opts_id b_0_4 --pheno_model_id=p1 ; python3 run_gcae.py evaluate --metrics "hull_error,f1_score_3" --datadir example_tiny/ --trainedmodelname ae.M1.ex3.b_0_4.issue_6_bin.p1
richelbilderbeek commented 2 years ago

Hmmm, if I run for 1000 epochs with the HO_superpopulations file ...

python3 run_gcae.py train --datadir example_tiny --data issue_6_bin --model_id M1  --epochs 1000 --save_interval 1  --train_opts_id ex3  --data_opts_id b_0_4 --pheno_model_id=p1 ; python3 run_gcae.py project --datadir example_tiny --data issue_6_bin --model_id M1 --train_opts_id ex3 --data_opts_id b_0_4 --pheno_model_id=p1 ; python3 run_gcae.py evaluate --metrics "hull_error,f1_score_3" --datadir example_tiny/ --trainedmodelname ae.M1.ex3.b_0_4.issue_6_bin.p1

... I get a hint:

Too few for hull: Uzbek
--  shape: (1, 2): skipping
Too few for hull: Xibo
--  shape: (1, 2): skipping
Too few for hull: Yakut
--  shape: (1, 2): skipping
Too few for hull: Yemeni
--  shape: (1, 2): skipping
Too few for hull: Yi
--  shape: (1, 2): skipping
Too few for hull: Yoruba
--  shape: (1, 2): skipping
Too few for hull: Yukagir
--  shape: (1, 2): skipping
Too few for hull: Zapotec
--  shape: (1, 2): skipping
------ hull error : 0.0
------ f1 score with 3NN :0.1927710843373494

It means that there are too few individuals per ?family. Aha, it is in the README.md:

hull_error: for every population p: define the convex hull created by the points of samples of p. calculate the fraction that other population's samples make up of all the points inside the hull. the hull error is the average of this over populations.

richelbilderbeek commented 2 years ago

Nope, this has nothing to do with the hull error, as when I run evaluate with ...

python3 run_gcae.py evaluate --metrics "f1_score_3" --datadir example_tiny/ --trainedmodelname ae.M1.ex3.b_0_4.issue_6_bin.p1

I still get:

------ f1 score with 3NN :0.1927710843373494
Traceback (most recent call last):
  File "/home/richel/miniconda3/lib/python3.9/site-packages/numpy/lib/npyio.py", line 1796, in genfromtxt
    fhd = iter(fid)
TypeError: 'NoneType' object is not iterable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 1616, in <module>
    main()
  File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 1561, in main
    write_f1_scores_to_csv(results_directory, "epoch_{0}".format(epoch), superpopulations_file, f1_scores_by_pop, coords_by_pop)
  File "/home/richel/GitHubs/GenoCAE/utils/visualization.py", line 603, in write_f1_scores_to_csv
    superpop_pop_dict = get_superpop_pop_dict(superpopulations_file)
  File "/home/richel/GitHubs/GenoCAE/utils/data_handler.py", line 869, in get_superpop_pop_dict
    pop_superpop_list = get_pop_superpop_list(pop_superpop_file)
  File "/home/richel/GitHubs/GenoCAE/utils/data_handler.py", line 626, in get_pop_superpop_list
    pop_superpop_list = np.genfromtxt(file, usecols=(0,1), dtype=str, delimiter=",")
  File "/home/richel/miniconda3/lib/python3.9/site-packages/numpy/lib/npyio.py", line 1798, in genfromtxt
    raise TypeError(
TypeError: fname must be a string, filehandle, list of strings, or generator. Got <class 'NoneType'> instead.
richelbilderbeek commented 2 years ago

Ah, with the HO_superpop file it does work:

python3 run_gcae.py evaluate --metrics "f1_score_3" --datadir example_tiny/ --trainedmodelname ae.M1.ex3.b_0_4.issue_6_bin.p1       --superpops example_tiny/HO_superpopulations

Gives:

... 2021-12-09 17:51:55.681304: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory 2021-12-09 17:51:55.681334: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2021-12-09 17:51:58.451444: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected 2021-12-09 17:51:58.451471: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (N141CU): /proc/driver/nvidia/version does not exist 2021-12-09 17:51:58.451766: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. tensorflow version 2.7.0

__ arguments __ train : False datadir : example_tiny/ data : None model_id : None train_opts_id : None data_opts_id : None save_interval : None epochs : None resume_from : None trainedmodeldir : None pheno_model_id : None project : False superpops : example_tiny/HO_superpopulations epoch : None pdata : None trainedmodelname : ae.M1.ex3.b_0_4.issue_6_bin.p1 plot : False animate : False evaluate : True metrics : f1_score_3

__ data opts __ sparsifies : [0.0, 0.1, 0.2, 0.3, 0.4] norm_opts : {'flip': False, 'missing_val': -1.0} norm_mode : genotypewise01 impute_missing : True validation_split : 0.2

__ train opts __ learning_rate : 0.00032 batch_size : 10 noise_std : 0.0032 n_samples : -1 loss : {'module': 'tf.keras.losses', 'class': 'CategoricalCrossentropy', 'args': {'from_logits': False}} regularizer : {'reg_factor': 1e-07, 'module': 'tf.keras.regularizers', 'class': 'l2'} lr_scheme : {'module': 'tf.keras.optimizers.schedules', 'class': 'ExponentialDecay', 'args': {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}}

Imputing originally missing genotypes to most common value. Evaluating epochs [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000] ########################### epoch 1 ########################### ------ f1 score with 3NN :0.1927710843373494 writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_1.csv ########################### epoch 2 ########################### ------ f1 score with 3NN :0.1686746987951807 writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_2.csv ########################### epoch 3 ########################### ------ f1 score with 3NN :0.1927710843373494 writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_3.csv [...] ########################### epoch 998 ########################### ------ f1 score with 3NN :0.1807228915662651 writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_998.csv ########################### epoch 999 ########################### ------ f1 score with 3NN :0.1927710843373494 writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_999.csv ########################### epoch 1000 ########################### ------ f1 score with 3NN :0.18674698795180722 writing f1 score per pop to /home/richel/GitHubs/GenoCAE/ae_out/ae.M1.ex3.b_0_4.issue_6_bin.p1/issue_6_bin/f1_scores_pops_epoch_1000.csv

richelbilderbeek commented 2 years ago

Cannot do this, you really need that superpop file :-)