kausmees / GenoCAE

Convolutional autoencoder for genotype data
BSD 3-Clause "New" or "Revised" License
15 stars 10 forks source link

What is the cause for 'ValueError: Dimensions must be equal'? #19

Closed richelbilderbeek closed 2 years ago

richelbilderbeek commented 2 years ago

Dear GenoCAE maintainers, hi @kausmees and @cnettel,

When I run the GenCAE experimental Pheno branch, I get an error of which I have no idea what to do with. Below the reprex.

Currently, the GitHub Actions script runs GenoCAE with the --help flag, showing the help successfully.

On my fork of GenoCAE in the GitHub Actions 'check.yaml' script, I added the following command to run:

python3 run_gcae.py train --datadir example_tiny --data issue_2_bin --model_id M1  --epochs 20 --save_interval 2  --train_opts_id ex3  --data_opts_id b_0_4 --pheno_model_id=p1

The --data issue_2_bin are the data files I supplied to Carl at this Issue and are already put in the example_tiny folder of my 'GenoCAE' fork.

GitHub Actions gives the following error:

ValueError: in user code:

    File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 424, in run_optimization  *
        loss_value += tf.math.reduce_sum(((-y_pred) * y_true)) * 1e-6

    ValueError: Dimensions must be equal, but are 2 and 4 for '{{node mul_21}} = Mul[T=DT_FLOAT](Neg_2, one_hot_2)' with input shapes: [2,4], [2,4,3].

Below is the full error log, which can also be found in this GitHub Actions log.

What does the error mean?

Thanks and cheers, Richel

Full error log

richel@N141CU:~/GitHubs/GenoCAE$ python3 run_gcae.py train --datadir example_tiny --data issue_2_bin --model_id M1  --epochs 20 --save_interval 2  --train_opts_id ex3  --data_opts_id b_0_4 --pheno_model_id=p1
2021-11-30 13:41:10.460286: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-11-30 13:41:10.460312: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-11-30 13:41:13.244831: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2021-11-30 13:41:13.244903: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (N141CU): /proc/driver/nvidia/version does not exist
2021-11-30 13:41:13.245186: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
tensorflow version 2.7.0

______________________________ arguments ______________________________
train : True
datadir : example_tiny
data : issue_2_bin
model_id : M1
train_opts_id : ex3
data_opts_id : b_0_4
save_interval : 2
epochs : 20
resume_from : None
trainedmodeldir : None
pheno_model_id : p1
project : False
superpops : None
epoch : None
pdata : None
trainedmodelname : None
plot : False
animate : False
evaluate : False
metrics : None

______________________________ data opts ______________________________
sparsifies : [0.0, 0.1, 0.2, 0.3, 0.4]
norm_opts : {'flip': False, 'missing_val': -1.0}
norm_mode : genotypewise01
impute_missing : True
validation_split : 0.2

______________________________ train opts ______________________________
learning_rate : 0.00032
batch_size : 10
noise_std : 0.0032
n_samples : -1
loss : {'module': 'tf.keras.losses', 'class': 'CategoricalCrossentropy', 'args': {'from_logits': False}}
regularizer : {'reg_factor': 1e-07, 'module': 'tf.keras.regularizers', 'class': 'l2'}
lr_scheme : {'module': 'tf.keras.optimizers.schedules', 'class': 'ExponentialDecay', 'args': {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}}
______________________________
Imputing originally missing genotypes to most common value.
Reading ind pop list from /home/richel/GitHubs/GenoCAE/example_tiny/issue_2_bin.fam
Reading ind pop list from /home/richel/GitHubs/GenoCAE/example_tiny/issue_2_bin.fam
Mapping files: 100%|███████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 362.20it/s]
Using learning rate schedule tf.keras.optimizers.schedules.ExponentialDecay with {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}

______________________________ Data ______________________________
N unique train samples: 800
--- training on : 800
N valid samples: 200
N markers: 4

______________________________ Building model ______________________________
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'strides': 1}
Adding layer: BatchNormalization: {}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: MaxPooling1D: {'pool_size': 5, 'strides': 2, 'padding': 'same'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu'}
Adding layer: BatchNormalization: {}
Adding layer: Flatten: {}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dense: {'units': 2, 'name': 'encoded'}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75, 'activation': 'elu'}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 16}
Adding layer: Reshape: {'target_shape': (2, 8), 'name': 'i_msvar'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu'}
Adding layer: BatchNormalization: {}
Adding layer: Reshape: {'target_shape': (2, 1, 8)}
Adding layer: UpSampling2D: {'size': (2, 1)}
Adding layer: Reshape: {'target_shape': (4, 8)}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu', 'name': 'nms'}
Adding layer: BatchNormalization: {}
Adding layer: Conv1D: {'filters': 1, 'kernel_size': 1, 'padding': 'same'}
Adding layer: Flatten: {'name': 'logits'}

______________________________ Building model ______________________________
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'strides': 1}
Adding layer: BatchNormalization: {}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: MaxPooling1D: {'pool_size': 5, 'strides': 2, 'padding': 'same'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same'}
Adding layer: BatchNormalization: {}
Adding layer: Flatten: {}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dense: {'units': 2, 'name': 'encoded'}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 16}
Adding layer: Reshape: {'target_shape': (2, 8), 'name': 'i_msvar'}
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same'}
Adding layer: BatchNormalization: {}
Adding layer: Reshape: {'target_shape': (2, 1, 8)}
Adding layer: UpSampling2D: {'size': (2, 1)}
Adding layer: Reshape: {'target_shape': (4, 8)}
Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
--- conv1d  filters: 8 kernel_size: 5
--- batch normalization
Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'name': 'nms'}
Adding layer: BatchNormalization: {}
Adding layer: Conv1D: {'filters': 1, 'kernel_size': 1, 'padding': 'same'}
Adding layer: Flatten: {'name': 'logits'}

______________________________ Building model ______________________________
Adding layer: Dense: {'units': 75}
Adding layer: LeakyReLU: {}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: LeakyReLU: {}
Adding layer: Dense: {'units': 75}
Adding layer: LeakyReLU: {}
Adding layer: Dropout: {'rate': 0.01}
Adding layer: Dense: {'units': 75}
Adding layer: LeakyReLU: {}
Adding layer: Dense: {'units': 1}
No marker specific variable.
ALLVARS [<tf.Variable 'autoencoder/conv1d/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/dense/kernel:0' shape=(16, 75) dtype=float32>, <tf.Variable 'autoencoder/dense/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_1/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_1/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder/dense_2/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_2/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_3/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_3/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_4/kernel:0' shape=(75, 16) dtype=float32>, <tf.Variable 'autoencoder/dense_4/bias:0' shape=(16,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 4) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 4) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/kernel:0' shape=(16, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/kernel:0' shape=(75, 16) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/bias:0' shape=(16,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 4) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 4) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_2/dense_11/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_2/dense_11/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_2/dense_12/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_2/dense_12/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_2/dense_13/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_2/dense_13/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_2/dense_14/kernel:0' shape=(75, 1) dtype=float32>, <tf.Variable 'autoencoder_2/dense_14/bias:0' shape=(1,) dtype=float32>] ###
Traceback (most recent call last):
  File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 1616, in <module>
    main()
  File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 1014, in main
    run_optimization(autoencoder, autoencoder2, optimizer, optimizer2, loss_func, input_init, targets_init, True, phenomodel=pheno_model, phenotargets=phenotargets_init)
  File "/home/richel/miniconda3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/richel/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1129, in autograph_handler
    raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

    File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 424, in run_optimization  *
        loss_value += tf.math.reduce_sum(((-y_pred) * y_true)) * 1e-6

    ValueError: Dimensions must be equal, but are 2 and 4 for '{{node mul_21}} = Mul[T=DT_FLOAT](Neg_2, one_hot_2)' with input shapes: [2,4], [2,4,3].
cnettel commented 2 years ago

Yeah, I realize that this line is adapted for models that output three separate genotype values in a softmax way, rather than a single allele frequency.

Since this whole extra loss_value is only a small additional regularization, one way to handle it in this branch would be to simply comment out lines 420-424. Then that component will always be zero, but that should be fine. The regularization in question was anywhere more called for in those model versions we used with softmax outputs.

Från: Richel Bilderbeek @.> Skickat: den 30 november 2021 13:51 Till: kausmees/GenoCAE @.> Kopia: Carl Nettelblad @.>; Mention @.> Ämne: [kausmees/GenoCAE] What is the cause for 'ValueError: Dimensions must be equal'? (Issue #19)

Dear GenoCAE maintainers, hi @kausmees https://github.com/kausmees and @cnettel https://github.com/cnettel ,

When I run the GenCAE experimental Pheno branch, I get an error of which I have no idea what to do with. Below the reprex.

Currently, the GitHub Actions script https://github.com/cnettel/GenoCAE/blob/master/.github/workflows/check.yaml#L29 runs GenoCAE with the --help flag, showing the help successfully.

On my fork of GenoCAE https://github.com/richelbilderbeek/GenoCAE in the GitHub Actions 'check.yaml' script https://github.com/richelbilderbeek/GenoCAE/blob/Pheno/.github/workflows/check.yaml#L35 , I added the following command to run:

python3 run_gcae.py train --datadir example_tiny --data issue_2_bin --model_id M1 --epochs 20 --save_interval 2 --train_opts_id ex3 --data_opts_id b_0_4 --pheno_model_id=p1

The --data issue_2_bin are the data files I supplied to Carl at this Issue https://github.com/richelbilderbeek/gcaer/issues/2#issuecomment-922850888 , where these can be downloaded as well.

GitHub Actions gives the following error:

ValueError: in user code:

File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 424, in run_optimization  *

    loss_value += tf.math.reduce_sum(((-y_pred) * y_true)) * 1e-6

ValueError: Dimensions must be equal, but are 2 and 4 for '{{node mul_21}} = Mul[T=DT_FLOAT](Neg_2, one_hot_2)' with input shapes: [2,4], [2,4,3].

Below is the full error log, which can also be found in this GitHub Actions log https://github.com/cnettel/GenoCAE/runs/4366466646?check_suite_focus=true#step:8:171 .

What does the error mean?

Thanks and cheers, Richel

Full error log

@.***:~/GitHubs/GenoCAE$ python3 run_gcae.py train --datadir example_tiny --data issue_2_bin --model_id M1 --epochs 20 --save_interval 2 --train_opts_id ex3 --data_opts_id b_0_4 --pheno_model_id=p1

2021-11-30 13:41:10.460286: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory

2021-11-30 13:41:10.460312: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

2021-11-30 13:41:13.244831: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected

2021-11-30 13:41:13.244903: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (N141CU): /proc/driver/nvidia/version does not exist

2021-11-30 13:41:13.245186: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

tensorflow version 2.7.0

__ arguments __

train : True

datadir : example_tiny

data : issue_2_bin

model_id : M1

train_opts_id : ex3

data_opts_id : b_0_4

save_interval : 2

epochs : 20

resume_from : None

trainedmodeldir : None

pheno_model_id : p1

project : False

superpops : None

epoch : None

pdata : None

trainedmodelname : None

plot : False

animate : False

evaluate : False

metrics : None

__ data opts __

sparsifies : [0.0, 0.1, 0.2, 0.3, 0.4]

norm_opts : {'flip': False, 'missing_val': -1.0}

norm_mode : genotypewise01

impute_missing : True

validation_split : 0.2

__ train opts __

learning_rate : 0.00032

batch_size : 10

noise_std : 0.0032

n_samples : -1

loss : {'module': 'tf.keras.losses', 'class': 'CategoricalCrossentropy', 'args': {'from_logits': False}}

regularizer : {'reg_factor': 1e-07, 'module': 'tf.keras.regularizers', 'class': 'l2'}

lr_scheme : {'module': 'tf.keras.optimizers.schedules', 'class': 'ExponentialDecay', 'args': {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}}


Imputing originally missing genotypes to most common value.

Reading ind pop list from /home/richel/GitHubs/GenoCAE/example_tiny/issue_2_bin.fam

Reading ind pop list from /home/richel/GitHubs/GenoCAE/example_tiny/issue_2_bin.fam

Mapping files: 100%|███████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 362.20it/s]

Using learning rate schedule tf.keras.optimizers.schedules.ExponentialDecay with {'decay_rate': 0.96, 'decay_steps': 100, 'staircase': False}

__ Data __

N unique train samples: 800

--- training on : 800

N valid samples: 200

N markers: 4

__ Building model __

Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'strides': 1}

Adding layer: BatchNormalization: {}

Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}

--- conv1d filters: 8 kernel_size: 5

--- batch normalization

--- conv1d filters: 8 kernel_size: 5

--- batch normalization

Adding layer: MaxPooling1D: {'pool_size': 5, 'strides': 2, 'padding': 'same'}

Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu'}

Adding layer: BatchNormalization: {}

Adding layer: Flatten: {}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 75}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 75, 'activation': 'elu'}

Adding layer: Dense: {'units': 2, 'name': 'encoded'}

Adding layer: Dense: {'units': 75, 'activation': 'elu'}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 75, 'activation': 'elu'}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 16}

Adding layer: Reshape: {'target_shape': (2, 8), 'name': 'i_msvar'}

Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu'}

Adding layer: BatchNormalization: {}

Adding layer: Reshape: {'target_shape': (2, 1, 8)}

Adding layer: UpSampling2D: {'size': (2, 1)}

Adding layer: Reshape: {'target_shape': (4, 8)}

Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}

--- conv1d filters: 8 kernel_size: 5

--- batch normalization

--- conv1d filters: 8 kernel_size: 5

--- batch normalization

Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'activation': 'elu', 'name': 'nms'}

Adding layer: BatchNormalization: {}

Adding layer: Conv1D: {'filters': 1, 'kernel_size': 1, 'padding': 'same'}

Adding layer: Flatten: {'name': 'logits'}

__ Building model __

Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'strides': 1}

Adding layer: BatchNormalization: {}

Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}

--- conv1d filters: 8 kernel_size: 5

--- batch normalization

--- conv1d filters: 8 kernel_size: 5

--- batch normalization

Adding layer: MaxPooling1D: {'pool_size': 5, 'strides': 2, 'padding': 'same'}

Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same'}

Adding layer: BatchNormalization: {}

Adding layer: Flatten: {}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 75}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 75}

Adding layer: Dense: {'units': 2, 'name': 'encoded'}

Adding layer: Dense: {'units': 75}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 75}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 16}

Adding layer: Reshape: {'target_shape': (2, 8), 'name': 'i_msvar'}

Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same'}

Adding layer: BatchNormalization: {}

Adding layer: Reshape: {'target_shape': (2, 1, 8)}

Adding layer: UpSampling2D: {'size': (2, 1)}

Adding layer: Reshape: {'target_shape': (4, 8)}

Adding layer: ResidualBlock2: {'filters': 8, 'kernel_size': 5}

--- conv1d filters: 8 kernel_size: 5

--- batch normalization

--- conv1d filters: 8 kernel_size: 5

--- batch normalization

Adding layer: Conv1D: {'filters': 8, 'kernel_size': 5, 'padding': 'same', 'name': 'nms'}

Adding layer: BatchNormalization: {}

Adding layer: Conv1D: {'filters': 1, 'kernel_size': 1, 'padding': 'same'}

Adding layer: Flatten: {'name': 'logits'}

__ Building model __

Adding layer: Dense: {'units': 75}

Adding layer: LeakyReLU: {}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 75}

Adding layer: LeakyReLU: {}

Adding layer: Dense: {'units': 75}

Adding layer: LeakyReLU: {}

Adding layer: Dropout: {'rate': 0.01}

Adding layer: Dense: {'units': 75}

Adding layer: LeakyReLU: {}

Adding layer: Dense: {'units': 1}

No marker specific variable.

ALLVARS [<tf.Variable 'autoencoder/conv1d/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_1/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_1/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/conv1d_2/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2/batch_normalization_2/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_3/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_3/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/dense/kernel:0' shape=(16, 75) dtype=float32>, <tf.Variable 'autoencoder/dense/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_1/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_1/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder/dense_2/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_2/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_3/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder/dense_3/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder/dense_4/kernel:0' shape=(75, 16) dtype=float32>, <tf.Variable 'autoencoder/dense_4/bias:0' shape=(16,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder/conv1d_4/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_4/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_5/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_5/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/conv1d_6/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/residual_block2_1/batch_normalization_6/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/batch_normalization_7/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder/conv1d_7/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 4) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 4) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/kernel:0' shape=(5, 3, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_8/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_8/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_9/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_9/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/conv1d_10/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_2/batch_normalization_10/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_11/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_11/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/kernel:0' shape=(16, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_5/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_6/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/kernel:0' shape=(75, 2) dtype=float32>, <tf.Variable 'autoencoder_1/encoded/bias:0' shape=(2,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_7/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_1/dense_8/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/kernel:0' shape=(75, 16) dtype=float32>, <tf.Variable 'autoencoder_1/dense_9/bias:0' shape=(16,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/kernel:0' shape=(5, 10, 8) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_12/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_12/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_13/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_13/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/conv1d_14/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/gamma:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/residual_block2_3/batch_normalization_14/beta:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/nms/kernel:0' shape=(5, 8, 8) dtype=float32>, <tf.Variable 'autoencoder_1/nms/bias:0' shape=(8,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/gamma:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/batch_normalization_15/beta:0' shape=(9,) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/kernel:0' shape=(1, 9, 1) dtype=float32>, <tf.Variable 'autoencoder_1/conv1d_15/bias:0' shape=(1,) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 4) dtype=float32>, <tf.Variable 'Variable:0' shape=(1, 4) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/kernel:0' shape=(2, 75) dtype=float32>, <tf.Variable 'autoencoder_2/dense_10/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_2/dense_11/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_2/dense_11/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_2/dense_12/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_2/dense_12/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_2/dense_13/kernel:0' shape=(75, 75) dtype=float32>, <tf.Variable 'autoencoder_2/dense_13/bias:0' shape=(75,) dtype=float32>, <tf.Variable 'autoencoder_2/dense_14/kernel:0' shape=(75, 1) dtype=float32>, <tf.Variable 'autoencoder_2/dense_14/bias:0' shape=(1,) dtype=float32>] ###

Traceback (most recent call last):

File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 1616, in

main()

File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 1014, in main

run_optimization(autoencoder, autoencoder2, optimizer, optimizer2, loss_func, input_init, targets_init, True, phenomodel=pheno_model, phenotargets=phenotargets_init)

File "/home/richel/miniconda3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler

raise e.with_traceback(filtered_tb) from None

File "/home/richel/miniconda3/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py", line 1129, in autograph_handler

raise e.ag_error_metadata.to_exception(e)

ValueError: in user code:

File "/home/richel/GitHubs/GenoCAE/run_gcae.py", line 424, in run_optimization  *

    loss_value += tf.math.reduce_sum(((-y_pred) * y_true)) * 1e-6

ValueError: Dimensions must be equal, but are 2 and 4 for '{{node mul_21}} = Mul[T=DT_FLOAT](Neg_2, one_hot_2)' with input shapes: [2,4], [2,4,3].

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kausmees/GenoCAE/issues/19 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBJKQMTUSRUF2KO5E7MZQDUOTCCVANCNFSM5JBXGT2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . https://github.com/notifications/beacon/ABBJKQM65LZG4EURIZ5IO7DUOTCCVA5CNFSM5JBXGT2KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4P44FNAQ.gif

richelbilderbeek commented 2 years ago

Thanks for the answer! I commented out the lines you mentioned. Let's see what the build does now :-)

richelbilderbeek commented 2 years ago

It works! Thanks @cnettel !