Bug: Terminating: fork() called from a process already using GNU OpenMP, this is unsafe

I'm running into the follow problem using STT

If you've found a bug, please provide the following information:

Describe the bug Running the docker image on my computer raises the following error

WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2022-07-19 17:53:57.933430: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2022-07-19 17:53:57.935519: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
I Enabling automatic mixed precision training.
I STARTING Optimization
I Training epoch 0...
epoch: 0
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.

Also pressing CTRL+C does not work. I have to kill it from a separate terminal.

To Reproduce I used the container from ghcr.io/coqui-ai/stt-train:main I tried running STT by

python -m coqui_stt_training.train \
    --train_files $CLIPS/train.csv \
    --alphabet_config_path=$CLIPS/alphabet.txt \
    --train_batch_size 128 \
    --n_hidden 2048 \
    --learning_rate 0.0001 \
    --dropout_rate 0.40 \
    --epochs 100 \
    --cache_for_epochs 10 \
    --log_level 1 \
    --checkpoint_dir ${OUT}/checkpoints \
    --summary_dir ${OUT}/tensorboard \
    --train_cudnn true \
    --automatic_mixed_precision true \
    --show_progressbar false \
    --skip_batch_test true

Expected behavior A clear and concise description of what you expected to happen.

Environment (please complete the following information):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux release 7.9.2009 (Core)
TensorFlow installed from (our builds, or upstream TensorFlow): everything from inside the current container image
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory: NVIDIA A100 40GB memory
Exact command to reproduce:

Additional context Add any other context about the problem here.

I have the same. When i ran docker container i didn't experience it but when i ran it on the host i got it. I'm not using nvidia but i'm using opensuse tumbleweed

it just prints

(coqui-stt-train-venv) foidbgen@pc:~/Programs/coqui-train> python -m coqui_stt_training.train --checkpoint_dir coqui-stt-1.3.0-checkpoint --train_files train.csv --dev_files dev.csv --test_files test.csv
I --alphabet_config_path not specified, but found an alphabet file alongside specified checkpoint (coqui-stt-1.3.0-checkpoint/alphabet.txt). Will use this alphabet file for this run.
I Performing dummy training to check for memory problems.
I If the following process crashes, you likely have batch sizes that are too big for your available system memory (or GPU memory).
I Loading best validating checkpoint from coqui-stt-1.3.0-checkpoint/best_dev-3663913
I Loading variable from checkpoint: beta1_power
I Loading variable from checkpoint: beta2_power
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias/Adam_1
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel/Adam
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel/Adam_1
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/bias/Adam
I Loading variable from checkpoint: layer_1/bias/Adam_1
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_1/weights/Adam
I Loading variable from checkpoint: layer_1/weights/Adam_1
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/bias/Adam
I Loading variable from checkpoint: layer_2/bias/Adam_1
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_2/weights/Adam
I Loading variable from checkpoint: layer_2/weights/Adam_1
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/bias/Adam
I Loading variable from checkpoint: layer_3/bias/Adam_1
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_3/weights/Adam
I Loading variable from checkpoint: layer_3/weights/Adam_1
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/bias/Adam
I Loading variable from checkpoint: layer_5/bias/Adam_1
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_5/weights/Adam
I Loading variable from checkpoint: layer_5/weights/Adam_1
I Loading variable from checkpoint: layer_6/bias
I Loading variable from checkpoint: layer_6/bias/Adam
I Loading variable from checkpoint: layer_6/bias/Adam_1
I Loading variable from checkpoint: layer_6/weights
I Loading variable from checkpoint: layer_6/weights/Adam
I Loading variable from checkpoint: layer_6/weights/Adam_1
I Loading variable from checkpoint: learning_rate
I STARTING Optimization
Epoch 0 |   Training | Elapsed Time: 0:00:00 | Steps: 0 | Loss: 0.000000                                                                                                                                                                                                     Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.

and does nothing and also ctrl+c does not work

coqui-ai / STT

Bug: Terminating: fork() called from a process already using GNU OpenMP, this is unsafe #2268