Unable to train joint model from pretrained cpg and dna model with tensorflow backend

[snajder-r]

Hi,

I was able to train the joint model successfully. Then I wanted to train the cpg and dna model separately and the joint model on top. Training cpg and dna model worked fine, but I started having problems when trying to train the joint model from the two pretrained models.

The error I got was:

tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value cpg/bidirectional_1/forward_gru_1/bias
     [[{{node cpg/bidirectional_1/forward_gru_1/bias/read}}]]

I found that the cause was in dcpg_train.py in get_callbacks:

if K._BACKEND == 'tensorflow' and not opts.no_tensorboard:
    K.set_session(K.tf.Session(config=K.tf.ConfigProto(
        intra_op_parallelism_threads=1,
        inter_op_parallelism_threads=1)))

I don't have too much experience with tensorflow or Keras, but it looks to me that setting a new session causes the model to lose the loaded weights. Since this is only done when tensorboard is used, I was able to workaround the problem by running with --no_tensorboard.

Best, Rene

[cangermueller]

Thanks for letting me know. I will try to fix this. For now, I suggest running without tensorboard.

On Fri, Aug 30, 2019 at 2:03 AM snajder-r notifications@github.com wrote:

Hi,

I was able to train the joint model successfully. Then I wanted to train the cpg and dna model separately and the joint model on top. Training cpg and dna model worked fine, but I started having problems when trying to train the joint model from the two pretrained models.

The error I got was:

tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value cpg/bidirectional_1/forward_gru_1/bias [[{{node cpg/bidirectional_1/forward_gru_1/bias/read}}]]

I found that the cause was in dcpg_train.py in get_callbacks:

if K._BACKEND == 'tensorflow' and not opts.no_tensorboard: K.set_session(K.tf.Session(config=K.tf.ConfigProto( intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)))

I don't have too much experience with tensorflow or Keras, but it looks to me that setting a new session causes the model to lose the loaded weights. Since this is only done when tensorboard is used, I was able to workaround the problem by running with --no_tensorboard.

Best, Rene

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cangermueller/deepcpg/issues/28?email_source=notifications&email_token=ABEVZM75L7HVDQ3FGFPPXQLQHDO5DA5CNFSM4ISLBZC2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HIMWZLQ, or mute the thread https://github.com/notifications/unsubscribe-auth/ABEVZMZ6WY7NIKEOSIODTDLQHDO5DANCNFSM4ISLBZCQ .

-- Christof Angermueller cangermueller@gmail.com https://cangermueller.com