USGS-R / river-dl

Deep learning model for predicting environmental variables on river systems
Creative Commons Zero v1.0 Universal
21 stars 15 forks source link

InaccessibleTensorError when running RGCN with Dropout #151

Closed SimonTopp closed 2 years ago

SimonTopp commented 2 years ago

When training the RGCN on TG GPU with dropout, you get the error:

InaccessibleTensorError: The tensor 'Tensor("lstm_cell/cond/Identity:0", shape=(None, 7), dtype=float32)' cannot be accessed here: it is defined in another function or code block. Use return values, explicit Python locals or TensorFlow collections to access it. Defined in: FuncGraph(name=call, id=46914785443568); accessed from: FuncGraph(name=keras_graph, id=46914397049016).

You don't get this error when setting dropout to zero. I think I've fixed it by updating the dropout mask in the forward pass as described here, but wanted to double check if this is new behavior since the training routine update PR #141. @jsadler2, have you run into this in the past/did you ever experiment with dropout? The defaults are set to 0, so I'm wondering if this has just never come up before.

jsadler2 commented 2 years ago

I don't remember ever running into this. I also don't think that I ever tried dropout in the RGCN, now that I think of it. I might have only tried that in the plain LSTMs that I was using for the multi-task experiments.

jzwart commented 2 years ago

I also don't think we've run the RGCN with dropout >0. We've tested RGCN with state updating but I think just left dropout as default

SimonTopp commented 2 years ago

Fixed in #154.