NoneType Blow up after 8000 timesteps

[taddyb]

What is going on:

It looks like the layer homogeneity is causing a problem when inserting water. When trying this code with another basin, it was found that this bug occurs when the wetting_front_free_drainage, or the deepest front effected by wetting, is the last layer.

When this happens, the code dies since you can't calculate a d_theta. This code wasn't built for a homogeneous soil column.

The fix is super sneaky... ~This implies there is some model blow up at this timestep~

Error Traceback:

(hydroRouting) tkb5476@e5-ce-csr-s2002:/mnt/sdb1/tkb5476/dpLGAR$ python -m dpLGAR ++save_name=small_top
[2023-07-13 22:40:13,793][models.dpLGAR][INFO] - Top Soil_depth = 15.0
[2023-07-13 22:40:13,811][agents.DifferentiableLGAR][INFO] - Running epoch: 0
[2023-07-13 22:40:13,811][agents.DifferentiableLGAR][INFO] - -----Current Params-----
[2023-07-13 22:40:13,811][models.dpLGAR][INFO] - Alpha for soil 1: 0.0100
[2023-07-13 22:40:13,811][models.dpLGAR][INFO] - Alpha for soil 2: 0.0100
[2023-07-13 22:40:13,811][models.dpLGAR][INFO] - n for soil 1: 1.6600
[2023-07-13 22:40:13,811][models.dpLGAR][INFO] - n for soil 2: 1.6600
[2023-07-13 22:40:13,811][models.dpLGAR][INFO] - Ksat for soil 1: 0.7560
[2023-07-13 22:40:13,811][models.dpLGAR][INFO] - Ksat for soil 2: 0.7560
[2023-07-13 22:40:13,811][models.dpLGAR][INFO] - Max Ponded Depth: 0.0000
Epoch 1 Training:  96%|████████████████████████████████████████████████████████████████▏  | 8393/8760 [22:36<00:59,  6.19it/s]
Error executing job with overrides: ['++save_name=small_top']
Traceback (most recent call last):
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/__main__.py", line 15, in main
    agent.run()
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/agents/DifferentiableLGAR.py", line 90, in run
    self.train()
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/agents/DifferentiableLGAR.py", line 106, in train
    self.train_one_epoch()
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/agents/DifferentiableLGAR.py", line 125, in train_one_epoch
    runoff, percolation = self.model(i, x.squeeze())
  File "/home/tkb5476/anaconda3/envs/hydroRouting/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/models/dpLGAR.py", line 266, in forward
    infiltration_sub, AET_sub = self.move_wetting_front(
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/models/dpLGAR.py", line 380, in move_wetting_front
    self.top_layer.wetting_fronts_cross_layer_boundary()
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/models/physics/layers/Layer.py", line 934, in wetting_fronts_cross_layer_boundary
    self.next_layer.wetting_fronts_cross_layer_boundary()
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/models/physics/layers/Layer.py", line 927, in wetting_fronts_cross_layer_boundary
    ) = self.recalibrate(
  File "/mnt/sdb1/tkb5476/dpLGAR/dpLGAR/models/physics/layers/Layer.py", line 980, in recalibrate
    next_theta_r = self.next_layer.attributes[
AttributeError: 'NoneType' object has no attribute 'attributes'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Steps to reproduce:

• Run dpLGAR from 01-01-23 to 12-31-23
• error occurs on 12-17-01
• for basin 01031500

[taddyb]

The code works for the whole month December for 2001 if run by itself. It looks to be a stability issue

[taddyb]

Here is the self.top_layer.print() that showed me there was an issue:

self.print()
[2023-07-17 14:30:01,932][models.physics.layers.Layer][INFO] - Printing the top_layer for rank: 4
[2023-07-17 14:30:01,933][models.physics.layers.Layer][INFO] - [     Depth       Theta   Layer_num   dzdt     k_cm_hr    psi   ]
[2023-07-17 14:30:01,933][models.physics.layers.WettingFront][INFO] - [10.0000, 0.1561597759, 0, 0.000000, 0.000000, 2011.4011]
[2023-07-17 14:30:01,934][models.physics.layers.WettingFront][INFO] - [25.0000, 0.1561597759, 1, 0.000000, 0.000000, 2011.4011]
[2023-07-17 14:30:01,934][models.physics.layers.WettingFront][INFO] - [109.2912, 0.1561597759, 2, 0.000000, 0.000000, 2011.4011]

If you look at the theta values, they are all the same. This means that inserting water will affect the whole column, which isn't what happens.

A proposed solution is to create a NN to estimate Van Genuchten params from soil %. This can give heterogeneity for the whole column!

[taddyb]

Closing this for now. Not sure how to address in the scope of this project