Validate dpLGAR using Streamflow

taddyb commented 1 year ago

We have to do the following to tune parameters

[x] Check the gradient chain
[x] Toy with max ponded depth
[x] Get streamflow data
[x] Test percolation
[x] See if params update
[ ] Train model

taddyb commented 1 year ago

Testing my mass balance with a different threshold: This is with 1e-6

[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - ********************************************************* 
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - -------------------- Simulation Summary ----------------- 
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - ------------------------ Mass balance ------------------- 
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Initial water in soil    =      45.115850 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Total precipitation      =       1.270000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Total infiltration       =       1.270000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Final water in soil      =      44.850618 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total AET                =       1.535232 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total PET                =       9.336486 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Global balance           =   1.255671e-10 cm
self.model.top_layer.print()
[2023-07-09 16:27:01,705][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
PyDev console: starting.

[2023-07-09 16:27:01,706][models.physics.layers.WettingFront][INFO] - [24.5311, 0.1816941172, 0, 0.149522, 0.000118, 1769.2941]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [26.4522, 0.1788332030, 0, 0.139832, 0.000103, 1837.9781]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [34.3777, 0.1765984955, 0, 0.076836, 0.000093, 1894.5824]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [38.7353, 0.1696611346, 0, 0.103690, 0.000065, 2089.0180]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

taddyb commented 1 year ago

This is a threshold of 1e-12

[2023-07-09 16:34:08,588][models.physics.MassBalance][INFO] - ********************************************************* 
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - -------------------- Simulation Summary ----------------- 
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - ------------------------ Mass balance ------------------- 
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Initial water in soil    =      45.115850 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total precipitation      =       1.270000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total infiltration       =       1.270000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Final water in soil      =      44.850691 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total AET                =       1.535160 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total PET                =       9.336486 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Global balance           =   1.308251e-10 cm
self.model.top_layer.print()
[2023-07-09 16:34:24,602][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
PyDev console: starting.
[2023-07-09 16:34:24,602][models.physics.layers.WettingFront][INFO] - [24.5357, 0.1816947524, 0, 0.149497, 0.000118, 1769.2792]
[2023-07-09 16:34:24,602][models.physics.layers.WettingFront][INFO] - [26.4578, 0.1788333258, 0, 0.139822, 0.000103, 1837.9750]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [34.3805, 0.1765990562, 0, 0.076830, 0.000093, 1894.5678]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [38.7417, 0.1696609414, 0, 0.103690, 0.000065, 2089.0238]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

It looks like the SE threshold is sensitive, this is something to note between versions of LGAR

taddyb commented 1 year ago

For context, this is the mass balance I want to benchmark against with LGAR-C:

[ (24.535741,0.18169475352642,1,1,0, 1.494965e-01, 0.000118 1769.27922230005083)
(26.457777,0.17883332607650,1,2,0, 1.398221e-01, 0.000103 1837.97502802446888)
(34.380516,0.17659905720241,1,3,0, 7.683014e-02, 0.000093 1894.56779938911541)
(38.741668,0.16966094100581,1,4,0, 1.036899e-01, 0.000065 2089.02381680663939)
(44.000000,0.16844630744222,1,5,1, 0.000000e+00, 0.000061 2126.31830323560462)
(175.000000,0.24912136632205,2,6,1, 0.000000e+00, 0.000001 2126.31830323560462)
(200.000000,0.17556960906662,3,7,1, 0.000000e+00, 0.000050 2126.31830323560507) ] 

********************************************************* 
-------------------- Simulation Summary ----------------- 
------------------------ Mass balance ------------------- 
Initial water in soil    =  45.1158503564 cm
Total precipitation      =   1.2700000000 cm
Total infiltration       =   1.2700000000 cm
Final water in soil      =  44.8506907205 cm
Surface ponded water     =   0.0000000000 cm
Surface runoff           =   0.0000000000 cm
GIUH runoff              =   0.0000000000 cm
Total percolation        =   0.0000000000 cm
Total AET                =   1.5351596358 cm
Total PET                =   9.3364856806 cm
Total discharge (Q)      =   0.0000000000 cm
Global balance           =   1.292975e-10 cm
Time                     =   0.2508 sec

taddyb commented 1 year ago

See PR #10 for why our mass balance is now different

taddyb commented 1 year ago

Setting ponded_depth_cm to be a nn.Parameter() since it is sensitive and varies from site to site.

Also, I changed the runoff calculation to this:

ponded_depth = ponded_depth_temp
_runoff_ = ponded_depth - infiltration
runoff = torch.clamp(_runoff_, min=0.0)

so there is always a runoff grad

taddyb commented 1 year ago

Debugging the updating soil parameters: Before update benchmark:

[2023-07-11 11:27:43,459][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [24.5357, 0.1816947524, 0, 0.149497, 0.000118, 1769.2792]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [26.4578, 0.1788333258, 0, 0.139822, 0.000103, 1837.9750]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [34.3805, 0.1765990562, 0, 0.076830, 0.000093, 1894.5678]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [38.7417, 0.1696609414, 0, 0.103690, 0.000065, 2089.0238]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-11 11:27:43,462][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-11 11:27:43,462][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

taddyb commented 1 year ago

Parameter updates work:

[2023-07-12 10:18:52,848][models.dpLGAR][INFO] - Alpha for soil 1: 0.0021
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Alpha for soil 2: 0.0073
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Alpha for soil 3: 0.0037
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 1: 1.6868
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 2: 1.2980
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 3: 1.6151
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 1: 0.4510
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 2: 0.0690
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 3: 0.4500
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Max Ponded Depth: 0.0000

taddyb commented 1 year ago

See PR #11 For more information on parameter updates and training

taddyb commented 1 year ago

First run with the new data:

Epoch 1 Training:   1%|          | 745/87672 [02:40<16:31:49,  1.46it/s]>>> 
self.mass_balance.report_mass(self.model)
[2023-07-13 06:33:46,013][models.physics.MassBalance][INFO] - ********************************************************* 
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - -------------------- Simulation Summary ----------------- 
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - ------------------------ Mass balance ------------------- 
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Initial water in soil    =      15.487900 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total precipitation      =      16.406079 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total infiltration       =      16.406079 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Final water in soil      =      26.638837 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total AET                =       5.255142 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total PET                =       8.277200 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Global balance           =   1.748894e-10 cm

taddyb commented 1 year ago

There is a small discrepency when using hourly intervals vs 15 min intervals:

Epoch 1 Training:   4%|▍         | 745/17544 [03:58<95:44:02, 20.52s/it]>>> self.mass_balance.report_mass(self.model) 
[2023-07-13 06:40:39,385][models.physics.MassBalance][INFO] - ********************************************************* 
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - -------------------- Simulation Summary ----------------- 
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - ------------------------ Mass balance ------------------- 
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Initial water in soil    =      15.487900 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total precipitation      =      16.406079 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total infiltration       =      16.406079 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Final water in soil      =      26.446677 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total AET                =       5.447301 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Total PET                =       8.277200 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Global balance           =   6.616716e-10 cm

taddyb commented 1 year ago

See #15

taddyb commented 1 year ago

training the model using torchrun:

torchrun --nproc_per_node=1 --master_port=47800 dpLGAR/__main__.py ++nproc=1 ++save_name=debug_inplace

taddyb commented 1 year ago

Starting to document on this issue again to make sure I don't forget things.

Currently, we're looking into getting an LSTM to work, and the normalization of inputs. Attributes look good (see below) , but forcings need work

taddyb commented 1 year ago

Updates from work today:

Switching back to an MLP away from the LSTM (for now) so that I can get results in the near term for AGU
- Going to be using all attributes (adding in bulk density too) in the MLP to predict alpha, n, ponded_water_max, theta_e, theta_r, and ksat
- This mimics what https://github.com/usda-ars-ussl/rosetta-soil/blob/main/rosetta/ANN_Module.py does
Data loading and normalization is complete
LGAR instantiation and MLP is good to go
Tomorrow i'll work on doing some training runs and testing.

NWC-CUAHSI-Summer-Institute / LGAR-py

Validate dpLGAR using Streamflow #8