search

USGS-R / river-dl

Deep learning model for predicting environmental variables on river systems
Creative Commons Zero v1.0 Universal
21 stars 15 forks source link

option to specify separate loss functions for pretrain and fine tune #123

Closed janetrbarclay closed 3 years ago

janetrbarclay commented 3 years ago

This PR adds the option to specify separate loss functions for the pretrain and the finetune epochs. If only one loss function is specified, it is used for both. This PR also moves the specification of the GW loss function into the snakefile (Snakefile_gw).

janetrbarclay commented 3 years ago

@jzwart , @jsadler2 , @SimonTopp - also, could one of you review this? Thanks!

SimonTopp commented 3 years ago

I'll take a look this afternoon!

janetrbarclay commented 3 years ago

@jzwart @SimonTopp @aappling-usgs I think today's commits address all the comments that were raised here and in #129. Let me know if you have any other thoughts / suggestions on this. Thanks!

jzwart commented 3 years ago

minor comment above, otherwise I think it looks good

janetrbarclay commented 3 years ago

Oh, okay, that makes sense. I think we should take it out (or comment it out if we want to leave it for future debugging of the gpu issue). I got it on a run the other day and it caused the whole run to fail (as was intended with the debugging). I wasn't able to reproduce it when I ran it again so I don't know if it would have otherwise recovered on the next batch or not, but I commented it out in one of my branches.

Janet Barclay U.S. Geological Survey New England Water Science Center Connecticut Office 101 Pitkin St. East Hartford, CT 06108

Phone (office) 860 291-6763 Fax 860 291-6799 Email @.**@*.**@*.***> https://www.usgs.gov/staff-profiles/janet-barclay

From: Simon Topp @.> Sent: Wednesday, August 25, 2021 11:00 AM To: USGS-R/river-dl @.> Cc: Barclay, Janet R @.>; State change @.> Subject: [EXTERNAL] Re: [USGS-R/river-dl] option to specify separate loss functions for pretrain and fine tune (#123)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

@SimonTopp commented on this pull request.

In river_dl/loss_functions.pyhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSGS-R%2Friver-dl%2Fpull%2F123%23discussion_r695841839&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf26fc1de5e04458688b608d967d97dcc%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637655006085651968%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FuDYweZvj0ghYj8OFm8h1JniboicOOx6JspbUCfHJpA%3D&reserved=0:

     rmse_Ar = rmse(Ar_obs,Ar_pred)

rmse_delPhi = rmse(delPhi_obs,delPhi_pred)

Haha, nah, I was referring to the assertion statement "Nans is a bad loss to have". I through that in there as a dummy error when troubleshooting the GPU issue. We can leave it, but should probably make it more informative if we do.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSGS-R%2Friver-dl%2Fpull%2F123%23discussion_r695841839&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf26fc1de5e04458688b608d967d97dcc%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637655006085661920%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pqlIeYJ8oT9K2ans4XBVB7xDyVY%2BsOo9lggeCJPgBzI%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA5H7UH3JZQ5W4C26CFOVG3T6UASDANCNFSM5A76SN6Q&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf26fc1de5e04458688b608d967d97dcc%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637655006085661920%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hRD2cQ5Ru6WdixYGI%2FM0wA9bOqqVPjhqKUWgzCd5UGM%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf26fc1de5e04458688b608d967d97dcc%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637655006085671878%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dtAGUGqk1osWc0hyeLYz29RqP95OnZOQZ3zAhxM2VuE%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26utm_campaign%3Dnotification-email&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf26fc1de5e04458688b608d967d97dcc%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637655006085671878%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ourtM3ckxy60wtTQHhBcfxOir8wz2us2ki49GIMeh4Y%3D&reserved=0.