search

USGS-R / river-dl

Deep learning model for predicting environmental variables on river systems
Creative Commons Zero v1.0 Universal
21 stars 15 forks source link

adding as-run config file to the model output directory #140

Closed janetrbarclay closed 2 years ago

janetrbarclay commented 2 years ago

This adds a rule and function to the Snakemake file that saves the "as-run" config file as discussed in #137 . In addition to the contents of the config file that is submitted to snakemake at runtime, this function adds the git branch and git commit tag, the run date and the file date and size of each of the input files.

I renamed the distance matrix entry in the config file so that all input files have the word "file" in their name. (the function gets the file date / size for any config entries with "file" in the key.

At this point this doesn't include anything on the data except the filename, file date, and file size. Calculating basic stats on the input data seems like a good idea, but it might better fit as part of the data prep pipeline outside of river-dl (maybe with the tags for the data version as was suggested, which could then be included here) or it could easily be added to the prep_io_data rule. Thoughts?

janetrbarclay commented 2 years ago

@SimonTopp @jzwart @jsadler2 Any thoughts on this PR?

SimonTopp commented 2 years ago

I think this is a great way to improve our documentation. I think my only comment would be to move the function out of the snakefile and put it in one of the utilities (probably preproc_utils)

janetrbarclay commented 2 years ago

I wondered about that (both moving it our and putting it into that script). I can do that this morning.

Janet Barclay U.S. Geological Survey New England Water Science Center Connecticut Office 101 Pitkin St. East Hartford, CT 06108

Phone (office) 860 291-6763 Fax 860 291-6799 Email @.**@*.**@*.***> https://www.usgs.gov/staff-profiles/janet-barclay

From: Simon Topp @.> Sent: Tuesday, November 2, 2021 4:48 PM To: USGS-R/river-dl @.> Cc: Barclay, Janet R @.>; Author @.> Subject: [EXTERNAL] Re: [USGS-R/river-dl] adding as-run config file to the model output directory (PR #140)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

I think this is a great way to improve our documentation. I think my only comment would be to move the function out of the snakefile and put it in one of the utilities (probably preproc_utils)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSGS-R%2Friver-dl%2Fpull%2F140%23issuecomment-958154386&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf3ab7f9f94cf45eec3bc08d99e423075%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637714829392032234%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vtXozvwaktg4G0k6r%2FGF3CClHEhO1qI8VI0lSGgUONA%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA5H7UDF6KVXGD2Y55MU7SLUKBMCJANCNFSM5G6CUC6Q&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf3ab7f9f94cf45eec3bc08d99e423075%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637714829392042190%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=m7vDyw6dFmyVMCHIoxlOYQjPM11m9v38EcaYLBTFmEQ%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf3ab7f9f94cf45eec3bc08d99e423075%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637714829392042190%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ooPGmBUSytbeH6797MB1jMRo1rNc%2BHm2qebCDu6TTws%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cjbarclay%40usgs.gov%7Cf3ab7f9f94cf45eec3bc08d99e423075%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637714829392052148%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cgc3KNnkNWXoS%2B%2BzqxUeABKNcESrBBnZxSew89IxWP4%3D&reserved=0.