search

USGS-R / river-dl

Deep learning model for predicting environmental variables on river systems
Creative Commons Zero v1.0 Universal
21 stars 15 forks source link

containerizing environment? #116

Closed jsadler2 closed 2 years ago

jsadler2 commented 3 years ago

We've all run into issues with the envionment giving us headaches. Should we move to a "river-dl" canonical container following @jzwart 's example in his pgdl-da work? This would hopefully help reduce the environment headaches.

SimonTopp commented 3 years ago

I'll do this.

SimonTopp commented 3 years ago

Github is apparently improving its container hosting/sharing capabilities. Might be worth looking into! https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry

SimonTopp commented 3 years ago

Update on this. I put together a singularity container which can run locally but still is having trouble finding the GPU drivers on Tallgrass. I've reached out to the HPC help desk and will hopefully have some answers soon.

janetrbarclay commented 3 years ago

I don't know if this helps with your issue, but I spent awhile on chat with the hpc helpdesk a week or so ago to solve issues with the path to the gpu drivers. To resolve it, we added this to the slurm file (after activating the conda env):

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/cm/local/apps/cuda/libs/current/lib64 I also had to switch my environment to python 3.6 (it worked earlier this summer without the explicit path specs and with python 3.7 - not sure what changed)

Janet Barclay U.S. Geological Survey New England Water Science Center Connecticut Office 101 Pitkin St. East Hartford, CT 06108

Phone (office) 860 291-6763 Fax 860 291-6799 Email @.**@*.**@*.***> https://www.usgs.gov/staff-profiles/janet-barclay

From: Simon Topp @.> Sent: Monday, July 26, 2021 10:57 AM To: USGS-R/river-dl @.> Cc: Subscribed @.***> Subject: [EXTERNAL] Re: [USGS-R/river-dl] containerizing environment? (#116)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Update on this. I put together a singularity container which can run locally but still is having trouble finding the GPU drivers on Tallgrass. I've reached out to the HPC help desk and will hopefully have some answers soon.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSGS-R%2Friver-dl%2Fissues%2F116%23issuecomment-886776750&data=04%7C01%7Cjbarclay%40usgs.gov%7Ce5134afac351434f2b9708d9504626da%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637629084499467953%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rwCje%2BN7O2L%2BVeXOMkiya%2FuR7j0IxRjzMNZ9q69IknU%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA5H7UAARKLPPTWVQVQO3GTTZVZXPANCNFSM462J2ANA&data=04%7C01%7Cjbarclay%40usgs.gov%7Ce5134afac351434f2b9708d9504626da%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637629084499477910%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kV8UHQgBExORpxoI5hgdWmidblnQY%2BuXmrAH3zXwmMk%3D&reserved=0.

SimonTopp commented 3 years ago

Ok, I think this is mostly resolved in #126 , but am going to leave open until merged and tested.

SimonTopp commented 2 years ago

This is addressed in #126