search

pacific-hake / hake-assessment

:zap: :fish: Build the assessment document using latex and knitr
MIT License
13 stars 6 forks source link

Research/Data need for performance or cloud computing? #779

Closed aaronmberger-nwfsc closed 3 years ago

aaronmberger-nwfsc commented 3 years ago

I was just thinking about storage and why my laptop always seems to hit or approach its limit during hake assessment time (I currently have a 500GB HD on this machine). Then I looked at my local github hake assessment set of folders which collectively are at 78 GB and will increase once adding all the age-1 survey retros. This also doesn't include all the test runs I've done which is another 5GB. My point here, I wonder if we should add under research and data needs a desire to have improved cloud and/or high performance computing capabilities. I think it qualifies even though not explicitly a "research" or "data" need; it relates to a practical assessment delivery need. Thoughts?

Probably at the bottom of the list in section 4. If three thumbs up, I'll add some text.

andrew-edwards commented 3 years ago

We have access to cloud computing here, I just haven't tried it. The SS stuff should be easy as it's just an .exe.

Maybe that's a good place to make the point about ADNUTS being more efficient for ONE run, but not when do every run using it.

aaronmberger-nwfsc commented 3 years ago

Yes, there should be options. I have access now to one at Oregon State and the NWFSC has been doing trials with Amazon. I have not done either one myself yet. I have previous experience using one at Michigan State though.

cgrandin commented 3 years ago

@andrew-edwards I tried to use the Government one, I got an account and spent a long time setting it up but it was typical of other Government computing resources, about 10 years behind. So, it is all command-line linux with no help from anyone on how to get things running, etc. I gave up at the time (3-4 years ago?). I would suggest something current and supported by outside resources, such as Amazon or Google. They just have so many users that the help you can find online is a benefit that's hard to beat.

aaronmberger-nwfsc commented 3 years ago

Re private sector: the key will be to have some sort of priority through an MOU with governments or something so that you don't submit a job and see that it didn't start the execution for like a week.

Ok, I'm just going to add the following blurb to the RandD section now. I'm hoping that "mechanisms" also means funding without explicitly saying funding. Feel free to adapt as you wish.

Develop mechanisms that improve computing capabilities and storage capacity through the use of cloud computing, local high performance computing clusters, or other similar productivity enhancements to improve assessment modeling and document building work flow.

kellijohnson-NOAA commented 3 years ago

Christine Stawitz has been successful in running SS in the cloud and we currently use Jenkins to test SS, so every time a commit is made to the repo hundreds of models are run and their output is checked against prior output.

On Fri, Feb 5, 2021 at 10:23 AM Aaron Berger notifications@github.com wrote:

Re private sector: the key will be to have some sort of priority through an MOU with governments or something so that you don't submit a job and see that it didn't start the execution for like a week.

Ok, I'm just going to add the following blurb to the RandD section now. I'm hoping that "mechanisms" also means funding without explicitly saying funding. Feel free to adapt as you wish.

Develop mechanisms that improve computing capabilities and storage capacity through the use of cloud computing, local high performance computing clusters, or other similar productivity enhancements to improve assessment modeling and document building work flow.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pacific-hake/hake-assessment/issues/779#issuecomment-774205112, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7LCFBWVQT5WTC57233LGTS5QZSTANCNFSM4XFFCN7Q .

-- Kelli Faye Johnson, PhD Research Fish Biologist Northwest Fisheries Science Center National Marine Fisheries Service 2725 Montlake Boulevard East Seattle, Washington 98112 (206) 860-3490 kelli.johnson@noaa.gov

andrew-edwards commented 3 years ago

I'm guessing that Jenkins is, like, Travis's brother?

kellijohnson-NOAA commented 3 years ago

We might even just be able to use github actions, which we already use for ss3sim to test the package that does simulations with SS works.

andrew-edwards commented 3 years ago

Travis has stopped working for some of my own packages, so I think I need to switch to GitHub Actions. Haven't had time though (and the examples I looked seemed overly complicated)....

cgrandin commented 3 years ago

It's easy @andrew-edwards, see the hake MSE repo, all you have to do is literally add this file: https://github.com/pacific-hake/pacifichakemse/blob/master/.github/workflows/R-CMD-check.yaml

aaronmberger-nwfsc commented 3 years ago

Seems like everyone is good with this so going to close the issue... but you can still edit the text if you like.

andrew-edwards commented 3 years ago

Thanks @cgrandin - those long .yaml files are what scared me off! Matt was doing one for herring but had trouble.

cgrandin commented 3 years ago

@andrew-edwards - I think you can literally copy that in without changing anything, remove the Travis Yaml file and it should work.

Note it is in a folder called .github/workflows so you have to add that manually and make sure your .gitignore is not ignoring it. Mine was and it took me awhile to figure that out