search

nbren12 / geostreams

Streaming data from fortran climate/weather models.
7 stars 3 forks source link

real world use case example: ICAR #20

Open jhamman opened 7 years ago

jhamman commented 7 years ago

Following up on our conversation yesterday, here's a few things:

  1. ICAR Github Page: https://github.com/NCAR/icar
  2. ICAR Publication: http://journals.ametsoc.org/doi/abs/10.1175/JHM-D-15-0155.1
  3. Sample dataset: https://www.dropbox.com/s/71pb93feesvz772/columbia_example.tar.gz?dl=0
    • This covers the Pacific Northwest at 6km grid spacing. The forcings cover a single year of which I can get more of.
nbren12 commented 7 years ago

This is awesome Joe. I'm excited to get this up and running, partly just because ICAR sounds like a really cool model.

On Fri, Oct 20, 2017 at 12:25 PM, Joe Hamman notifications@github.com wrote:

Following up on our conversation yesterday, here's a few things:

  1. ICAR Github Page: https://github.com/NCAR/icar
  2. ICAR Publication: http://journals.ametsoc.org/ doi/abs/10.1175/JHM-D-15-0155.1 http://journals.ametsoc.org/doi/abs/10.1175/JHM-D-15-0155.1
  3. Sample dataset: https://www.dropbox.com/s/71pb93feesvz772/columbia_ example.tar.gz?dl=0
    • This covers the Pacific Northwest at 6km grid spacing. The forcings cover a single year of which I can get more of.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nbren12/geostreams/issues/20, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUokiPaiXlcHjL1ziQ-rHQQRROyX4ytks5suPOigaJpZM4QBHld .

-- Noah Brenowitz nbren12@uw.edu http://atmos.washington.edu/~nbren12/

nbren12 commented 7 years ago

I think I will just fork ICAR, and create a Dockerfile. This Dockerfile can install geostreams as an external dependency.

nbren12 commented 7 years ago

Here is my fork: https://github.com/nbren12/icar/tree/develop

nbren12 commented 7 years ago

@jhamman I setup a docker image on nbren12/icar, and made an example test case in geostreams that downloads the test case and runs the ICAR docker image. Unfortunately, my laptop does not appear to have enough memory to run the test case. How many GB do you think we will need?

nbren12 commented 7 years ago

Okay I started it running on my work desktop which has a lot of RAM. It looks like ICAR needs around 15GB for this test case. So far I am stuck on this step

Initializing linear winds                                                     
   Generating a spatially variable linear perturbation look up table           
 Percent Completed:

I will just leave it running and see what happens ;)

jhamman commented 7 years ago

Cool. Keep me posted. My understanding is that this takes a while the first time it runs.

nbren12 commented 7 years ago

Ok. I think it finished generating this look up table, but I ran it all night and it only reached 1 hr of simulation time.

jhamman commented 7 years ago

Do you have OMP_NUM_THREADS set? I'm wondering if you are running single threaded.

nbren12 commented 7 years ago

I am running in single threaded mode, but shouldn't that only make my code run twice as slow? I reached 1 hr of model time after just a few minutes, and then the code didn't output anything else all night.

nbren12 commented 7 years ago

Just because I know the model, I was able to get SAM working in Docker without much problems. It uses around 400 MB of memory, so it might be more convenient to use as a testing model, until we can arrange shared access to a machine that is powerful enough to run ICAR. Since the model is not openly available, I have it setup in a private bitbucket repo and docker hub build.

On a 4 CPU google cloud instance, I could generate about 10 minutes of model time in about 1 minute real time.

jhamman commented 7 years ago

@nbren12 - I think you should get roughly linear scaling on whatever machine you're using. So 10procs = 10x speedup.