search

nbren12 / uwnet

Neural Networks based unified physics parameterization for atmospheric models
MIT License
3 stars 2 forks source link

Stochastic state model class #66

Closed sarenehan closed 5 years ago

sarenehan commented 5 years ago

On the branch "stochastic-state-model-when-working", when I run the GCM with the stochastic state model class, it behaves as it did when I showed you and Chris. However, when I run it on the branch "stochastic-state-model-class", the "SUM OF NN QT tendency" rises and rises, and the weather gets more and more extreme (ie, precipitation). Using ncview, the second situation seems a lot worse.

The trouble is, there are no code differences in the stochastic_state_model, or parameters2.json file that is used to generate the CASE file. However, on "stochastic-state-model-class", there is new code that I pulled from master a few days ago.

@nbren12 are you able to explain the fairly significant differences in the two situations?

(please ignore the first pull request that I closed a few minutes ago, this is the direction I want to compare)

nbren12 commented 5 years ago

That sounds like the classic numerical blow-up we have been seeing. It could just be random, does it happen robustly? Can you upload the actual prm file in the directory you are using to a gist and link it here?

If not, my guess is that a135c55 is breaking it since that changes the sam_interface a fair amount. You could try reverting that commit.

From your code, it does seem that your linear regression models depend on the upper atmospheric variables, which makes me nervous. This code in residual_stochastic_state_model.py isn't what I meant:

            rows_to_keep = (
                (x_data['QT'][:, :34] < self.max_qt_for_residual_model) &
                (x_data['QT'][:, 34:68] < self.max_sli_for_residual_model)

This thresholds the variables exceeding a certain value. What you should do instead is removing the upper atmospheric data from the inputs altogether. In other words, your residual models should only depend on qt[:,:15,:,:] and sli[:,:18,:,:]. Does that make sense?

sarenehan commented 5 years ago

@nbren12 I have tried this out four or five times on both branches, so I am pretty confident it happens robustly.

Is there a way for me to conveniently turn off the white noise stochastic parameterization?

I'll also try removing the upper atmosphere data from the inputs all together, but my implementation wasn't blowing up initially so I don't think this is the root cause.

It will be informative to know exactly which of these changes is causing the gcm to blow up.

nbren12 commented 5 years ago

Are you sure your prm files are identical between the broken and working examples? There are loads of things in there that could cause blow-up.

nbren12 commented 5 years ago

I would also recommend rebasing rather than merging when you want to grab changes from the master branch.

nbren12 commented 5 years ago

Is there a way for me to conveniently turn off the white noise stochastic parameterization?

It shouldn't be on. That commit just introduces changes to the interfaces which might be affecting your code.

sarenehan commented 5 years ago

Are you sure your prm files are identical between the broken and working examples? There are loads of things in there that could cause blow-up.

Yes, they are identical except for the nsavepython = 30 line.

sarenehan commented 5 years ago

I would also recommend rebasing rather than merging when you want to grab changes from the master branch.

Good idea, that is a cleaner way to do it. I'll do that from now on.

nbren12 commented 5 years ago

@sarenehan Do we still need this PR? Seems like you figured out the problem.

nbren12 commented 5 years ago

Thanks!

On Tue, Apr 2, 2019 at 9:04 AM Stewart Renehan notifications@github.com wrote:

Closed #66 https://github.com/nbren12/uwnet/pull/66.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nbren12/uwnet/pull/66#event-2247061176, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUokh0oX7fbLiw_C_X-CAiO8hfNCZVUks5vc393gaJpZM4cPYGG .