search

Ouranosinc / raven

WPS services related to hydrological modeling
https://pavics-raven.readthedocs.io
MIT License
37 stars 12 forks source link

WPS to perform generic forecasts from forecast weather data and initial model states. #261

Closed richardarsenault closed 3 years ago

richardarsenault commented 4 years ago

This WPS should take as inputs:

It will output the simulated forecasts as a N-days x M-members streamflow data netcdf.

It should be generic in that it could then be used for the Climatology-ESP, Caspar and Geomet forecasts, as well as making assimilation and pre-processing more convenient.

richardarsenault commented 4 years ago

@julemai do you have a good idea on how to use a previous run's storage files to hot-start another run?

julemai commented 4 years ago

Raven is writing an output file solutions.rvc. This contains all the states of the last time step. Replace the current, for example, raven-hmets.rvc with the solutions.rvc and you will be able to start with the states you finished with.

richardarsenault commented 4 years ago

@huard Could you look into this, how we can manage to do this without breaking the current setup of "when run Raven --> setup rv files" by substituting another one from a previous run?

huard commented 4 years ago
  1. Quick and dirty option: write a method that replaces the rvc file or template by the solution file.
  2. Elaborate option: template initial values in emulators, parse the solution and feed the parameters to the template.

Option 2. will let also users specify initial conditions in general, which is currently not possible.

richardarsenault commented 4 years ago

Option 2 will enable data assimilation, a huge part of future forecasting endeavors. So i would propose that if it's not TOO bad. Thanks 

julemai commented 4 years ago

I am usually not using the RVC file. I don't know how often this is changing with revisions of Raven and between different model setups. This is something we will need to keep in mind when using templates.

But definitely sounds fancy!

huard commented 4 years ago

I realized that it was probably good to have 1 anyway, so started with this, and wondering how numerically close should we expect results to be when restarting a run from a final state ? I get differences of about .1%

richardarsenault commented 4 years ago

Hmmm it should be exactly the same, it's basically the same run that continues. Maybe an issue with the date of the initialization (i.e. sometimes its taken at midnight (early same day) or midnight (late that day after the sim)... probably have to compare to a longer run to see exactly when the cutoff occurs

julemai commented 4 years ago

It should be exactly the same. But since I have no experience with the RVC file, I'll need to ask James.

richardarsenault commented 4 years ago

Is the rvc file the only one that has information across time steps? Does the solution file or the rv output have other info that is required?

julemai commented 4 years ago

This is all the documentation says:

solution.rvc - the solution file
Stores the complete state of the system at the end of the simulation. This file can be used as initial conditions for a later model run. This file may also be generated at user-specified intervals du- ring simulation as a defense against computer breakdown for massive computationally-demanding models.

I already texted James but he's not online.

julemai commented 4 years ago

Talked to James. He says that it should be the same but it might be that he is not printing enough digits of some variables to the RVC. He needs to see it. @huard Could you send James the setup you are using?! Including all the data such that he can have a look. But James already said that he can't promise a quick turn-around this week.

huard commented 4 years ago

Ok. Will investigate a bit more on my end before that.

huard commented 4 years ago

Ok, my mistake, I thought I needed to start the simulation one day after the end of the last one, but it should be the same day (start of second = end of first). In this case errors are < 1E-5, which is consistent with the number of digits in the rvc file.

julemai commented 4 years ago

That’s great news!! You think that this precision is enough for our purpose?

richardarsenault commented 4 years ago

Yes I think so, i mean forecasts are full of uncertainty anyways. If we ever update raven for more digits or its not too much to ask of James then great, we should do it, but I would live well with that degree of precision.

huard commented 4 years ago

At the moment, these are the number of state variables within current rvc files:

But, in reality there are dozens of state variables (convolutions, snow temp, snow cover, etc) that are not exposed. I suppose we should template all state variables that end up in the WatershedStorage output ?

richardarsenault commented 4 years ago

Yes ideally for it to work well we'd need to have the exact same states carried over. Is that not all in the storage rv file? Like Mohyse I know has at least three state variables. If we go with the file with 0 states... How would that even work? Sent from my Samsung Galaxy smartphone. -------- Original message --------From: David Huard notifications@github.com Date: 2020-04-29 21:38 (GMT-05:00) To: Ouranosinc/raven raven@noreply.github.com Cc: richardarsenault richard.arsenault@etsmtl.ca, Author author@noreply.github.com Subject: Re: [Ouranosinc/raven] WPS to perform generic forecasts from forecast weather data and initial model states. (#261) I would think so, yes. At the moment, these are the number of state variables within current rvc files:

HMETS: 2 GR4J: 2 HBC-EC: 1 MOHYSE: 0

But, in reality there are dozens of state variables (convolutions, snow temp, snow cover, etc) that are not exposed. I suppose we should template all state variables that end up in the WatershedStorage output ?

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/Ouranosinc/raven/issues/261#issuecomment-621560768", "url": "https://github.com/Ouranosinc/raven/issues/261#issuecomment-621560768", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

julemai commented 4 years ago

Yes, all these state variables are given in the Watershed storage or ForcingFunctions files.

richardarsenault commented 4 years ago

these the "ForcingFunction" files, where can they be found? and can we update them during hot-starting?

julemai commented 4 years ago

You need to switch on “:WriteForcingFunctions” in the RVI file.

All Raven needs to restart is the RVC file (and of course forcings). So, I think, only adjustments in these files will perturb the model. If you want to make adjustments to forcing, you’d need to do that in the forcings file (or through LinearTransform in the RVT file).

I am however not entirely sure if EVERY variable that is dumped to the forcing file can be used as an input variable in the RVT file. We might need to implement some new variables if necessary.