Dockerize Raven (2) - Githubissues

huard commented 6 years ago

Create Dockerfile describing how to build a docker image for Raven and Ostrich as this will be critical for deployment of this bird.

RAVEN code (Version 2.8.1) is found here: http://raven.uwaterloo.ca/Downloads.html OSTRICH code can be found here: http://www.eng.buffalo.edu/~lsmatott/Ostrich/OstrichMain.html

There is a subversion repo as well.

huard commented 6 years ago

@tomLandry is this on your end ?

tomLandry commented 6 years ago

Well you said it was "critical" for deployment. I guess CRIM should take this, yes! Is it urgent, now.

huard commented 6 years ago

Not urgent, and I "think" it's critical, which doesn't mean it is. I just assumed that if we wanted to simplify deployment, this was a must have, but I could be wrong.

richardarsenault commented 5 years ago

RAVEN can ingest and output NetCDF files.

The forcings are either (A) GRIDDED NetCDF variables have 3 dimensions: lat, lon, time (in any ordering) see attachment: York_daily_3d.nc (precipitation, temperature inputs for York catchment on a 4x6 cell grid)

or

(B) TIME SERIES i.e. a time series at various locations NetCDF variables have 2 dimensions: nLocations, time (in any ordering) see attachment: York_daily_2d.nc (precipitation, temperature inputs for York catchment at 22 stations)

RAVEN then always needs a text file containing weightings on how much every grid cell contributes to the HRUs/sub-basins defined in RAVEN. I’ve attached the two example files for this as well (see GriddedForcings_2D.txt and GriddedForcings_3D.txt).

The outputs can also be in NetCDF format (or ASCII). Attached are one file of the simulated discharge being dumped to a NetCDF file (Hydrographs.nc) and the watershed storages over time (WatershedStorage.nc).

See here for files: https://www.dropbox.com/sh/wfyr81bbz5xpdlr/AAA6jrTuHm05ReGyEZhKNudYa?dl=0

richardarsenault commented 5 years ago

RAVEN code (Version 2.8.1) is found here: http://raven.uwaterloo.ca/Downloads.html

OSTRICH code can be found here: http://www.eng.buffalo.edu/~lsmatott/Ostrich/OstrichMain.html

tomLandry commented 5 years ago

Bien reçu, j'active la demande.

huard commented 5 years ago

Is it possible while doing this to create conda packages for Raven and Ostrich and upload them to the birdhouse channel ? I'm at the point where I need to install Raven to create the WPS service.

tomLandry commented 5 years ago

Understood. Will see what we can do. Hopefully, we'll have the answers for the technical kick off.

julemai commented 5 years ago

@huard and @tomLandry Let me know if I can be of any help with compiling Raven or get it to run.

tomLandry commented 5 years ago

@julemai : we would need access to the SVN repos

julemai commented 5 years ago

@tomLandry Sure. You can sign up for SVN access through the CHyMS portal. You might want to contact James Craig after you've got access such that he has an overview over who's on there.

lalondma commented 5 years ago

@julemai I get a 'sign in' pop-up window when I use your link (no sign-up service). Should I contact James Craig and ask him to create an account for me?

julemai commented 5 years ago

@lalondma James can‘t create an account for you either. The SVN is hosted by the NRC. You would need to contact Alan Barton there (alan.barton@nrc-cnrc.gc.ca).

Sorry for the inconvenience. I didn’t remember how that link looks when you’re not yet registered.

julemai commented 5 years ago

@lalondma Please see here (section "Get access to Raven SVN") for all information. I also provide there the documents you have to fill out to get an account.

tomLandry commented 5 years ago

@davidcaron : addding you as an assignee with Marc (in charge of the task), so you coordinate with him on these two critical components.

lalondma commented 5 years ago

FYI, Both Raven and Ostrich are now dockerized ("singularized" in fact, since I'm using Singularity instead of Docker), and the images are available on a registry that I set up at CRIM (132.217.141.54).

Raven: I got a working python script that 1) copies local data to my ComputeCanada home directory, 2) submits a script to slurm that pulls the raven image from the registry and executes it, 3) polls the slurm batch queue and 4) copies the output files back to my local directory when the job is done.
Ostrich: the image contains the OpenMPI version of the executable and I get results when I manually push the image to slurm as a batch job (with Demo2 data). Next step is to adapt the aforementioned python script so it can handle Ostrich in addition to Raven. File transfer and remote execution are based on ssh, so the script expects that the ConputeCanada account is properly set up (i.e. user's public key present in ~/.ssh/authorized_keys file)

huard commented 5 years ago

Ok. David B. suggested I use the same container set-up to deploy Raven. Is this still a good idea with singularity?

lalondma commented 5 years ago

I'm not completely sure I understand your question, tell me if I missed your point but my goal is to provide an abstraction layer between your code and what I have right now, so that running the Raven app locally (on your server) or on CC infrasructure (using a unique singularity image) would be possible with the same interface. David B. told be your server will be dockerized, so the question is: Can we run a singularity image inside docker? That, I don't know, I'll have to do some tests in the next few days.

huard commented 5 years ago

Yes, something like that. My question is "Can I use what you're doing for a standard deployment (running on a local machine or on our server") ?

At the moment, to install the server, I do make install which download the source and compiles it. David B. thought this was fragile and suggested I use a docker image instead. What would your suggestion be for deployment ? At the moment, given that the latest version of source code is not yet released, my tendency would be to add a src/ directory to github with the latest version of RAVEN. The Dockerfile can then just execute the makefile.

    @echo "Downloading RAVEN hydrological framework ..."
    @test -f $(CURDIR)/src/RAVEN.zip || curl $(RAVEN_URL) --output "$(CURDIR)/src/RAVEN.zip"
    @echo "Unzipping RAVEN ..."
    @test -d $(RAVEN_SRC) || unzip $(CURDIR)/src/RAVEN.zip -d "$(RAVEN_SRC)"
    @echo "Compiling RAVEN ..."
    @test -f $(RAVEN_SRC)/raven_rev.exe || $(MAKE) -C $(RAVEN_SRC) -j4
    @test -d bin || mkdir bin
    @-bash -c "cp $(RAVEN_SRC)/raven_rev.exe ./bin/raven"

lalondma commented 5 years ago

If you look at the README in the branch I made (https://github.com/Ouranosinc/raven/tree/docker_singularity/raven/singularity), you'll see how to execute the raven container I'm maintaining. It's the exact same container I'm using for my tests with my CC account. In theory, this piece of solution would be applicable not only to interactions with HPCs but also to your own server, considering that we have indications that executing a singularity container within a docker container seems to be possible. About the short test in the README: it should take 10-15min, so give it a try and let me know how it turns out.

lalondma commented 5 years ago

@julemai I'm looking at the zipped package containing Ostrich and I see a bunch of executables for Demos 1 to 9: Bluebird, bigfoot, split and so on. I'm thinking that it's actually raven that will be used by Ostrich for the current project, have I got that right?

julemai commented 5 years ago

@lalondma It seems you have downloaded the Ostrich Tutorial examples. Please don't use that. I made the OSTRICH source code available in the Wiki (.zip).

I also made it available as a download. This is how we handle RAVEN in the Makefile right now. You can use the following URL: http://www.civil.uwaterloo.ca/jmai/raven/Ostrich_2017-12-19_plus_progressJSON.zip

You will need to compile it using: make GNU

Please use either of the above mentioned source codes (they are the same). They contain the functionality of dumping the progress of the calibration to a file containing a JSON string. That's PAVICS specific and not available in the released OSTRICH. It will be included in the next release.

Is that the information you are looking for?

lalondma commented 5 years ago

Actually I didn't know about the wiki, and that's where I found what I needed, thanks!

huard commented 5 years ago

@lalondma Finally got around to look at what you've done. Sorry again about the long delay.

I've installed singularity and I'm trying to run the image but I get connection refused error. Do I need some kind of ssh keys to access shub ? Or do you need to add me as a user with read access ? I guess the image should be visible to all, no ?

huard commented 5 years ago

@lalondma My mistake, I switched environment and didn't realize I hadn't re-set the env variable.

huard commented 5 years ago

It works ! Are you OK if I merge master in your docker-singularity branch ? I've tried locally and there was no conflict.

I suggest the container building commands go into the Makefile. Does it make sense to you ? Should the makefile also have a pull_img command to store the image in the bin/ directory ?

lalondma commented 5 years ago

Go ahead with the merge: my stuff's inside specific folders, so there shouldn't be any interference with the rest of the project. Hmm, what is it that's working ? :-) Running a singularity image ? As for container building, I guess it would make sense, although pushing a new image to the singularity registry would require credentials (could be a specific target, not the default target).

huard commented 5 years ago

I'm now able to set-up model configuration files and run either on the local bash version or the local singularity image.

    def test_singularity(self):
        rvs = TESTDATA['raven-gr4j-cemaneige-nc-rv']
        ts = TESTDATA['raven-gr4j-cemaneige-nc-ts']

        model = Raven()
        model.singularity = True
        model.configure(rvs)
        model.run(ts)

Ok, so we'll keep it simple for now and simply let the makefile grab the image and download it locally.

Ouranosinc / raven

Dockerize Raven (2) #33