search

UW-Hydro / VIC

The Variable Infiltration Capacity (VIC) Macroscale Hydrologic Model
http://vic.readthedocs.io
MIT License
260 stars 383 forks source link

VIC Image Driver CPU Limited? Ways to increase memory use? #942

Closed dwoodson-usbr closed 6 months ago

dwoodson-usbr commented 8 months ago

Bug Reports

-Version of VIC: VIC.5.1.0 -Name and version of the C compiler you are using: gcc (GCC) 8.5.0 and Open MPI v4.1.1 -Operating system: Linux - RedHat -A description of relevant model settings: I am running the VIC Image driver to simulate 1/16th degree LOCA CMIP5 projections for the Colorado River Basin using updated soil and vegetation parameters from Currier et al. 2023. I am running with 44 processors using mpiexec. Running the Livneh historical dataset covering 1950-2013 took ~12 hours and each CMIP5 trace is estimated to take 27 hours.

Here is the shell script I am using to run the VIC image driver:

!/bin/bash

export HDF5_USE_FILE_LOCKING="FALSE"

/usr/lib64/openmpi/bin/mpiexec -np 44 /mnt/raid/models/vic/VIC-5.1.0/vic/drivers/image/vic_image.exe -g ../params/global.VIC5.LOCA.rechunked.txt >& vicRunLogChunked_LOCA

My CPU load is at 100% but memory use is only ~20 GB while running VIC. Is VIC CPU limited by nature or is there something I'm missing with respect to VIC settings or makefile/dependency settings that are increasing my runtime significantly? The only change I made in the makefile is to add the full path to point to mpicc. Is there a way to increase my memory use so that runtime is reduced?

Thank you.

dwoodson-usbr commented 6 months ago

VIC was running slow most likely because OMP_NUM_THREADS was not set to 1 when running the VIC image driver with OpenMPI. OpenMP and OpenMPI seem to conflict and cause not only slower run times but also NA values in the output.