micah-prime
commented
5 years ago This is the result of the line profiler over one timestep in the Tuolumne, which should be a good representation of where the code is spending it's time. This was using the cubic method for interpolation, which, unlike the linear method, does not show a speedup when changing from griddata to grid_interpolate_deconstructed.
It's evident here that the biggest offenders are:
122 852 1121563.0 1316.4 13.4 elev = pv.loc[drow].elevation
123 852 1395132.0 1637.5 16.7 pp = np.polyfit(elev, pv.loc[drow].data, 1)
130 852 1807012.0 2120.9 21.6 pv.loc[grid_name, ['slope', 'intercept']] = pp
138 1 635758.0 635758.0 7.6 grid_slope_og = griddata((pv.utm_x, pv.utm_y), pv.slope, (self.GridX, self.GridY), method=grid_method)
139 1 615866.0 615866.0 7.4 grid_slope = grid_interpolate_deconstructed(self.tri, pv.slope.values[:], (self.GridX, self.GridY), method=grid_method)
142 1 654457.0 654457.0 7.8 grid_intercept_og = griddata((pv.utm_x, pv.utm_y), pv.intercept, (self.GridX, self.GridY), method=grid_method)
143 1 626213.0 626213.0 7.5 grid_intercept = grid_interpolate_deconstructed(self.tri, pv.intercept.values[:], (self.GridX, self.GridY), method=grid_method)
151 1 655289.0 655289.0 7.8 idtrend_og = griddata((pv.utm_x, pv.utm_y), dtrend, (self.GridX, self.GridY), method=grid_method)
152 1 620712.0 620712.0 7.4 idtrend = grid_interpolate_deconstructed(self.tri, dtrend, (self.GridX, self.GridY), method=grid_method)The griddata functions account for ~22% of the time. The pandas data grabs and the polyfit are quite slow.
Here is the full print out:
Slope difference: 0.0
Intercept difference: 0.0
idtrend difference: 0.0
Timer unit: 1e-06 s
Total time: 8.34768 s
File: /home/micahsandusky/Documents/Code/virtualenvirons/env3/lib/python3.5/site-packages/smrf-0.8.8-py3.5-linux-x86_64.egg/smrf/spatial/grid.py
Function: detrendedInterpolationLocal at line 105
Line # Hits Time Per Hit % Time Line Contents
==============================================================
105 def detrendedInterpolationLocal(self, data, flag=0, grid_method='linear'):
106 """
107 Interpolate using a detrended approach
108
109 Args:
110 data: data to interpolate
111 grid_method: scipy.interpolate.griddata interpolation method
112 """
113
114 # copy the metadata to append the slope/intercept and data to
115 1 254.0 254.0 0.0 pv = self.metadata.copy()
116 1 972.0 972.0 0.0 pv['slope'] = np.nan
117 1 903.0 903.0 0.0 pv['intercept'] = np.nan
118 1 919.0 919.0 0.0 pv['data'] = data
119
120 853 191236.0 224.2 2.3 for grid_name,drow in self.dist_df.iterrows():
121
122 852 1121563.0 1316.4 13.4 elev = pv.loc[drow].elevation
123 852 1395132.0 1637.5 16.7 pp = np.polyfit(elev, pv.loc[drow].data, 1)
124
125 # apply trend constraints
126 852 2690.0 3.2 0.0 if flag == 1 and pp[0] < 0:
127 186 968.0 5.2 0.0 pp = np.array([0, 0])
128 666 602.0 0.9 0.0 elif (flag == -1 and pp[0] > 0):
129 pp = np.array([0, 0])
130 852 1807012.0 2120.9 21.6 pv.loc[grid_name, ['slope', 'intercept']] = pp
131
132 # get triangulation
133 1 1.0 1.0 0.0 if self.tri is None:
134 xy = _ndim_coords_from_arrays((pv.utm_x, pv.utm_y))
135 self.tri = qhull.Delaunay(xy)
136
137 # interpolate the slope/intercept
138 1 635758.0 635758.0 7.6 grid_slope_og = griddata((pv.utm_x, pv.utm_y), pv.slope, (self.GridX, self.GridY), method=grid_method)
139 1 615866.0 615866.0 7.4 grid_slope = grid_interpolate_deconstructed(self.tri, pv.slope.values[:], (self.GridX, self.GridY), method=grid_method)
140 1 2981.0 2981.0 0.0 print('Slope difference: {}'.format(np.sum(grid_slope_og-grid_slope)))
141
142 1 654457.0 654457.0 7.8 grid_intercept_og = griddata((pv.utm_x, pv.utm_y), pv.intercept, (self.GridX, self.GridY), method=grid_method)
143 1 626213.0 626213.0 7.5 grid_intercept = grid_interpolate_deconstructed(self.tri, pv.intercept.values[:], (self.GridX, self.GridY), method=grid_method)
144 1 3084.0 3084.0 0.0 print('Intercept difference: {}'.format(np.sum(grid_intercept_og-grid_intercept)))
145
146 # remove the elevation trend from the HRRR precip
147 1 788.0 788.0 0.0 el_trend = pv.elevation * pv.slope + pv.intercept
148 1 361.0 361.0 0.0 dtrend = pv.data - el_trend
149
150 # interpolate the residuals over the DEM
151 1 655289.0 655289.0 7.8 idtrend_og = griddata((pv.utm_x, pv.utm_y), dtrend, (self.GridX, self.GridY), method=grid_method)
152 1 620712.0 620712.0 7.4 idtrend = grid_interpolate_deconstructed(self.tri, dtrend, (self.GridX, self.GridY), method=grid_method)
153 1 3563.0 3563.0 0.0 print('idtrend difference: {}'.format(np.sum(idtrend_og-idtrend)))
154
155 # reinterpolate
156 1 6350.0 6350.0 0.1 rtrend = idtrend + grid_slope * self.GridZ + grid_intercept
157
158 1 3.0 3.0 0.0 return rtrend
scotthavens
In the San Joaquin model run for a single day, we are seeing about 25% slow down due to the calculations in the local gradient method. Profiling the
grid.pyfor the unit testtest_load_data.test_grid_hrrr_localshows that about 90% of the time spent is in the calculation of the slopes and intercepts. The newinterp_speedchanges helps to speed up the interpolation but the test case is too small to truly see the speed increases.