While debugging the above issue I noticed that the raster had some sort of scaling applied to the latitude and longitude, but not to the height. I took that scale factor into account while triangulating the terrain and now Hillshade and Viewshed
produce almost identical results compared to their numpy counterparts.
On the first row you have:
the original terrain | numpy viewshed | cupy rtx viewshed
On the second row you have:
the original terrain | numpy hillshade | cupy rtx viewshed with shadows
Fix for wrong camera ray generation in viewshed.
While debugging the above issue I noticed that the raster had some sort of scaling applied to the latitude and longitude, but not to the height. I took that scale factor into account while triangulating the terrain and now Hillshade and Viewshed produce almost identical results compared to their numpy counterparts.