search

Hongqing-work / cudnn-learning-framework

A tiny learning framework built by cudnn and cublas.
21 stars 3 forks source link

What is the purpose of this code? #1

Open scuizhibin opened 2 years ago

scuizhibin commented 2 years ago

int requestedAlgoCount = CUDNN_CONVOLUTION_FWD_ALGO_COUNT; int returnedAlgoCount = -1; cudnnConvolutionFwdAlgoPerf_t fwd_results[2 requestedAlgoCount]; checkCUDNN(cudnnFindConvolutionForwardAlgorithm(handle, src_tensor_desc, filter_desc, conv_desc, dst_tensor_desc, requestedAlgoCount, &returnedAlgoCount, fwd_results)); fwd_algo = fwd_results[0].algo;

Hongqing-work commented 2 years ago

You can refer to cudnnFindConvolutionForwardAlgorithm() in cuDNN-API document. cuDNN implement different algorithm for convolution. Different algorithms require different need for memory and have different performance. This function attempts all algorithms available for cudnnConvolutionForward(). It will attempt both the provided convDesc mathType and CUDNN_DEFAULT_MATH. Therefore, you can see the "fwd_results" is a 2 * requestedAlgoCount array. This output "fwd_results" is a user-allocated array to store performance metrics sorted ascending by compute time. So I choose the first algorithm. If you want lower memory size, you can make other choices. “fwd_algo” will later be used in cudnnGetConvolutionForwardWorkspaceSize() to allocate temporary memory size according to the algorithm you choose and the actual execution of convolution cudnnConvolutionForward() also need it.

scuizhibin commented 2 years ago

Why are there two algorithm choices in backward propagation, cudnnConvolutionBackwardFilter and cudnnConvolutionBackwardData. What is their role? Can you answer it?

Hongqing-work commented 2 years ago

In backward propagation, cudnnConvolutionBackwardFilter is used to get the differential of filter, and cudnnConvolutionBackwardData is used to get the differential of input. These two operation is different, so they need different algorithm.

Hongqing-work commented 2 years ago

The total number of resulting algorithms can be queried through the API cudnnGetConvolutionForwardAlgorithmMaxCount(). Also, you can find it in cudnn.h

scuizhibin commented 2 years ago

What is the process of cudnn backpropagation? There is too little information on the Internet.

Hongqing-work commented 2 years ago

cuDNN doesn’t provide api for getting differential from loss, so you should write your own code. After that, you can refer to backpropagation algorithm and use cuDNN api to compute the differential.

scuizhibin commented 2 years ago

What does this ”cudnnConvolutionBackwardData is used to get the differential of input“ mean? like Is the derivative of p in y(p(x))?

Hongqing-work commented 2 years ago

What does this ”cudnnConvolutionBackwardData is used to get the differential of input“ mean? like Is the derivative of p in y(p(x))?

Yes. After that, you can use the derivative of p to get the derivative of parameters in p(x), so the output of cudnnConvolutionBackwardData would be passed to the backforward function of its previous operation. You can refer to main.cu.