Open p1nksnow opened 1 month ago
I am also confused about the hvp
function. What part of the paper does this correspond to? Thanks for any helpful response.
Sorry for the confusion. I removed the feature and have not added it back yet.
The return_words_loss
should be "return_tokens_loss". When it is true, it will work in this way (batch_size = 1):
tokens_grad_list = []
if return_tokens_loss == True:
for i, token_loss in enumerate(loss):
if label[i] == IGNORE_INDEX:
tokens_grad_list.append(None)
continue
grad = torch.cat([x.reshape(-1) for x in list(grad(token_loss, params, retain_graph=True))])
tokens_grad_list.append(grad)
It calculates the gradient for each token if the label exists. The params
should be all the trainable parameters.
The hvp
function is for influence function we compared. Our RapidIn does not use hvp
.
Edit:
I am not sure if the backward
can do the same thing:
loss[i].backward()
# or
model.backward(loss[i])
Thank you for you reply! I have one more question, the influence function
The hvp function is for influence function we compared.
is it TracIn or something else?
No, it is Influence Function: Koh, Pang Wei, and Percy Liang. "Understanding black-box predictions via influence functions." International conference on machine learning. PMLR, 2017.
Could you provide a config_caching.json
and the corresponding deepspeed config
file on how to use deepspeed multi-GPU parallelism?
Our multi-GPU parallelism is not implemented by using deepspeed. The deepspeed implementation in RapidIn has not tested yet.
If you have multiple gpus, the multi-GPU parallelism should be auto-enabled. You can see you will have different processes in your gpus. And you can set the n_threads
to increase the number of threads of each GPU to maximize the use of GPU memory.
The
grad_z
function does not seem to support calculating the influence of a single word. There isreturn_words_loss
in the function parameter, but this parameter is not used in the function body. Is this part of the code the final version?