Open David-Li0406 opened 3 weeks ago
Thanks for your interest on our work!
Our modeling_xxx.py scripts are modified based on the original model hubs on HuggingFace. Specifically, we modify the forward() function of the CausalLM class to add an extra argument decoder_layer_idx
for implementing the second pass with top_k=ALL, as described in Section 5 in our paper.
Hence, if you aim to obtain the original top_k gate score/indices (i.e., the second pass is unnecessary), I believe importing from transformers should function the same using our code, as long as (1) your model checkpoints are compatible with it; (2) differences between variable names (if any) are probably handled.
Also, could you provide the error message (and which model script triggers the error) so that we can deal with the potential bugs of our code? Thanks ;)
Thanks for this prompt response! The error was appeared when I ran the "Norms of Expert Outputs and Gate Scores" part in dynamic_analysis.ipynb:
Hi,
Thanks for sharing the code and I am currently working based on it. I only need to get the gate score/ indices from each model (part of "Norms of expert outputs and gate scores" in dynamic_analysis.ipynb).
In this case, is there any difference between using the model implementations from your repository (modeling_xxx.py) or directly importing from transformers? I am asking because it will produce a CUDA trigger error for me to use your model implementation.
Thanks for your help!