search

HazyResearch / fly

Apache License 2.0
181 stars 20 forks source link

Minor questions about the paper and code #1

Open os-hxfan opened 2 years ago

os-hxfan commented 2 years ago

Thanks a lot for the interesting work! I am really enjoying reading the paper and the code. I actually have two minor questions. It will be really appreciated if any hints can be provided:

  1. I notice that, in Section 5.1, Pixelfly is only applied on the projection step of Attention and MLP, without sparsifying the attention matrix (score matrix). While in T2T-Vit, the Pixelfly is only applied on the attention matrix without sparsifying MLP and projection. Are there any reasons for this? Also, are there any experimental results if Pixelfly is applied on all layers (MLP and attention matrix)?
  2. I saw there are many options for /model/t2tattn_cfg with T2T-Vit, such as sblocal, performer. It seems like sblocal uses sparse + low rank. Maby I know which one should I choose if I want to use flat butterfly + low rank?
  3. In the experiment folder under config, it seems like only the scripts for MLP-mixer, T2T-vit are provided. Do you have plans to release all scripts for other experiments? such as Vit, GPT etc......
abhishektyaagi commented 5 months ago

@os-hxfan Is it possible for you to share with me if you were able to figure out an answer to:

I saw there are many options for /model/t2tattn_cfg with T2T-Vit, such as sblocal, performer. It seems like sblocal uses sparse + low rank.