search

IntelLabs / Model-Compression-Research-Package

A library for researching neural networks compression and acceleration methods.
Apache License 2.0
134 stars 23 forks source link

Upstream pruning #1

Closed eldarkurtic closed 2 years ago

eldarkurtic commented 2 years ago

Hi! First of all, thanks for open-sourcing your code for the "Prune Once for All" paper. I would like to ask a few questions:

  1. Are you planning to release your teacher model for upstream task? I have noticed that at https://huggingface.co/Intel , only the sparse checkpoints have been released. I would like to run some experiments with your compression package.
  2. From the published scripts, I have noticed that you have been using only English Wikipedia dataset for pruning at upstream tasks (MLM and NSP) but the bert-base-uncased model you use as a starting point is pre-trained on BookCorpus and English Wikipedia. Is there any specific reason why you haven't included BookCorpus dataset too?
ofirzaf commented 2 years ago

Hi,

Glad you found the paper interesting.

  1. We don't plan to release the teachers we used. You can produce your own teacher using the training scripts in the examples with the default hyper-parameters published in original BERT paper or HuggingFace/transformers repo. We chose the best model out of 5 experiments with different seeds to be our teacher. Also, if you already have a trained model for the task you want to test or find one in the models hub, you can use it as well.
  2. No special reason other than time. We started experimenting with only Wikipedia and we simply didn't have enough time to redo all the experiments with the addition of BookCorpus.