Run baseline experiments for perplexities

python main.py \
    --model hf-causal \
    --model_args pretrained=facebook/opt-125m \
    --tasks wikitext,pile_pile-cc\
    --device cuda:0
                                                                                                                           | 1/52852 [00:03<50:57:14,  3.47s/it]Token indices sequence length is longer than the specified maximum sequence length for this model (2901 > 2048). Running this sequence through the model will result in indexing errors
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 52852/52852 [2:00:28<00:00,  7.31it/s]
{
  "results": {
    "pile_pile-cc": {
      "word_perplexity": 66.34653081275015,
      "byte_perplexity": 2.0128304977078764,
      "bits_per_byte": 1.0092256867091036
    },
    "wikitext": {
      "word_perplexity": 31.947303588722487,
      "byte_perplexity": 1.9113345806558002,
      "bits_per_byte": 0.9345803453494412
    }
  },
  "versions": {
    "pile_pile-cc": 1,
    "wikitext": 1
  },
  "config": {
    "model": "hf-causal",
    "model_args": "pretrained=facebook/opt-125m",
    "num_fewshot": 0,
    "batch_size": null,
    "device": "cuda:0",
    "no_cache": false,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal (pretrained=facebook/opt-125m), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
|    Task    |Version|    Metric     | Value |   |Stderr|
|------------|------:|---------------|------:|---|------|
|pile_pile-cc|      1|word_perplexity|66.3465|   |      |
|            |       |byte_perplexity| 2.0128|   |      |
|            |       |bits_per_byte  | 1.0092|   |      |
|wikitext    |      1|word_perplexity|31.9473|   |      |
|            |       |byte_perplexity| 1.9113|   |      |
|            |       |bits_per_byte  | 0.9346|   |      |

python main.py \
    --model hf-causal \
    --model_args pretrained=facebook/opt-350m \
    --tasks wikitext,pile_pile-cc\
    --device cuda:0
                                                                                                                                     | 0/52852 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (7524 > 2048). Running this sequence through the model will result in indexing errors
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 52852/52852 [2:29:32<00:00,  5.89it/s]
{ 
  "results": {
    "wikitext": {
      "word_perplexity": 25.420972253409094,
      "byte_perplexity": 1.8313778510712453,
      "bits_per_byte": 0.8729294796265972
    },
    "pile_pile-cc": {
      "word_perplexity": 51.77632131224143,
      "byte_perplexity": 1.9312978992579177,
      "bits_per_byte": 0.9495707145910748
    }
  },
  "versions": {
    "wikitext": 1,
    "pile_pile-cc": 1
  },
  "config": {
    "model": "hf-causal",
    "model_args": "pretrained=facebook/opt-350m",
    "num_fewshot": 0,
    "batch_size": null,
    "device": "cuda:0",
    "no_cache": false,
    "limit": null,
    "bootstrap_iters": 100000,
    "description_dict": {}
  }
}
hf-causal (pretrained=facebook/opt-350m), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
|    Task    |Version|    Metric     | Value |   |Stderr|
|------------|------:|---------------|------:|---|------|
|wikitext    |      1|word_perplexity|25.4210|   |      |
|            |       |byte_perplexity| 1.8314|   |      |
|            |       |bits_per_byte  | 0.8729|   |      |
|pile_pile-cc|      1|word_perplexity|51.7763|   |      |
|            |       |byte_perplexity| 1.9313|   |      |
|            |       |bits_per_byte  | 0.9496|   |      |

[x] install hugging face transformer locally

git clone git@github.com:emma-mens/transformers.git
cd transformers/
# https://discuss.huggingface.co/t/how-to-edit-different-classes-in-transformers-and-have-the-transformer-installed-with-the-changes/12781
# pip install .
# https://huggingface.co/transformers/v4.7.0/installation.html
pip install -e .

https://discuss.huggingface.co/t/how-to-login-to-huggingface-hub-with-access-token/22498

Quickstart to get hugging face cli for login

pip install --upgrade huggingface_hub
huggingface-cli login

[x] Error to fix


Repository Not Found for url: https://huggingface.co/EleutherAI/opt-125m/resolve/main/config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/mmfs1/gscratch/scrubbed/emazuh/lm-evaluation-harness/main.py", line 108, in main() File "/mmfs1/gscratch/scrubbed/emazuh/lm-evaluation-harness/main.py", line 79, in main results = evaluator.simple_evaluate( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/scrubbed/emazuh/lm-evaluation-harness/lm_eval/utils.py", line 160, in _wrapper return fn(*args, kwargs) ^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/scrubbed/emazuh/lm-evaluation-harness/lm_eval/evaluator.py", line 64, in simple_evaluate lm = lm_eval.models.get_model(model).create_from_arg_string( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/scrubbed/emazuh/lm-evaluation-harness/lm_eval/base.py", line 114, in create_from_arg_string return cls(args, args2) ^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/scrubbed/emazuh/lm-evaluation-harness/lm_eval/models/huggingface.py", line 149, in init self._config = self.AUTO_CONFIG_CLASS.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/stf/emazuh/miniconda3/envs/llm/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 916, in from_pretrained config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/stf/emazuh/miniconda3/envs/llm/lib/python3.11/site-packages/transformers/configuration_utils.py", line 573, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/gscratch/stf/emazuh/miniconda3/envs/llm/lib/python3.11/site-packages/transformers/configuration_utils.py", line 628, in _get_config_dict resolved_config_file = cached_file( ^^^^^^^^^^^^ File "/gscratch/stf/emazuh/miniconda3/envs/llm/lib/python3.11/site-packages/transformers/utils/hub.py", line 424, in cached_file raise EnvironmentError( OSError: EleutherAI/opt-125m is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

emma-mens / transformers

Run baseline experiments for perplexities #2