search

Lightning-Universe / lightning-transformers

Flexible components pairing 🤗 Transformers with :zap: Pytorch Lightning
https://lightning-transformers.readthedocs.io
Apache License 2.0
607 stars 77 forks source link

[bug] trainer got unexpected keyword argument 'truncated_bptt_steps' #212

Closed mglowacki100 closed 2 years ago

mglowacki100 commented 2 years ago

🐛 Bug

To Reproduce

  1. Environment: google colab. I've also get the same error on ubuntu 18.
  2. !pip install lightning-transformers and restart as requested by google colab
  3. !pl-transformers-train task=nlp/summarization dataset=nlp/summarization/cnn_dailymail backbone.pretrained_model_name_or_path=t5-base
  4. I've got error: TypeError: Error instantiating 'pytorch_lightning.trainer.trainer.Trainer' : __init__() got an unexpected keyword argument 'truncated_bptt_steps' also there is a warning from hydra.
  5. I suspect that there is some mismatch in libraries versions (e.g. lightning and hydra need to be downgraded or lightning-transformer updated).

Full traceback:

/usr/local/lib/python3.7/dist-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
  warnings.warn(msg, UserWarning)
dataset:
  _target_: lightning_transformers.task.nlp.summarization.CNNDailyMailSummarizationDataModule
  cfg:
    batch_size: ${training.batch_size}
    num_workers: ${training.num_workers}
    dataset_name: cnn_dailymail
    dataset_config_name: 3.0.0
    train_file: null
    validation_file: null
    test_file: null
    train_val_split: null
    max_samples: null
    cache_dir: null
    padding: max_length
    truncation: only_first
    preprocessing_num_workers: 1
    load_from_cache_file: true
    max_length: 128
    limit_train_samples: null
    limit_val_samples: null
    limit_test_samples: null
    max_source_length: 1024
    max_target_length: 128
task:
  _recursive_: false
  _target_: lightning_transformers.task.nlp.summarization.SummarizationTransformer
  optimizer: ${optimizer}
  scheduler: ${scheduler}
  backbone: ${backbone}
  downstream_model_type: transformers.AutoModelForSeq2SeqLM
  cfg:
    use_stemmer: true
    rouge_newline_sep: true
    val_target_max_length: 142
    num_beams: null
    compute_generate_metrics: true
tokenizer:
  _target_: transformers.AutoTokenizer.from_pretrained
  pretrained_model_name_or_path: ${backbone.pretrained_model_name_or_path}
  use_fast: true
backbone:
  pretrained_model_name_or_path: t5-base
optimizer:
  _target_: torch.optim.AdamW
  lr: ${training.lr}
  weight_decay: 0.001
scheduler:
  _target_: transformers.get_linear_schedule_with_warmup
  num_training_steps: -1
  num_warmup_steps: 0.1
training:
  run_test_after_fit: true
  lr: 5.0e-05
  output_dir: .
  batch_size: 16
  num_workers: 16
trainer:
  _target_: pytorch_lightning.Trainer
  logger: true
  checkpoint_callback: true
  callbacks: null
  default_root_dir: null
  gradient_clip_val: 0.0
  process_position: 0
  num_nodes: 1
  num_processes: 1
  gpus: null
  auto_select_gpus: false
  tpu_cores: null
  log_gpu_memory: null
  progress_bar_refresh_rate: 1
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 1
  min_epochs: 1
  max_steps: null
  min_steps: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  val_check_interval: 1.0
  flush_logs_every_n_steps: 100
  log_every_n_steps: 50
  accelerator: null
  sync_batchnorm: false
  precision: 32
  weights_summary: top
  weights_save_path: null
  num_sanity_val_steps: 2
  truncated_bptt_steps: null
  resume_from_checkpoint: null
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_epoch: false
  auto_lr_find: false
  replace_sampler_ddp: true
  terminate_on_nan: false
  auto_scale_batch_size: false
  prepare_data_per_node: true
  plugins: null
  amp_backend: native
  amp_level: O2
  move_metrics_to_cpu: false
experiment_name: ${now:%Y-%m-%d}_${now:%H-%M-%S}
log: false
ignore_warnings: true

Downloading: 100% 1.17k/1.17k [00:00<00:00, 980kB/s]
Downloading: 100% 773k/773k [00:00<00:00, 866kB/s]
Downloading: 100% 1.32M/1.32M [00:01<00:00, 1.27MB/s]
Downloading: 9.35kB [00:00, 7.86MB/s]       
Downloading: 9.50kB [00:00, 8.57MB/s]       
Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.28 GiB, post-processed: Unknown size, total: 1.82 GiB) to /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234...
  0% 0/5 [00:00<?, ?it/s]
Downloading:   0% 0.00/159M [00:00<?, ?B/s]
Downloading:   2% 3.23M/159M [00:00<00:04, 32.3MB/s]
Downloading:   4% 6.79M/159M [00:00<00:04, 34.2MB/s]
Downloading:   6% 10.2M/159M [00:00<00:04, 34.2MB/s]
Downloading:   9% 13.6M/159M [00:00<00:04, 34.3MB/s]
Downloading:  11% 17.3M/159M [00:00<00:04, 35.1MB/s]
Downloading:  13% 20.8M/159M [00:00<00:03, 35.0MB/s]
Downloading:  15% 24.3M/159M [00:00<00:03, 34.6MB/s]
Downloading:  18% 27.8M/159M [00:00<00:03, 34.6MB/s]
Downloading:  20% 31.2M/159M [00:00<00:03, 34.6MB/s]
Downloading:  22% 34.9M/159M [00:01<00:03, 35.2MB/s]
Downloading:  24% 38.4M/159M [00:01<00:03, 35.1MB/s]
Downloading:  26% 41.9M/159M [00:01<00:03, 34.6MB/s]
Downloading:  29% 45.5M/159M [00:01<00:03, 35.1MB/s]
Downloading:  31% 49.1M/159M [00:01<00:03, 35.1MB/s]
Downloading:  33% 52.6M/159M [00:01<00:03, 34.9MB/s]
Downloading:  35% 56.2M/159M [00:01<00:02, 35.3MB/s]
Downloading:  38% 59.9M/159M [00:01<00:02, 35.8MB/s]
Downloading:  40% 63.5M/159M [00:01<00:02, 34.7MB/s]
Downloading:  42% 67.0M/159M [00:01<00:03, 30.3MB/s]
Downloading:  44% 70.1M/159M [00:02<00:02, 30.5MB/s]
Downloading:  47% 74.1M/159M [00:02<00:02, 33.2MB/s]
Downloading:  49% 78.2M/159M [00:02<00:02, 35.4MB/s]
Downloading:  52% 81.8M/159M [00:02<00:02, 35.4MB/s]
Downloading:  54% 85.7M/159M [00:02<00:01, 36.5MB/s]
Downloading:  56% 89.6M/159M [00:02<00:01, 37.1MB/s]
Downloading:  59% 93.3M/159M [00:02<00:01, 36.7MB/s]
Downloading:  61% 97.0M/159M [00:02<00:01, 35.5MB/s]
Downloading:  63% 101M/159M [00:02<00:01, 35.4MB/s] 
Downloading:  66% 104M/159M [00:02<00:01, 36.1MB/s]
Downloading:  68% 108M/159M [00:03<00:01, 36.7MB/s]
Downloading:  71% 112M/159M [00:03<00:01, 36.3MB/s]
Downloading:  73% 115M/159M [00:03<00:01, 36.3MB/s]
Downloading:  75% 119M/159M [00:03<00:01, 36.8MB/s]
Downloading:  78% 123M/159M [00:03<00:00, 36.9MB/s]
Downloading:  80% 127M/159M [00:03<00:00, 37.3MB/s]
Downloading:  82% 131M/159M [00:03<00:00, 37.9MB/s]
Downloading:  85% 135M/159M [00:03<00:00, 36.9MB/s]
Downloading:  87% 139M/159M [00:03<00:00, 38.0MB/s]
Downloading:  90% 142M/159M [00:04<00:00, 38.2MB/s]
Downloading:  92% 146M/159M [00:04<00:00, 38.5MB/s]
Downloading:  95% 150M/159M [00:04<00:00, 38.8MB/s]
Downloading:  97% 154M/159M [00:04<00:00, 38.7MB/s]
Downloading: 100% 159M/159M [00:04<00:00, 35.9MB/s]
 20% 1/5 [00:05<00:23,  5.91s/it]
Downloading:   0% 0.00/376M [00:00<?, ?B/s]
Downloading:   1% 3.06M/376M [00:00<00:12, 30.6MB/s]
Downloading:   2% 6.43M/376M [00:00<00:11, 32.4MB/s]
Downloading:   3% 9.88M/376M [00:00<00:10, 33.4MB/s]
Downloading:   4% 13.5M/376M [00:00<00:10, 34.5MB/s]
Downloading:   5% 17.1M/376M [00:00<00:10, 35.2MB/s]
Downloading:   5% 20.7M/376M [00:00<00:10, 34.6MB/s]
Downloading:   6% 24.3M/376M [00:00<00:09, 35.3MB/s]
Downloading:   7% 27.9M/376M [00:00<00:09, 35.5MB/s]
Downloading:   8% 31.5M/376M [00:00<00:09, 35.3MB/s]
Downloading:   9% 35.0M/376M [00:01<00:09, 34.8MB/s]
Downloading:  10% 38.5M/376M [00:01<00:09, 34.5MB/s]
Downloading:  11% 41.9M/376M [00:01<00:09, 33.5MB/s]
Downloading:  12% 45.5M/376M [00:01<00:09, 34.1MB/s]
Downloading:  13% 48.9M/376M [00:01<00:09, 33.6MB/s]
Downloading:  14% 52.6M/376M [00:01<00:09, 34.4MB/s]
Downloading:  15% 56.2M/376M [00:01<00:09, 34.9MB/s]
Downloading:  16% 59.7M/376M [00:01<00:10, 29.4MB/s]
Downloading:  17% 63.6M/376M [00:01<00:09, 32.0MB/s]
Downloading:  18% 67.7M/376M [00:01<00:08, 34.6MB/s]
Downloading:  19% 71.3M/376M [00:02<00:09, 32.0MB/s]
Downloading:  20% 75.2M/376M [00:02<00:08, 33.7MB/s]
Downloading:  21% 79.1M/376M [00:02<00:08, 35.3MB/s]
Downloading:  22% 83.1M/376M [00:02<00:08, 36.6MB/s]
Downloading:  23% 87.0M/376M [00:02<00:07, 37.2MB/s]
Downloading:  24% 91.3M/376M [00:02<00:07, 38.9MB/s]
Downloading:  25% 95.3M/376M [00:02<00:07, 39.3MB/s]
Downloading:  27% 99.6M/376M [00:02<00:06, 40.6MB/s]
Downloading:  28% 104M/376M [00:02<00:06, 40.6MB/s] 
Downloading:  29% 108M/376M [00:03<00:06, 40.6MB/s]
Downloading:  30% 112M/376M [00:03<00:06, 41.1MB/s]
Downloading:  31% 116M/376M [00:03<00:06, 41.3MB/s]
Downloading:  32% 120M/376M [00:03<00:06, 40.9MB/s]
Downloading:  33% 124M/376M [00:03<00:06, 40.1MB/s]
Downloading:  34% 129M/376M [00:03<00:06, 40.5MB/s]
Downloading:  35% 133M/376M [00:03<00:05, 41.0MB/s]
Downloading:  36% 137M/376M [00:03<00:05, 41.2MB/s]
Downloading:  38% 141M/376M [00:03<00:05, 41.9MB/s]
Downloading:  39% 146M/376M [00:03<00:05, 41.6MB/s]
Downloading:  40% 150M/376M [00:04<00:05, 41.1MB/s]
Downloading:  41% 154M/376M [00:04<00:05, 41.3MB/s]
Downloading:  42% 158M/376M [00:04<00:05, 41.1MB/s]
Downloading:  43% 162M/376M [00:04<00:05, 39.2MB/s]
Downloading:  44% 166M/376M [00:04<00:05, 40.0MB/s]
Downloading:  45% 170M/376M [00:04<00:05, 40.5MB/s]
Downloading:  46% 175M/376M [00:04<00:05, 39.8MB/s]
Downloading:  48% 179M/376M [00:04<00:04, 40.4MB/s]
Downloading:  49% 183M/376M [00:04<00:04, 40.6MB/s]
Downloading:  50% 187M/376M [00:04<00:04, 40.4MB/s]
Downloading:  51% 191M/376M [00:05<00:04, 40.4MB/s]
Downloading:  52% 195M/376M [00:05<00:05, 34.2MB/s]
Downloading:  53% 199M/376M [00:05<00:05, 32.9MB/s]
Downloading:  54% 203M/376M [00:05<00:04, 34.9MB/s]
Downloading:  55% 207M/376M [00:05<00:04, 35.9MB/s]
Downloading:  56% 211M/376M [00:05<00:04, 37.2MB/s]
Downloading:  57% 214M/376M [00:05<00:04, 37.5MB/s]
Downloading:  58% 218M/376M [00:05<00:04, 37.4MB/s]
Downloading:  59% 222M/376M [00:05<00:03, 38.6MB/s]
Downloading:  60% 227M/376M [00:06<00:03, 39.6MB/s]
Downloading:  61% 231M/376M [00:06<00:03, 39.9MB/s]
Downloading:  62% 235M/376M [00:06<00:03, 40.0MB/s]
Downloading:  63% 239M/376M [00:06<00:03, 38.2MB/s]
Downloading:  65% 243M/376M [00:06<00:03, 38.6MB/s]
Downloading:  66% 247M/376M [00:06<00:03, 39.8MB/s]
Downloading:  67% 251M/376M [00:06<00:03, 39.9MB/s]
Downloading:  68% 255M/376M [00:06<00:02, 40.3MB/s]
Downloading:  69% 259M/376M [00:06<00:02, 41.4MB/s]
Downloading:  70% 264M/376M [00:06<00:02, 41.9MB/s]
Downloading:  71% 268M/376M [00:07<00:02, 41.7MB/s]
Downloading:  72% 272M/376M [00:07<00:02, 41.6MB/s]
Downloading:  74% 276M/376M [00:07<00:02, 42.1MB/s]
Downloading:  75% 281M/376M [00:07<00:02, 40.8MB/s]
Downloading:  76% 285M/376M [00:07<00:02, 41.6MB/s]
Downloading:  77% 289M/376M [00:07<00:02, 41.8MB/s]
Downloading:  78% 293M/376M [00:07<00:01, 41.7MB/s]
Downloading:  79% 298M/376M [00:07<00:01, 42.1MB/s]
Downloading:  80% 302M/376M [00:07<00:01, 41.9MB/s]
Downloading:  81% 306M/376M [00:07<00:01, 40.6MB/s]
Downloading:  83% 310M/376M [00:08<00:01, 41.0MB/s]
Downloading:  84% 314M/376M [00:08<00:01, 40.9MB/s]
Downloading:  85% 319M/376M [00:08<00:01, 41.8MB/s]
Downloading:  86% 323M/376M [00:08<00:01, 40.9MB/s]
Downloading:  87% 327M/376M [00:08<00:01, 41.1MB/s]
Downloading:  88% 331M/376M [00:08<00:01, 33.0MB/s]
Downloading:  89% 335M/376M [00:08<00:01, 34.2MB/s]
Downloading:  90% 339M/376M [00:08<00:01, 35.4MB/s]
Downloading:  91% 343M/376M [00:08<00:00, 36.6MB/s]
Downloading:  92% 347M/376M [00:09<00:00, 37.8MB/s]
Downloading:  93% 351M/376M [00:09<00:00, 37.9MB/s]
Downloading:  94% 355M/376M [00:09<00:00, 39.0MB/s]
Downloading:  96% 359M/376M [00:09<00:00, 38.3MB/s]
Downloading:  97% 363M/376M [00:09<00:00, 39.4MB/s]
Downloading:  98% 367M/376M [00:09<00:00, 39.8MB/s]
Downloading:  99% 371M/376M [00:09<00:00, 39.0MB/s]
Downloading: 100% 376M/376M [00:09<00:00, 38.3MB/s]
 40% 2/5 [00:17<00:28,  9.50s/it]
Downloading: 2.11MB [00:00, 35.5MB/s]      
 60% 3/5 [00:18<00:10,  5.46s/it]
Downloading:   0% 0.00/12.3M [00:00<?, ?B/s]
Downloading:  45% 5.50M/12.3M [00:00<00:00, 54.9MB/s]
Downloading:  98% 12.0M/12.3M [00:00<00:00, 60.7MB/s]
Downloading: 18.0MB [00:00, 57.2MB/s]                
Downloading: 23.8MB [00:00, 56.7MB/s]
Downloading: 29.5MB [00:00, 55.0MB/s]
Downloading: 35.0MB [00:00, 54.1MB/s]
Downloading: 40.4MB [00:00, 54.3MB/s]
Downloading: 46.4MB [00:00, 55.5MB/s]
 80% 4/5 [00:24<00:05,  5.79s/it]
Downloading: 2.43MB [00:00, 36.6MB/s]      
100% 5/5 [00:25<00:00,  5.10s/it]
100% 5/5 [01:59<00:00, 23.96s/it]
Dataset cnn_dailymail downloaded and prepared to /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234. Subsequent calls will reuse this data.
100% 3/3 [00:00<00:00, 239.53it/s]
100% 288/288 [13:51<00:00,  2.89s/ba]
100% 14/14 [00:39<00:00,  2.80s/ba]
100% 12/12 [00:33<00:00,  2.79s/ba]
Downloading: 100% 850M/850M [00:24<00:00, 35.7MB/s]
Error executing job with overrides: ['task=nlp/summarization', 'dataset=nlp/summarization/cnn_dailymail', 'backbone.pretrained_model_name_or_path=t5-base']
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 62, in _call_target
    return _target_(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/env_vars_connector.py", line 38, in insert_env_defaults
    return fn(self, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'truncated_bptt_steps'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/lightning_transformers/cli/train.py", line 84, in hydra_entry
    main(cfg)
  File "/usr/local/lib/python3.7/dist-packages/lightning_transformers/cli/train.py", line 78, in main
    logger=logger,
  File "/usr/local/lib/python3.7/dist-packages/lightning_transformers/cli/train.py", line 58, in run
    logger=logger,
  File "/usr/local/lib/python3.7/dist-packages/lightning_transformers/core/instantiator.py", line 111, in trainer
    return self.instantiate(cfg, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/lightning_transformers/core/instantiator.py", line 114, in instantiate
    return hydra.utils.instantiate(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 180, in instantiate
    return instantiate_node(config, *args, recursive=_recursive_, convert=_convert_)
  File "/usr/local/lib/python3.7/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 249, in instantiate_node
    return _call_target(_target_, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 66, in _call_target
    ).with_traceback(sys.exc_info()[2])
  File "/usr/local/lib/python3.7/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 62, in _call_target
    return _target_(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/connectors/env_vars_connector.py", line 38, in insert_env_defaults
    return fn(self, **kwargs)
TypeError: Error instantiating 'pytorch_lightning.trainer.trainer.Trainer' : __init__() got an unexpected keyword argument 'truncated_bptt_steps'

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Google colab install traceback:

Collecting lightning-transformers
  Downloading lightning_transformers-0.1.0-py3-none-any.whl (99 kB)
     |████████████████████████████████| 99 kB 3.0 MB/s 
Collecting pytorch-lightning>=1.2.4
  Downloading pytorch_lightning-1.5.3-py3-none-any.whl (523 kB)
     |████████████████████████████████| 523 kB 26.1 MB/s 
Collecting transformers
  Downloading transformers-4.12.5-py3-none-any.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 32.9 MB/s 
Collecting fairscale
  Downloading fairscale-0.4.3.tar.gz (232 kB)
     |████████████████████████████████| 232 kB 43.3 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting hydra-core>=1.1.0.dev4
  Downloading hydra_core-1.1.1-py3-none-any.whl (145 kB)
     |████████████████████████████████| 145 kB 50.6 MB/s 
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from lightning-transformers) (1.19.5)
Collecting torchmetrics>=0.2.0
  Downloading torchmetrics-0.6.0-py3-none-any.whl (329 kB)
     |████████████████████████████████| 329 kB 48.7 MB/s 
Requirement already satisfied: torch>=1.6 in /usr/local/lib/python3.7/dist-packages (from lightning-transformers) (1.10.0+cu111)
Collecting datasets
  Downloading datasets-1.16.1-py3-none-any.whl (298 kB)
     |████████████████████████████████| 298 kB 46.6 MB/s 
Collecting sentencepiece
  Downloading sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 37.0 MB/s 
Collecting rouge-score
  Downloading rouge_score-0.0.4-py2.py3-none-any.whl (22 kB)
Collecting omegaconf==2.1.*
  Downloading omegaconf-2.1.1-py3-none-any.whl (74 kB)
     |████████████████████████████████| 74 kB 3.2 MB/s 
Collecting antlr4-python3-runtime==4.8
  Downloading antlr4-python3-runtime-4.8.tar.gz (112 kB)
     |████████████████████████████████| 112 kB 52.0 MB/s 
Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from hydra-core>=1.1.0.dev4->lightning-transformers) (5.4.0)
Collecting PyYAML>=5.1.0
  Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
     |████████████████████████████████| 596 kB 47.7 MB/s 
Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning>=1.2.4->lightning-transformers) (21.3)
Collecting pyDeprecate==0.3.1
  Downloading pyDeprecate-0.3.1-py3-none-any.whl (10 kB)
Collecting fsspec[http]!=2021.06.0,>=2021.05.0
  Downloading fsspec-2021.11.0-py3-none-any.whl (132 kB)
     |████████████████████████████████| 132 kB 44.7 MB/s 
Collecting future>=0.17.1
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 27.0 MB/s 
Requirement already satisfied: tqdm>=4.41.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning>=1.2.4->lightning-transformers) (4.62.3)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning>=1.2.4->lightning-transformers) (2.7.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from pytorch-lightning>=1.2.4->lightning-transformers) (3.10.0.2)
Collecting aiohttp
  Downloading aiohttp-3.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 46.6 MB/s 
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.2.4->lightning-transformers) (2.23.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=17.0->pytorch-lightning>=1.2.4->lightning-transformers) (3.0.6)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (1.35.0)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (0.37.0)
Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (3.17.3)
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (1.42.0)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (57.4.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (3.3.6)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (0.6.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (0.4.6)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (0.12.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (1.8.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (1.0.1)
Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from absl-py>=0.4->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (1.15.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (4.2.4)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (4.7.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (1.3.0)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (4.8.2)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (3.6.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (0.4.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.2.4->lightning-transformers) (2021.10.8)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.2.4->lightning-transformers) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.2.4->lightning-transformers) (1.24.3)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.2.4->lightning-transformers) (2.10)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->pytorch-lightning>=1.2.4->lightning-transformers) (3.1.1)
Collecting frozenlist>=1.1.1
  Downloading frozenlist-1.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (192 kB)
     |████████████████████████████████| 192 kB 51.7 MB/s 
Collecting aiosignal>=1.1.2
  Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.2.4->lightning-transformers) (2.0.7)
Collecting yarl<2.0,>=1.0
  Downloading yarl-1.7.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)
     |████████████████████████████████| 271 kB 48.5 MB/s 
Collecting async-timeout<5.0,>=4.0.0a3
  Downloading async_timeout-4.0.1-py3-none-any.whl (5.7 kB)
Collecting multidict<7.0,>=4.5
  Downloading multidict-5.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (160 kB)
     |████████████████████████████████| 160 kB 46.6 MB/s 
Collecting asynctest==0.13.0
  Downloading asynctest-0.13.0-py3-none-any.whl (26 kB)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning>=1.2.4->lightning-transformers) (21.2.0)
Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from datasets->lightning-transformers) (0.3.4)
Requirement already satisfied: pyarrow!=4.0.0,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from datasets->lightning-transformers) (3.0.0)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from datasets->lightning-transformers) (1.1.5)
Collecting huggingface-hub<1.0.0,>=0.1.0
  Downloading huggingface_hub-0.1.2-py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 6.3 MB/s 
Requirement already satisfied: multiprocess in /usr/local/lib/python3.7/dist-packages (from datasets->lightning-transformers) (0.70.12.2)
Collecting xxhash
  Downloading xxhash-2.0.2-cp37-cp37m-manylinux2010_x86_64.whl (243 kB)
     |████████████████████████████████| 243 kB 49.7 MB/s 
Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0.0,>=0.1.0->datasets->lightning-transformers) (3.4.0)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets->lightning-transformers) (2.8.2)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->datasets->lightning-transformers) (2018.9)
Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from rouge-score->lightning-transformers) (3.2.5)
Collecting tokenizers<0.11,>=0.10.1
  Downloading tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3 MB)
     |████████████████████████████████| 3.3 MB 40.0 MB/s 
Collecting sacremoses
  Downloading sacremoses-0.0.46-py3-none-any.whl (895 kB)
     |████████████████████████████████| 895 kB 44.8 MB/s 
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers->lightning-transformers) (2019.12.20)
Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers->lightning-transformers) (1.1.0)
Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers->lightning-transformers) (7.1.2)
Building wheels for collected packages: antlr4-python3-runtime, future, fairscale
  Building wheel for antlr4-python3-runtime (setup.py) ... done
  Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.8-py3-none-any.whl size=141230 sha256=b10cc3e02bb8c1b4cc30250eca332daaddc2d6d17255ffb9043cb74c13744cf0
  Stored in directory: /root/.cache/pip/wheels/ca/33/b7/336836125fc9bb4ceaa4376d8abca10ca8bc84ddc824baea6c
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491070 sha256=c47f9bacbcdda8ff437f9cbfe2ec7897d0d68c15e3f0afdbe2236ca8a5c83ec8
  Stored in directory: /root/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0
  Building wheel for fairscale (PEP 517) ... done
  Created wheel for fairscale: filename=fairscale-0.4.3-py3-none-any.whl size=288187 sha256=ff3cd935e6ca410d80e9a2ccbc6244a1394074f6569d54e7bfa8ecad6af43a1c
  Stored in directory: /root/.cache/pip/wheels/4c/a1/94/5fc7119d83755b65de6a5d1664704926d8664731ec296d31e7
Successfully built antlr4-python3-runtime future fairscale
Installing collected packages: multidict, frozenlist, yarl, asynctest, async-timeout, aiosignal, PyYAML, fsspec, antlr4-python3-runtime, aiohttp, xxhash, torchmetrics, tokenizers, sacremoses, pyDeprecate, omegaconf, huggingface-hub, future, transformers, sentencepiece, rouge-score, pytorch-lightning, hydra-core, fairscale, datasets, lightning-transformers
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 3.13
    Uninstalling PyYAML-3.13:
      Successfully uninstalled PyYAML-3.13
  Attempting uninstall: future
    Found existing installation: future 0.16.0
    Uninstalling future-0.16.0:
      Successfully uninstalled future-0.16.0
Successfully installed PyYAML-6.0 aiohttp-3.8.1 aiosignal-1.2.0 antlr4-python3-runtime-4.8 async-timeout-4.0.1 asynctest-0.13.0 datasets-1.16.1 fairscale-0.4.3 frozenlist-1.2.0 fsspec-2021.11.0 future-0.18.2 huggingface-hub-0.1.2 hydra-core-1.1.1 lightning-transformers-0.1.0 multidict-5.2.0 omegaconf-2.1.1 pyDeprecate-0.3.1 pytorch-lightning-1.5.3 rouge-score-0.0.4 sacremoses-0.0.46 sentencepiece-0.1.96 tokenizers-0.10.3 torchmetrics-0.6.0 transformers-4.12.5 xxhash-2.0.2 yarl-1.7.2
WARNING: The following packages were previously imported in this runtime:
  [pydevd_plugins]
You must restart the runtime in order to use newly installed versions.
mglowacki100 commented 2 years ago

Downgrading pytorch-ligthning seems to fix an issue: !pip install pytorch-lightning==1.4 or installing directly from git (this one is preffered): !pip install git+https://github.com/PytorchLightning/lightning-transformers.git@master --upgrade For version 1.4, I've got error in multi-gpu setting.

Borda commented 2 years ago

It shall be fixed now on master... :rabbit:

ChristopherMarais commented 2 years ago

For me downgrading pytorch-lightning also worked just to add to this conversation.

swcrazyfan commented 2 years ago

This is still a problem when I run inside Colab. When I downgrade, it works. Will this be fixed?