Fine tuning on existing Dataset fails after setup in README

[psmukherjee009]

$ python3 pegasus/bin/train.py --params=aeslc_transformer \

--param_overrides=vocab_filename=ckpt/pegasus_ckpt/c4.unigram.newline.10pct.96000.model \ --train_init_checkpoint=ckpt/pegasus_ckpt/model.ckpt-1500000 \ --model_dir=ckpt/pegasus_ckpt/aeslc Traceback (most recent call last): File "pegasus/bin/train.py", line 17, in from pegasus.data import infeed File "/media/partha/ExternalStore/stanford/nlu/cs224u-master/TextSummarization/pegasus/pegasus/data/infeed.py", line 18, in from pegasus.data import all_datasets File "/media/partha/ExternalStore/stanford/nlu/cs224u-master/TextSummarization/pegasus/pegasus/data/all_datasets.py", line 18, in from pegasus.data import datasets File "/media/partha/ExternalStore/stanford/nlu/cs224u-master/TextSummarization/pegasus/pegasus/data/datasets.py", line 22, in import tensorflow_datasets as tfds File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/init.py", line 43, in from tensorflow_datasets.core import tf_compat File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/init.py", line 21, in tf_compat.ensure_tf_install() File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/tf_compat.py", line 63, in ensure_tf_install "This version of TensorFlow Datasets requires TensorFlow " ImportError: This version of TensorFlow Datasets requires TensorFlow version >= 2.1.0; Detected an installation of version 1.15.2. Please upgrade TensorFlow to proceed. $ cat requirements.txt absl-py mock numpy rouge-score sacrebleu sentencepiece tensorflow-text==1.15.0rc0 tensor2tensor==1.15.0 tfds-nightly tensorflow-gpu==1.15.2 $

$ uname -a Linux XXXX 5.8.0-44-generic #50~20.04.1-Ubuntu SMP Wed Feb 10 21:07:30 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ python3 Python 3.6.9 (default, Mar 7 2021, 15:28:30) [GCC 9.3.0] on linux

On upgrading tensorflow to latest, tensorflow-text and tensorflow-gpu needed to be upgraded and then it fails at " 'No module named 'tensorflow.contrib' for python project"

[JingqingZ]

Please check tensorflow_datasets==3.0

[psmukherjee009]

Yes, it got past that error. We should add this on requirements.txt file

But the validation fails at File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/download/checksums.py", line 128, in parse_sizes_checksums url, size, checksum = line.rsplit(' ', 2) ValueError: not enough values to unpack (expected 3, got 1)

[psmukherjee009]

The full trace File "pegasus/bin/train.py", line 94, in tf.app.run(main) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/absl/app.py", line 300, in run _run_main(main, args) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) File "pegasus/bin/train.py", line 89, in main max_steps=train_steps) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3035, in train rendezvous.raise_errors() File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 136, in raise_errors six.reraise(typ, value, traceback) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/six.py", line 703, in reraise raise value File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 3030, in train saving_listeners=saving_listeners) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1188, in _train_model_default input_fn, ModeKeys.TRAIN)) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1025, in _get_features_and_labels_from_input_fn self._call_input_fn(input_fn, mode)) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2987, in _call_input_fn return input_fn(kwargs) File "/media/partha/ExternalStore/stanford/nlu/cs224u-master/TextSummarization/pegasus/pegasus/data/infeed.py", line 41, in input_fn dataset = all_datasets.get_dataset(input_pattern, training) File "/media/partha/ExternalStore/stanford/nlu/cs224u-master/TextSummarization/pegasus/pegasus/data/all_datasets.py", line 52, in getdataset dataset, = builder.build(input_pattern, shuffle_files) File "/media/partha/ExternalStore/stanford/nlu/cs224u-master/TextSummarization/pegasus/pegasus/data/datasets.py", line 200, in build dataset, num_examples = self.load(build_name, split, shuffle_files) File "/media/partha/ExternalStore/stanford/nlu/cs224u-master/TextSummarization/pegasus/pegasus/data/datasets.py", line 158, in load data_dir=self.data_dir) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/api_utils.py", line 53, in disallow_positional_args_dec return fn(args, kwargs) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/registered.py", line 339, in load dbuilder.download_and_prepare(download_and_prepare_kwargs) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/api_utils.py", line 53, in disallow_positional_args_dec return fn(args, kwargs) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/dataset_builder.py", line 349, in download_and_prepare download_config=download_config) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/dataset_builder.py", line 767, in _make_download_manager register_checksums=download_config.register_checksums, File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/api_utils.py", line 53, in disallow_positional_args_dec return fn(*args, **kwargs) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/download/download_manager.py", line 199, in init self._sizes_checksums = checksums.get_all_sizes_checksums() File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/download/checksums.py", line 138, in get_all_sizes_checksums data = _get_sizes_checksums(path) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/download/checksums.py", line 117, in _get_sizes_checksums return parse_sizes_checksums(checksums_file) File "/home/partha/.virtualenvs/pegasus/lib/python3.6/site-packages/tensorflow_datasets/core/download/checksums.py", line 128, in parse_sizes_checksums url, size, checksum = line.rsplit(' ', 2) ValueError: not enough values to unpack (expected 3, got 1)

[psmukherjee009]

tensorflow-datasets==3.0.0 tfds-nightly==3.1.0.dev202006230105

Solved my issue.

[KHELIFA-Souhail]

tensorflow-datasets==3.0.0 tfds-nightly==3.1.0.dev202006230105

Solved my issue.

thanks, it works well