Closed mayank-nference closed 1 year ago
Can you try binarizing your data (i.e. running when formatted as a JSONL file instead?) Using this script - https://github.com/NVIDIA/NeMo/blob/main/scripts/nlp_language_modeling/preprocess_data_for_megatron.py
File format:
{'text: "Example 1"}
{'text': "Example 2"}
Thanks for the reply @MaximumEntropy I will convert the data as per the format you shared. Will ping here incase if still it doesn't work.
Also, I noticed that you are trying to use the csv_map
, I don't think we support that for GPT-based models yet, please use data_impl: mmap
.
@MaximumEntropy I am need small help in understanding following fields present in megatron_gpt_config.yaml
model.data.data_prefix
, what is correct way to pass this parameter. Currently I am passing the value as [1.0,"/home/mayanksharma/nemo_data/train_binary_files/_text_document",2,"/home/mayanksharma/nemo_data/valid_binary_files/_text_document",3, "/home/mayanksharma/nemo_data/test_binary_files/_text_document"]
model.data.splits_string
, what is split_string?, currently I am using default values only 900,50,50
can I change these values?I need to understand how to load my fine-tuned GPT 20 Billion model for inference, I want to validate the trained model on the golden dataset.
I would really appreciate if you can help me with the above queries.
@MaximumEntropy I want to fine-tune the pre-trained Nemo-Megatron-GPT 20Billlion model; on prompt classification, I got following error. The scrip which was using was for pre-training. The above script worked successfully after the changes you mentioned. However, I am trying to fintune on a downstream task for prompt classification for which I am referring the following pretraining script - https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/tuning/megatron_gpt_adapter_tuning.py and following config file - https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/tuning/conf/megatron_gpt_adapter_tuning_config.yaml. I am using following model for fine-tuning - https://huggingface.co/nvidia/nemo-megatron-gpt-20B/tree/main
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/mayanksharma/nemo_gpt_fintuning.py:109 in <module> │
│ │
│ 106 │
│ 107 │
│ 108 if __name__ == '__main__': │
│ ❱ 109 │ main() │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/core/config/hydra_runner.py:104 in │
│ wrapper │
│ │
│ 101 │ │ │ │ │
│ 102 │ │ │ │ # no return value from run_hydra() as it may sometime actually run the t │
│ 103 │ │ │ │ # multiple times (--multirun) │
│ ❱ 104 │ │ │ │ _run_hydra( │
│ 105 │ │ │ │ │ args_parser=_argparse_wrapper(args), │
│ 106 │ │ │ │ │ task_function=task_function, │
│ 107 │ │ │ │ │ config_path=config_path, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:377 in _run_hydra │
│ │
│ 374 │ │ if num_commands == 0: │
│ 375 │ │ │ args.run = True │
│ 376 │ │ if args.run: │
│ ❱ 377 │ │ │ run_and_report( │
│ 378 │ │ │ │ lambda: hydra.run( │
│ 379 │ │ │ │ │ config_name=config_name, │
│ 380 │ │ │ │ │ task_function=task_function, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:214 in │
│ run_and_report │
│ │
│ 211 │ │ return func() │
│ 212 │ except Exception as ex: │
│ 213 │ │ if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger(): │
│ ❱ 214 │ │ │ raise ex │
│ 215 │ │ else: │
│ 216 │ │ │ try: │
│ 217 │ │ │ │ if isinstance(ex, CompactHydraException): │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:211 in │
│ run_and_report │
│ │
│ 208 │
│ 209 def run_and_report(func: Any) -> Any: │
│ 210 │ try: │
│ ❱ 211 │ │ return func() │
│ 212 │ except Exception as ex: │
│ 213 │ │ if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger(): │
│ 214 │ │ │ raise ex │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:378 in <lambda> │
│ │
│ 375 │ │ │ args.run = True │
│ 376 │ │ if args.run: │
│ 377 │ │ │ run_and_report( │
│ ❱ 378 │ │ │ │ lambda: hydra.run( │
│ 379 │ │ │ │ │ config_name=config_name, │
│ 380 │ │ │ │ │ task_function=task_function, │
│ 381 │ │ │ │ │ overrides=args.overrides, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/hydra.py:111 in run │
│ │
│ 108 │ │ callbacks.on_run_end(config=cfg, config_name=config_name, job_return=ret) │
│ 109 │ │ │
│ 110 │ │ # access the result to trigger an exception in case the job failed. │
│ ❱ 111 │ │ _ = ret.return_value │
│ 112 │ │ │
│ 113 │ │ return ret │
│ 114 │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/core/utils.py:233 in return_value │
│ │
│ 230 │ │ │ sys.stderr.write( │
│ 231 │ │ │ │ f"Error executing job with overrides: {self.overrides}" + os.linesep │
│ 232 │ │ │ ) │
│ ❱ 233 │ │ │ raise self._return_value │
│ 234 │ │
│ 235 │ @return_value.setter │
│ 236 │ def return_value(self, value: Any) -> None: │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/core/utils.py:160 in run_job │
│ │
│ 157 │ │ with env_override(hydra_cfg.hydra.job.env_set): │
│ 158 │ │ │ callbacks.on_job_start(config=config) │
│ 159 │ │ │ try: │
│ ❱ 160 │ │ │ │ ret.return_value = task_function(task_cfg) │
│ 161 │ │ │ │ ret.status = JobStatus.COMPLETED │
│ 162 │ │ │ except Exception as e: │
│ 163 │ │ │ │ ret.return_value = e │
│ │
│ /home/mayanksharma/nemo_gpt_fintuning.py:103 in main │
│ │
│ 100 │ │ │ cfg.model.restore_path, cfg.model, trainer=trainer, save_restore_connector=N │
│ 101 │ │ ) │
│ 102 │ else: │
│ ❱ 103 │ │ model = MegatronGPTAdapterLearningModel(cfg.model, trainer=trainer) │
│ 104 │ │
│ 105 │ trainer.fit(model) │
│ 106 │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/collections/nlp/models/language_modelin │
│ g/megatron_gpt_adapter_model.py:94 in __init__ │
│ │
│ 91 │ │ for _, module in self.frozen_model.named_modules(): │
│ 92 │ │ │ if isinstance(module, adapter_mixins.AdapterModuleMixin): │
│ 93 │ │ │ │ for adapter_key in self.adapter_name_keys: │
│ ❱ 94 │ │ │ │ │ module.add_adapter( │
│ 95 │ │ │ │ │ │ name=adapter_key, cfg=adapter_cfg, │
│ 96 │ │ │ │ │ ) │
│ 97 │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/core/classes/mixins/adapter_mixins.py:1 │
│ 64 in add_adapter │
│ │
│ 161 │ │ """ │
│ 162 │ │ # Convert to DictConfig from dict or Dataclass │
│ 163 │ │ if is_dataclass(cfg): │
│ ❱ 164 │ │ │ cfg = OmegaConf.structured(cfg) │
│ 165 │ │ │
│ 166 │ │ if not isinstance(cfg, DictConfig): │
│ 167 │ │ │ cfg = DictConfig(cfg) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:123 in structured │
│ │
│ 120 │ │ parent: Optional[BaseContainer] = None, │
│ 121 │ │ flags: Optional[Dict[str, bool]] = None, │
│ 122 │ ) -> Any: │
│ ❱ 123 │ │ return OmegaConf.create(obj, parent, flags) │
│ 124 │ │
│ 125 │ @staticmethod │
│ 126 │ @overload │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:176 in create │
│ │
│ 173 │ │ parent: Optional[BaseContainer] = None, │
│ 174 │ │ flags: Optional[Dict[str, bool]] = None, │
│ 175 │ ) -> Union[DictConfig, ListConfig]: │
│ ❱ 176 │ │ return OmegaConf._create_impl( │
│ 177 │ │ │ obj=obj, │
│ 178 │ │ │ parent=parent, │
│ 179 │ │ │ flags=flags, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:885 in _create_impl │
│ │
│ 882 │ │ │ │ │ │ │ f"Object of unsupported type: '{type(obj).__name__}'" │
│ 883 │ │ │ │ │ │ ) │
│ 884 │ │ except OmegaConfBaseException as e: │
│ ❱ 885 │ │ │ format_and_raise(node=None, key=None, value=None, msg=str(e), cause=e) │
│ 886 │ │ │ assert False │
│ 887 │ │
│ 888 │ @staticmethod │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:820 in format_and_raise │
│ │
│ 817 │ │ if type_override is not None: │
│ 818 │ │ │ ex = type_override(str(cause)) │
│ 819 │ │ │ ex.__dict__ = copy.deepcopy(cause.__dict__) │
│ ❱ 820 │ │ _raise(ex, cause) │
│ 821 │ │
│ 822 │ object_type: Optional[Type[Any]] │
│ 823 │ object_type_str: Optional[str] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:798 in _raise │
│ │
│ 795 │ │ ex.__cause__ = cause │
│ 796 │ else: │
│ 797 │ │ ex.__cause__ = None │
│ ❱ 798 │ raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace │
│ 799 │
│ 800 │
│ 801 def format_and_raise( │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:846 in _create_impl │
│ │
│ 843 │ │ │ │ │ else: │
│ 844 │ │ │ │ │ │ obj_type = OmegaConf.get_type(obj) │
│ 845 │ │ │ │ │ │ key_type, element_type = get_dict_key_value_types(obj_type) │
│ ❱ 846 │ │ │ │ │ │ return DictConfig( │
│ 847 │ │ │ │ │ │ │ content=obj, │
│ 848 │ │ │ │ │ │ │ parent=parent, │
│ 849 │ │ │ │ │ │ │ key_type=key_type, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:111 in __init__ │
│ │
│ 108 │ │ │ │ │ self.__dict__["_metadata"] = metadata │
│ 109 │ │ │ │ self._set_value(content, flags=flags) │
│ 110 │ │ except Exception as ex: │
│ ❱ 111 │ │ │ format_and_raise(node=None, key=key, value=None, cause=ex, msg=str(ex)) │
│ 112 │ │
│ 113 │ def __deepcopy__(self, memo: Dict[int, Any]) -> "DictConfig": │
│ 114 │ │ res = DictConfig(None) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:820 in format_and_raise │
│ │
│ 817 │ │ if type_override is not None: │
│ 818 │ │ │ ex = type_override(str(cause)) │
│ 819 │ │ │ ex.__dict__ = copy.deepcopy(cause.__dict__) │
│ ❱ 820 │ │ _raise(ex, cause) │
│ 821 │ │
│ 822 │ object_type: Optional[Type[Any]] │
│ 823 │ object_type_str: Optional[str] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:798 in _raise │
│ │
│ 795 │ │ ex.__cause__ = cause │
│ 796 │ else: │
│ 797 │ │ ex.__cause__ = None │
│ ❱ 798 │ raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace │
│ 799 │
│ 800 │
│ 801 def format_and_raise( │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:94 in __init__ │
│ │
│ 91 │ │ │ │ raise KeyValidationError(f"Unsupported key type {key_type}") │
│ 92 │ │ │ │
│ 93 │ │ │ if is_structured_config(content) or is_structured_config(ref_type): │
│ ❱ 94 │ │ │ │ self._set_value(content, flags=flags) │
│ 95 │ │ │ │ if is_structured_config_frozen(content) or is_structured_config_frozen( │
│ 96 │ │ │ │ │ ref_type │
│ 97 │ │ │ │ ): │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:647 in _set_value │
│ │
│ 644 │ │ │ self._set_value_impl(value, flags) │
│ 645 │ │ except Exception as e: │
│ 646 │ │ │ self.__dict__["_content"] = previous_content │
│ ❱ 647 │ │ │ raise e │
│ 648 │ │
│ 649 │ def _set_value_impl( │
│ 650 │ │ self, value: Any, flags: Optional[Dict[str, bool]] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:644 in _set_value │
│ │
│ 641 │ def _set_value(self, value: Any, flags: Optional[Dict[str, bool]] = None) -> None: │
│ 642 │ │ try: │
│ 643 │ │ │ previous_content = self.__dict__["_content"] │
│ ❱ 644 │ │ │ self._set_value_impl(value, flags) │
│ 645 │ │ except Exception as e: │
│ 646 │ │ │ self.__dict__["_content"] = previous_content │
│ 647 │ │ │ raise e │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:674 in │
│ _set_value_impl │
│ │
│ 671 │ │ │ if is_structured_config(value): │
│ 672 │ │ │ │ self._metadata.object_type = None │
│ 673 │ │ │ │ ao = self._get_flag("allow_objects") │
│ ❱ 674 │ │ │ │ data = get_structured_config_data(value, allow_objects=ao) │
│ 675 │ │ │ │ with flag_override(self, ["struct", "readonly"], False): │
│ 676 │ │ │ │ │ for k, v in data.items(): │
│ 677 │ │ │ │ │ │ self.__setitem__(k, v) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:494 in │
│ get_structured_config_data │
│ │
│ 491 │ obj: Any, allow_objects: Optional[bool] = None │
│ 492 ) -> Dict[str, Any]: │
│ 493 │ if is_dataclass(obj): │
│ ❱ 494 │ │ return get_dataclass_data(obj, allow_objects=allow_objects) │
│ 495 │ elif is_attr_class(obj): │
│ 496 │ │ return get_attr_data(obj, allow_objects=allow_objects) │
│ 497 │ else: │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:422 in │
│ get_dataclass_data │
│ │
│ 419 │ │ │ │ parent=dummy_parent, │
│ 420 │ │ │ ) │
│ 421 │ │ except (ValidationError, GrammarParseError) as ex: │
│ ❱ 422 │ │ │ format_and_raise( │
│ 423 │ │ │ │ node=dummy_parent, key=name, value=value, cause=ex, msg=str(ex) │
│ 424 │ │ │ ) │
│ 425 │ │ d[name]._set_parent(None) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:820 in format_and_raise │
│ │
│ 817 │ │ if type_override is not None: │
│ 818 │ │ │ ex = type_override(str(cause)) │
│ 819 │ │ │ ex.__dict__ = copy.deepcopy(cause.__dict__) │
│ ❱ 820 │ │ _raise(ex, cause) │
│ 821 │ │
│ 822 │ object_type: Optional[Type[Any]] │
│ 823 │ object_type_str: Optional[str] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:798 in _raise │
│ │
│ 795 │ │ ex.__cause__ = cause │
│ 796 │ else: │
│ 797 │ │ ex.__cause__ = None │
│ ❱ 798 │ raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace │
│ 799 │
│ 800 │
│ 801 def format_and_raise( │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:414 in │
│ get_dataclass_data │
│ │
│ 411 │ │ │ ) │
│ 412 │ │ │ format_and_raise(node=None, key=None, value=value, cause=e, msg=str(e)) │
│ 413 │ │ try: │
│ ❱ 414 │ │ │ d[name] = _maybe_wrap( │
│ 415 │ │ │ │ ref_type=type_, │
│ 416 │ │ │ │ is_optional=is_optional, │
│ 417 │ │ │ │ key=name, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:1090 in _maybe_wrap │
│ │
│ 1087 │ │ value._set_parent(parent) │
│ 1088 │ │ return value │
│ 1089 │ else: │
│ ❱ 1090 │ │ return _node_wrap( │
│ 1091 │ │ │ ref_type=ref_type, │
│ 1092 │ │ │ parent=parent, │
│ 1093 │ │ │ is_optional=is_optional, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:989 in _node_wrap │
│ │
│ 986 │ node: Node │
│ 987 │ if is_dict_annotation(ref_type) or (is_primitive_dict(value) and ref_type is Any): │
│ 988 │ │ key_type, element_type = get_dict_key_value_types(ref_type) │
│ ❱ 989 │ │ node = DictConfig( │
│ 990 │ │ │ content=value, │
│ 991 │ │ │ key=key, │
│ 992 │ │ │ parent=parent, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:111 in __init__ │
│ │
│ 108 │ │ │ │ │ self.__dict__["_metadata"] = metadata │
│ 109 │ │ │ │ self._set_value(content, flags=flags) │
│ 110 │ │ except Exception as ex: │
│ ❱ 111 │ │ │ format_and_raise(node=None, key=key, value=None, cause=ex, msg=str(ex)) │
│ 112 │ │
│ 113 │ def __deepcopy__(self, memo: Dict[int, Any]) -> "DictConfig": │
│ 114 │ │ res = DictConfig(None) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:900 in format_and_raise │
│ │
│ 897 │ │ ex.ref_type = ref_type │
│ 898 │ │ ex.ref_type_str = ref_type_str │
│ 899 │ │
│ ❱ 900 │ _raise(ex, cause) │
│ 901 │
│ 902 │
│ 903 def type_str(t: Any, include_module_name: bool = False) -> str: │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:798 in _raise │
│ │
│ 795 │ │ ex.__cause__ = cause │
│ 796 │ else: │
│ 797 │ │ ex.__cause__ = None │
│ ❱ 798 │ raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace │
│ 799 │
│ 800 │
│ 801 def format_and_raise( │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:94 in __init__ │
│ │
│ 91 │ │ │ │ raise KeyValidationError(f"Unsupported key type {key_type}") │
│ 92 │ │ │ │
│ 93 │ │ │ if is_structured_config(content) or is_structured_config(ref_type): │
│ ❱ 94 │ │ │ │ self._set_value(content, flags=flags) │
│ 95 │ │ │ │ if is_structured_config_frozen(content) or is_structured_config_frozen( │
│ 96 │ │ │ │ │ ref_type │
│ 97 │ │ │ │ ): │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:647 in _set_value │
│ │
│ 644 │ │ │ self._set_value_impl(value, flags) │
│ 645 │ │ except Exception as e: │
│ 646 │ │ │ self.__dict__["_content"] = previous_content │
│ ❱ 647 │ │ │ raise e │
│ 648 │ │
│ 649 │ def _set_value_impl( │
│ 650 │ │ self, value: Any, flags: Optional[Dict[str, bool]] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:644 in _set_value │
│ │
│ 641 │ def _set_value(self, value: Any, flags: Optional[Dict[str, bool]] = None) -> None: │
│ 642 │ │ try: │
│ 643 │ │ │ previous_content = self.__dict__["_content"] │
│ ❱ 644 │ │ │ self._set_value_impl(value, flags) │
│ 645 │ │ except Exception as e: │
│ 646 │ │ │ self.__dict__["_content"] = previous_content │
│ 647 │ │ │ raise e │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:658 in │
│ _set_value_impl │
│ │
│ 655 │ │ │ flags = {} │
│ 656 │ │ │
│ 657 │ │ assert not isinstance(value, ValueNode) │
│ ❱ 658 │ │ self._validate_set(key=None, value=value) │
│ 659 │ │ │
│ 660 │ │ if _is_none(value, resolve=True): │
│ 661 │ │ │ self.__dict__["_content"] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:200 in _validate_set │
│ │
│ 197 │ │ if is_container_annotation(target_type) and not is_container_annotation( │
│ 198 │ │ │ value_type │
│ 199 │ │ ): │
│ ❱ 200 │ │ │ raise ValidationError( │
│ 201 │ │ │ │ f"Cannot assign {type_str(value_type)} to {type_str(target_type)}" │
│ 202 │ │ │ ) │
│ 203 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: Cannot assign ResidualAddAdapterStrategyConfig to Dict[Any, Any]
full_key: adapter_strategy
object_type=None
I am using following fine-tuning config
name: adapter_tuning_${model.new_tasks[0]}_max_epochs${trainer.max_epochs}_adapter_dim${model.adapter_tuning.adapter_dim}
trainer:
devices: 4
accelerator: gpu
num_nodes: 1
precision: 16
logger: False # logger provided by exp_manager
enable_checkpointing: False
replace_sampler_ddp: False
max_epochs: 10
max_steps: -1 # consumed_samples = global_step * micro_batch_size * data_parallel_size * accumulate_grad_batches
log_every_n_steps: 10
val_check_interval: 0.2
accumulate_grad_batches: 1
gradient_clip_val: 1.0
resume_from_checkpoint: null # The path to a checkpoint file to continue the training, restores the whole state including the epoch, step, LR schedulers, apex, etc.
benchmark: False
exp_manager:
explicit_log_dir: null
exp_dir: "nemo_experiment"
name: ${name}
create_wandb_logger: null
wandb_logger_kwargs:
project: null
name: null
resume_if_exists: True
resume_ignore_no_checkpoint: True
create_checkpoint_callback: True
checkpoint_callback_params:
monitor: val_loss
save_top_k: 1
mode: min
save_nemo_on_train_end: True # Should be false, correct prompt learning model file is saved at model.nemo_path set below,
filename: 'megatron_gpt_adapter_tuning--{val_loss:.3f}-{step}'
model_parallel_size: ${model.tensor_model_parallel_size}
save_best_model: True
model:
seed: 1234
nemo_path: ${exp_manager.exp_dir}/${name}.nemo # .nemo filename/absolute path to where the virtual prompt model parameters will be saved
virtual_prompt_style: 'no-prompts' # adapter tuning requires no virtual prompts
encoder_seq_length: 2048
gradient_as_bucket_view: false
tensor_model_parallel_size: 1 # intra-layer model parallelism
pipeline_model_parallel_size: 4 # inter-layer model parallelism
global_batch_size: 4
micro_batch_size: 1
restore_path: null # Path to an existing adapter .nemo model you wish to add new tasks to or run inference with
language_model_path: "/home/mayanksharma/nemo_model/nemo-megatron-gpt-20B/nemo_gpt20B_bf16_tp4.nemo" # Path to the GPT language model .nemo file, always required
existing_tasks: [] # List of tasks the model has already been p-tuned/prompt-tuned for, needed when a restore path is given
new_tasks: ["drug_to_target_cls"] # List of new tasknames to be prompt-tuned
task_templates: # Add more/replace tasks as needed, these are just examples
- taskname: "drug_to_target_cls" # Extreme Summarization
prompt_template: "<|VIRTUAL_PROMPT_0|> Sentence: {sentence} \nLabel: {label}"
total_virtual_tokens: 2048
virtual_token_splits: []
truncate_field: null
answer_only_loss: True
answer_field: "label"
# - taskname: "boolq" # The task name
# prompt_template: "Passage: {passage} \nQuestion: {question} \nAnswer: {answer}" # Prompt template for task, specify virtual prompt positions with <|VIRTUAL_PROMPT_#|>
# total_virtual_tokens: 0 # Sum of tokens in virtual_token_splits must add to this number. Can differ between new and existing tasks, but must match across all new tasks being tuned at the same time.
# virtual_token_splits: [] # number of virtual tokens to be inserted at each VIRTUAL PROMPT location, must add to total_virtual_tokens
# truncate_field: "passage" # The {field} in the prompt template whose text will be truncated if the input is too long, if null, inputs that are too long will just be skipped.
# answer_only_loss: True
# answer_field: "answer"
# - taskname: "intent_and_slot" # Intent Detection and Slot Filling
# prompt_template: "intent options: {intent_options} slot options: {slot_options} {utterance} \nintent: {intent} \nslot: {slot}"
# total_virtual_tokens: 0
# answer_only_loss: False
# virtual_token_splits: []
# truncate_field: null
# - taskname: "rte" # Recognizing Textual Entailment
# prompt_template: "sentence1: {premise} sentence2: {hypothesis} Answer: {answer}"
# total_virtual_tokens: 0
# virtual_token_splits: []
# truncate_field: null
# answer_only_loss: True
# answer_field: "answer"
# - taskname: "squad" # Standford Question-Answering
# prompt_template: "context: {context} question: {question} answer: {answer}"
# total_virtual_tokens: 0
# virtual_token_splits: []
# truncate_field: null
# answer_only_loss: True
# answer_field: "answer"
# - taskname: "arc-challenge" # Abstraction and Reasoning Challenge
# prompt_template: "question: {question} choices: {choices} answer: {answer}"
# total_virtual_tokens: 0
# virtual_token_splits: []
# truncate_field: null
# answer_only_loss: True
# answer_field: "answer"
# - taskname: "xsum" # Extreme Summarization
# prompt_template: "{source} Summary: {target}"
# total_virtual_tokens: 0
# virtual_token_splits: []
# truncate_field: null
# answer_only_loss: True
# answer_field: "target"
adapter_tuning:
type: 'parallel_adapter' # this should be either 'parallel_adapter' or 'linear_adapter'
adapter_dim: 50
adapter_dropout: 0.1
norm_position: 'pre' # This can be set to 'pre' or 'post', 'pre' is normally what is used.
column_init_method: 'xavier' # IGNORED if linear_adapter is used, options: xavier, zero or normal
row_init_method: 'zero' # IGNORED if linear_adapter is used, options: xavier, zero or normal
norm_type: 'mixedfusedlayernorm' # IGNORED if layer_adapter is used, options are ['layernorm', 'mixedfusedlayernorm']
data:
train_ds: "nemo_data/json_files/train.jsonl" # expects a list of paths to training data files
validation_ds: "nemo_data/json_files/valid.jsonl" # expects a paths to validation data files
add_eos: True
shuffle: True
num_workers: 24
pin_memory: True
optim:
name: fused_adam
lr: 1e-4
weight_decay: 0.01
betas:
- 0.9
- 0.98
sched:
name: CosineAnnealing
warmup_steps: 50
constant_steps: 0 # Constant steps should also be 0 when min_lr=0
min_lr: 0.0 # min_lr must be 0.0 for prompt learning
monitor: val_loss
reduce_on_plateau: false
Please let me know in case I am using the wrong config or pertaining script.
Regarding your first two questions
data_prefix
- That is the correct format to provide the data prefix however it is not meant to take train/val/test separately. It is meant to take a dataset and internally create train/val/test splits from it. So if you specify something like 0.3,"/path/to/data_1",0.4,"/path/to/data_2",0.3, "/path/to/data_3"]
it will sample from these three different datasets with ratio 0.3,0.4,0.3 respectively.@arendu do you have any idea about the adapter error?
Thanks for the clear explanation, @MaximumEntropy.
@MaximumEntropy @arendu, it's been 5 days can someone please help me with the adapter issue?
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
This issue was closed because it has been inactive for 7 days since being marked as stale.
Describe the bug
Steps/Code to reproduce bug I am using scrip borrowed from - [https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/megatron_gpt_pretraining.py]()
Config File That I am Using I have borrowed config from here [https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml]()
Inpu train,valid, test files I am Using Input small_train, small_valid, small_test files look like
Expected behaviour The train, valid and test files for gpt model should be processed correctly, as a similar file properly formatted for megatron-bert got correctly processed. for the megatron-bert model, the input train file looked something like this. It was a binary classification problem.
Environment overview (please complete the following information)
Linux nemo-training 4.19.0-22-cloud-amd64 #1 SMP Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux
Cloud(specify cloud provider - GCP)
pip install nemo-toolkit["all"]
andnemo-version: 1.12.0
.Environment details
If an NVIDIA docker image is used, you don't need to specify these. Otherwise, please provide:
Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux
1.13.0
1.7.6
python3.8
Additional context Please correct me if my input is formatted incorrectly for a gpt model. I could not find any suitable sample example for fine-tuning/pretraining the neo-megatron-20 billion models. Neither could I find any sample input files, or how they should be formatted to process correctly for GPT