Closed mayank-nference closed 1 year ago
MaximumEntropy
commented
1 year ago Can you try binarizing your data (i.e. running when formatted as a JSONL file instead?) Using this script - https://github.com/NVIDIA/NeMo/blob/main/scripts/nlp_language_modeling/preprocess_data_for_megatron.py
File format:
{'text: "Example 1"}
{'text': "Example 2"}
mayank-nference
commented
1 year ago Thanks for the reply @MaximumEntropy I will convert the data as per the format you shared. Will ping here incase if still it doesn't work.
MaximumEntropy
commented
1 year ago Also, I noticed that you are trying to use the csv_map, I don't think we support that for GPT-based models yet, please use data_impl: mmap.
mayank-nference
commented
1 year ago @MaximumEntropy I am need small help in understanding following fields present in megatron_gpt_config.yaml
model.data.data_prefix, what is correct way to pass this parameter. Currently I am passing the value as [1.0,"/home/mayanksharma/nemo_data/train_binary_files/_text_document",2,"/home/mayanksharma/nemo_data/valid_binary_files/_text_document",3, "/home/mayanksharma/nemo_data/test_binary_files/_text_document"]model.data.splits_string, what is split_string?, currently I am using default values only 900,50,50 can I change these values?I need to understand how to load my fine-tuned GPT 20 Billion model for inference, I want to validate the trained model on the golden dataset.
I would really appreciate if you can help me with the above queries.
mayank-nference
commented
1 year ago @MaximumEntropy I want to fine-tune the pre-trained Nemo-Megatron-GPT 20Billlion model; on prompt classification, I got following error. The scrip which was using was for pre-training. The above script worked successfully after the changes you mentioned. However, I am trying to fintune on a downstream task for prompt classification for which I am referring the following pretraining script - https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/tuning/megatron_gpt_adapter_tuning.py and following config file - https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/tuning/conf/megatron_gpt_adapter_tuning_config.yaml. I am using following model for fine-tuning - https://huggingface.co/nvidia/nemo-megatron-gpt-20B/tree/main
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/mayanksharma/nemo_gpt_fintuning.py:109 in <module> │
│ │
│ 106 │
│ 107 │
│ 108 if __name__ == '__main__': │
│ ❱ 109 │ main() │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/core/config/hydra_runner.py:104 in │
│ wrapper │
│ │
│ 101 │ │ │ │ │
│ 102 │ │ │ │ # no return value from run_hydra() as it may sometime actually run the t │
│ 103 │ │ │ │ # multiple times (--multirun) │
│ ❱ 104 │ │ │ │ _run_hydra( │
│ 105 │ │ │ │ │ args_parser=_argparse_wrapper(args), │
│ 106 │ │ │ │ │ task_function=task_function, │
│ 107 │ │ │ │ │ config_path=config_path, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:377 in _run_hydra │
│ │
│ 374 │ │ if num_commands == 0: │
│ 375 │ │ │ args.run = True │
│ 376 │ │ if args.run: │
│ ❱ 377 │ │ │ run_and_report( │
│ 378 │ │ │ │ lambda: hydra.run( │
│ 379 │ │ │ │ │ config_name=config_name, │
│ 380 │ │ │ │ │ task_function=task_function, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:214 in │
│ run_and_report │
│ │
│ 211 │ │ return func() │
│ 212 │ except Exception as ex: │
│ 213 │ │ if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger(): │
│ ❱ 214 │ │ │ raise ex │
│ 215 │ │ else: │
│ 216 │ │ │ try: │
│ 217 │ │ │ │ if isinstance(ex, CompactHydraException): │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:211 in │
│ run_and_report │
│ │
│ 208 │
│ 209 def run_and_report(func: Any) -> Any: │
│ 210 │ try: │
│ ❱ 211 │ │ return func() │
│ 212 │ except Exception as ex: │
│ 213 │ │ if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger(): │
│ 214 │ │ │ raise ex │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/utils.py:378 in <lambda> │
│ │
│ 375 │ │ │ args.run = True │
│ 376 │ │ if args.run: │
│ 377 │ │ │ run_and_report( │
│ ❱ 378 │ │ │ │ lambda: hydra.run( │
│ 379 │ │ │ │ │ config_name=config_name, │
│ 380 │ │ │ │ │ task_function=task_function, │
│ 381 │ │ │ │ │ overrides=args.overrides, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/_internal/hydra.py:111 in run │
│ │
│ 108 │ │ callbacks.on_run_end(config=cfg, config_name=config_name, job_return=ret) │
│ 109 │ │ │
│ 110 │ │ # access the result to trigger an exception in case the job failed. │
│ ❱ 111 │ │ _ = ret.return_value │
│ 112 │ │ │
│ 113 │ │ return ret │
│ 114 │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/core/utils.py:233 in return_value │
│ │
│ 230 │ │ │ sys.stderr.write( │
│ 231 │ │ │ │ f"Error executing job with overrides: {self.overrides}" + os.linesep │
│ 232 │ │ │ ) │
│ ❱ 233 │ │ │ raise self._return_value │
│ 234 │ │
│ 235 │ @return_value.setter │
│ 236 │ def return_value(self, value: Any) -> None: │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/hydra/core/utils.py:160 in run_job │
│ │
│ 157 │ │ with env_override(hydra_cfg.hydra.job.env_set): │
│ 158 │ │ │ callbacks.on_job_start(config=config) │
│ 159 │ │ │ try: │
│ ❱ 160 │ │ │ │ ret.return_value = task_function(task_cfg) │
│ 161 │ │ │ │ ret.status = JobStatus.COMPLETED │
│ 162 │ │ │ except Exception as e: │
│ 163 │ │ │ │ ret.return_value = e │
│ │
│ /home/mayanksharma/nemo_gpt_fintuning.py:103 in main │
│ │
│ 100 │ │ │ cfg.model.restore_path, cfg.model, trainer=trainer, save_restore_connector=N │
│ 101 │ │ ) │
│ 102 │ else: │
│ ❱ 103 │ │ model = MegatronGPTAdapterLearningModel(cfg.model, trainer=trainer) │
│ 104 │ │
│ 105 │ trainer.fit(model) │
│ 106 │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/collections/nlp/models/language_modelin │
│ g/megatron_gpt_adapter_model.py:94 in __init__ │
│ │
│ 91 │ │ for _, module in self.frozen_model.named_modules(): │
│ 92 │ │ │ if isinstance(module, adapter_mixins.AdapterModuleMixin): │
│ 93 │ │ │ │ for adapter_key in self.adapter_name_keys: │
│ ❱ 94 │ │ │ │ │ module.add_adapter( │
│ 95 │ │ │ │ │ │ name=adapter_key, cfg=adapter_cfg, │
│ 96 │ │ │ │ │ ) │
│ 97 │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/nemo/core/classes/mixins/adapter_mixins.py:1 │
│ 64 in add_adapter │
│ │
│ 161 │ │ """ │
│ 162 │ │ # Convert to DictConfig from dict or Dataclass │
│ 163 │ │ if is_dataclass(cfg): │
│ ❱ 164 │ │ │ cfg = OmegaConf.structured(cfg) │
│ 165 │ │ │
│ 166 │ │ if not isinstance(cfg, DictConfig): │
│ 167 │ │ │ cfg = DictConfig(cfg) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:123 in structured │
│ │
│ 120 │ │ parent: Optional[BaseContainer] = None, │
│ 121 │ │ flags: Optional[Dict[str, bool]] = None, │
│ 122 │ ) -> Any: │
│ ❱ 123 │ │ return OmegaConf.create(obj, parent, flags) │
│ 124 │ │
│ 125 │ @staticmethod │
│ 126 │ @overload │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:176 in create │
│ │
│ 173 │ │ parent: Optional[BaseContainer] = None, │
│ 174 │ │ flags: Optional[Dict[str, bool]] = None, │
│ 175 │ ) -> Union[DictConfig, ListConfig]: │
│ ❱ 176 │ │ return OmegaConf._create_impl( │
│ 177 │ │ │ obj=obj, │
│ 178 │ │ │ parent=parent, │
│ 179 │ │ │ flags=flags, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:885 in _create_impl │
│ │
│ 882 │ │ │ │ │ │ │ f"Object of unsupported type: '{type(obj).__name__}'" │
│ 883 │ │ │ │ │ │ ) │
│ 884 │ │ except OmegaConfBaseException as e: │
│ ❱ 885 │ │ │ format_and_raise(node=None, key=None, value=None, msg=str(e), cause=e) │
│ 886 │ │ │ assert False │
│ 887 │ │
│ 888 │ @staticmethod │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:820 in format_and_raise │
│ │
│ 817 │ │ if type_override is not None: │
│ 818 │ │ │ ex = type_override(str(cause)) │
│ 819 │ │ │ ex.__dict__ = copy.deepcopy(cause.__dict__) │
│ ❱ 820 │ │ _raise(ex, cause) │
│ 821 │ │
│ 822 │ object_type: Optional[Type[Any]] │
│ 823 │ object_type_str: Optional[str] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:798 in _raise │
│ │
│ 795 │ │ ex.__cause__ = cause │
│ 796 │ else: │
│ 797 │ │ ex.__cause__ = None │
│ ❱ 798 │ raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace │
│ 799 │
│ 800 │
│ 801 def format_and_raise( │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:846 in _create_impl │
│ │
│ 843 │ │ │ │ │ else: │
│ 844 │ │ │ │ │ │ obj_type = OmegaConf.get_type(obj) │
│ 845 │ │ │ │ │ │ key_type, element_type = get_dict_key_value_types(obj_type) │
│ ❱ 846 │ │ │ │ │ │ return DictConfig( │
│ 847 │ │ │ │ │ │ │ content=obj, │
│ 848 │ │ │ │ │ │ │ parent=parent, │
│ 849 │ │ │ │ │ │ │ key_type=key_type, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:111 in __init__ │
│ │
│ 108 │ │ │ │ │ self.__dict__["_metadata"] = metadata │
│ 109 │ │ │ │ self._set_value(content, flags=flags) │
│ 110 │ │ except Exception as ex: │
│ ❱ 111 │ │ │ format_and_raise(node=None, key=key, value=None, cause=ex, msg=str(ex)) │
│ 112 │ │
│ 113 │ def __deepcopy__(self, memo: Dict[int, Any]) -> "DictConfig": │
│ 114 │ │ res = DictConfig(None) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:820 in format_and_raise │
│ │
│ 817 │ │ if type_override is not None: │
│ 818 │ │ │ ex = type_override(str(cause)) │
│ 819 │ │ │ ex.__dict__ = copy.deepcopy(cause.__dict__) │
│ ❱ 820 │ │ _raise(ex, cause) │
│ 821 │ │
│ 822 │ object_type: Optional[Type[Any]] │
│ 823 │ object_type_str: Optional[str] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:798 in _raise │
│ │
│ 795 │ │ ex.__cause__ = cause │
│ 796 │ else: │
│ 797 │ │ ex.__cause__ = None │
│ ❱ 798 │ raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace │
│ 799 │
│ 800 │
│ 801 def format_and_raise( │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:94 in __init__ │
│ │
│ 91 │ │ │ │ raise KeyValidationError(f"Unsupported key type {key_type}") │
│ 92 │ │ │ │
│ 93 │ │ │ if is_structured_config(content) or is_structured_config(ref_type): │
│ ❱ 94 │ │ │ │ self._set_value(content, flags=flags) │
│ 95 │ │ │ │ if is_structured_config_frozen(content) or is_structured_config_frozen( │
│ 96 │ │ │ │ │ ref_type │
│ 97 │ │ │ │ ): │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:647 in _set_value │
│ │
│ 644 │ │ │ self._set_value_impl(value, flags) │
│ 645 │ │ except Exception as e: │
│ 646 │ │ │ self.__dict__["_content"] = previous_content │
│ ❱ 647 │ │ │ raise e │
│ 648 │ │
│ 649 │ def _set_value_impl( │
│ 650 │ │ self, value: Any, flags: Optional[Dict[str, bool]] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:644 in _set_value │
│ │
│ 641 │ def _set_value(self, value: Any, flags: Optional[Dict[str, bool]] = None) -> None: │
│ 642 │ │ try: │
│ 643 │ │ │ previous_content = self.__dict__["_content"] │
│ ❱ 644 │ │ │ self._set_value_impl(value, flags) │
│ 645 │ │ except Exception as e: │
│ 646 │ │ │ self.__dict__["_content"] = previous_content │
│ 647 │ │ │ raise e │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:674 in │
│ _set_value_impl │
│ │
│ 671 │ │ │ if is_structured_config(value): │
│ 672 │ │ │ │ self._metadata.object_type = None │
│ 673 │ │ │ │ ao = self._get_flag("allow_objects") │
│ ❱ 674 │ │ │ │ data = get_structured_config_data(value, allow_objects=ao) │
│ 675 │ │ │ │ with flag_override(self, ["struct", "readonly"], False): │
│ 676 │ │ │ │ │ for k, v in data.items(): │
│ 677 │ │ │ │ │ │ self.__setitem__(k, v) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:494 in │
│ get_structured_config_data │
│ │
│ 491 │ obj: Any, allow_objects: Optional[bool] = None │
│ 492 ) -> Dict[str, Any]: │
│ 493 │ if is_dataclass(obj): │
│ ❱ 494 │ │ return get_dataclass_data(obj, allow_objects=allow_objects) │
│ 495 │ elif is_attr_class(obj): │
│ 496 │ │ return get_attr_data(obj, allow_objects=allow_objects) │
│ 497 │ else: │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:422 in │
│ get_dataclass_data │
│ │
│ 419 │ │ │ │ parent=dummy_parent, │
│ 420 │ │ │ ) │
│ 421 │ │ except (ValidationError, GrammarParseError) as ex: │
│ ❱ 422 │ │ │ format_and_raise( │
│ 423 │ │ │ │ node=dummy_parent, key=name, value=value, cause=ex, msg=str(ex) │
│ 424 │ │ │ ) │
│ 425 │ │ d[name]._set_parent(None) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:820 in format_and_raise │
│ │
│ 817 │ │ if type_override is not None: │
│ 818 │ │ │ ex = type_override(str(cause)) │
│ 819 │ │ │ ex.__dict__ = copy.deepcopy(cause.__dict__) │
│ ❱ 820 │ │ _raise(ex, cause) │
│ 821 │ │
│ 822 │ object_type: Optional[Type[Any]] │
│ 823 │ object_type_str: Optional[str] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:798 in _raise │
│ │
│ 795 │ │ ex.__cause__ = cause │
│ 796 │ else: │
│ 797 │ │ ex.__cause__ = None │
│ ❱ 798 │ raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace │
│ 799 │
│ 800 │
│ 801 def format_and_raise( │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:414 in │
│ get_dataclass_data │
│ │
│ 411 │ │ │ ) │
│ 412 │ │ │ format_and_raise(node=None, key=None, value=value, cause=e, msg=str(e)) │
│ 413 │ │ try: │
│ ❱ 414 │ │ │ d[name] = _maybe_wrap( │
│ 415 │ │ │ │ ref_type=type_, │
│ 416 │ │ │ │ is_optional=is_optional, │
│ 417 │ │ │ │ key=name, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:1090 in _maybe_wrap │
│ │
│ 1087 │ │ value._set_parent(parent) │
│ 1088 │ │ return value │
│ 1089 │ else: │
│ ❱ 1090 │ │ return _node_wrap( │
│ 1091 │ │ │ ref_type=ref_type, │
│ 1092 │ │ │ parent=parent, │
│ 1093 │ │ │ is_optional=is_optional, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/omegaconf.py:989 in _node_wrap │
│ │
│ 986 │ node: Node │
│ 987 │ if is_dict_annotation(ref_type) or (is_primitive_dict(value) and ref_type is Any): │
│ 988 │ │ key_type, element_type = get_dict_key_value_types(ref_type) │
│ ❱ 989 │ │ node = DictConfig( │
│ 990 │ │ │ content=value, │
│ 991 │ │ │ key=key, │
│ 992 │ │ │ parent=parent, │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:111 in __init__ │
│ │
│ 108 │ │ │ │ │ self.__dict__["_metadata"] = metadata │
│ 109 │ │ │ │ self._set_value(content, flags=flags) │
│ 110 │ │ except Exception as ex: │
│ ❱ 111 │ │ │ format_and_raise(node=None, key=key, value=None, cause=ex, msg=str(ex)) │
│ 112 │ │
│ 113 │ def __deepcopy__(self, memo: Dict[int, Any]) -> "DictConfig": │
│ 114 │ │ res = DictConfig(None) │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:900 in format_and_raise │
│ │
│ 897 │ │ ex.ref_type = ref_type │
│ 898 │ │ ex.ref_type_str = ref_type_str │
│ 899 │ │
│ ❱ 900 │ _raise(ex, cause) │
│ 901 │
│ 902 │
│ 903 def type_str(t: Any, include_module_name: bool = False) -> str: │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/_utils.py:798 in _raise │
│ │
│ 795 │ │ ex.__cause__ = cause │
│ 796 │ else: │
│ 797 │ │ ex.__cause__ = None │
│ ❱ 798 │ raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace │
│ 799 │
│ 800 │
│ 801 def format_and_raise( │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:94 in __init__ │
│ │
│ 91 │ │ │ │ raise KeyValidationError(f"Unsupported key type {key_type}") │
│ 92 │ │ │ │
│ 93 │ │ │ if is_structured_config(content) or is_structured_config(ref_type): │
│ ❱ 94 │ │ │ │ self._set_value(content, flags=flags) │
│ 95 │ │ │ │ if is_structured_config_frozen(content) or is_structured_config_frozen( │
│ 96 │ │ │ │ │ ref_type │
│ 97 │ │ │ │ ): │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:647 in _set_value │
│ │
│ 644 │ │ │ self._set_value_impl(value, flags) │
│ 645 │ │ except Exception as e: │
│ 646 │ │ │ self.__dict__["_content"] = previous_content │
│ ❱ 647 │ │ │ raise e │
│ 648 │ │
│ 649 │ def _set_value_impl( │
│ 650 │ │ self, value: Any, flags: Optional[Dict[str, bool]] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:644 in _set_value │
│ │
│ 641 │ def _set_value(self, value: Any, flags: Optional[Dict[str, bool]] = None) -> None: │
│ 642 │ │ try: │
│ 643 │ │ │ previous_content = self.__dict__["_content"] │
│ ❱ 644 │ │ │ self._set_value_impl(value, flags) │
│ 645 │ │ except Exception as e: │
│ 646 │ │ │ self.__dict__["_content"] = previous_content │
│ 647 │ │ │ raise e │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:658 in │
│ _set_value_impl │
│ │
│ 655 │ │ │ flags = {} │
│ 656 │ │ │
│ 657 │ │ assert not isinstance(value, ValueNode) │
│ ❱ 658 │ │ self._validate_set(key=None, value=value) │
│ 659 │ │ │
│ 660 │ │ if _is_none(value, resolve=True): │
│ 661 │ │ │ self.__dict__["_content"] = None │
│ │
│ /opt/conda/envs/llm_env/lib/python3.8/site-packages/omegaconf/dictconfig.py:200 in _validate_set │
│ │
│ 197 │ │ if is_container_annotation(target_type) and not is_container_annotation( │
│ 198 │ │ │ value_type │
│ 199 │ │ ): │
│ ❱ 200 │ │ │ raise ValidationError( │
│ 201 │ │ │ │ f"Cannot assign {type_str(value_type)} to {type_str(target_type)}" │
│ 202 │ │ │ ) │
│ 203 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValidationError: Cannot assign ResidualAddAdapterStrategyConfig to Dict[Any, Any]
full_key: adapter_strategy
object_type=NoneI am using following fine-tuning config
name: adapter_tuning_${model.new_tasks[0]}_max_epochs${trainer.max_epochs}_adapter_dim${model.adapter_tuning.adapter_dim}
trainer:
devices: 4
accelerator: gpu
num_nodes: 1
precision: 16
logger: False # logger provided by exp_manager
enable_checkpointing: False
replace_sampler_ddp: False
max_epochs: 10
max_steps: -1 # consumed_samples = global_step * micro_batch_size * data_parallel_size * accumulate_grad_batches
log_every_n_steps: 10
val_check_interval: 0.2
accumulate_grad_batches: 1
gradient_clip_val: 1.0
resume_from_checkpoint: null # The path to a checkpoint file to continue the training, restores the whole state including the epoch, step, LR schedulers, apex, etc.
benchmark: False
exp_manager:
explicit_log_dir: null
exp_dir: "nemo_experiment"
name: ${name}
create_wandb_logger: null
wandb_logger_kwargs:
project: null
name: null
resume_if_exists: True
resume_ignore_no_checkpoint: True
create_checkpoint_callback: True
checkpoint_callback_params:
monitor: val_loss
save_top_k: 1
mode: min
save_nemo_on_train_end: True # Should be false, correct prompt learning model file is saved at model.nemo_path set below,
filename: 'megatron_gpt_adapter_tuning--{val_loss:.3f}-{step}'
model_parallel_size: ${model.tensor_model_parallel_size}
save_best_model: True
model:
seed: 1234
nemo_path: ${exp_manager.exp_dir}/${name}.nemo # .nemo filename/absolute path to where the virtual prompt model parameters will be saved
virtual_prompt_style: 'no-prompts' # adapter tuning requires no virtual prompts
encoder_seq_length: 2048
gradient_as_bucket_view: false
tensor_model_parallel_size: 1 # intra-layer model parallelism
pipeline_model_parallel_size: 4 # inter-layer model parallelism
global_batch_size: 4
micro_batch_size: 1
restore_path: null # Path to an existing adapter .nemo model you wish to add new tasks to or run inference with
language_model_path: "/home/mayanksharma/nemo_model/nemo-megatron-gpt-20B/nemo_gpt20B_bf16_tp4.nemo" # Path to the GPT language model .nemo file, always required
existing_tasks: [] # List of tasks the model has already been p-tuned/prompt-tuned for, needed when a restore path is given
new_tasks: ["drug_to_target_cls"] # List of new tasknames to be prompt-tuned
task_templates: # Add more/replace tasks as needed, these are just examples
- taskname: "drug_to_target_cls" # Extreme Summarization
prompt_template: "<|VIRTUAL_PROMPT_0|> Sentence: {sentence} \nLabel: {label}"
total_virtual_tokens: 2048
virtual_token_splits: []
truncate_field: null
answer_only_loss: True
answer_field: "label"
# - taskname: "boolq" # The task name
# prompt_template: "Passage: {passage} \nQuestion: {question} \nAnswer: {answer}" # Prompt template for task, specify virtual prompt positions with <|VIRTUAL_PROMPT_#|>
# total_virtual_tokens: 0 # Sum of tokens in virtual_token_splits must add to this number. Can differ between new and existing tasks, but must match across all new tasks being tuned at the same time.
# virtual_token_splits: [] # number of virtual tokens to be inserted at each VIRTUAL PROMPT location, must add to total_virtual_tokens
# truncate_field: "passage" # The {field} in the prompt template whose text will be truncated if the input is too long, if null, inputs that are too long will just be skipped.
# answer_only_loss: True
# answer_field: "answer"
# - taskname: "intent_and_slot" # Intent Detection and Slot Filling
# prompt_template: "intent options: {intent_options} slot options: {slot_options} {utterance} \nintent: {intent} \nslot: {slot}"
# total_virtual_tokens: 0
# answer_only_loss: False
# virtual_token_splits: []
# truncate_field: null
# - taskname: "rte" # Recognizing Textual Entailment
# prompt_template: "sentence1: {premise} sentence2: {hypothesis} Answer: {answer}"
# total_virtual_tokens: 0
# virtual_token_splits: []
# truncate_field: null
# answer_only_loss: True
# answer_field: "answer"
# - taskname: "squad" # Standford Question-Answering
# prompt_template: "context: {context} question: {question} answer: {answer}"
# total_virtual_tokens: 0
# virtual_token_splits: []
# truncate_field: null
# answer_only_loss: True
# answer_field: "answer"
# - taskname: "arc-challenge" # Abstraction and Reasoning Challenge
# prompt_template: "question: {question} choices: {choices} answer: {answer}"
# total_virtual_tokens: 0
# virtual_token_splits: []
# truncate_field: null
# answer_only_loss: True
# answer_field: "answer"
# - taskname: "xsum" # Extreme Summarization
# prompt_template: "{source} Summary: {target}"
# total_virtual_tokens: 0
# virtual_token_splits: []
# truncate_field: null
# answer_only_loss: True
# answer_field: "target"
adapter_tuning:
type: 'parallel_adapter' # this should be either 'parallel_adapter' or 'linear_adapter'
adapter_dim: 50
adapter_dropout: 0.1
norm_position: 'pre' # This can be set to 'pre' or 'post', 'pre' is normally what is used.
column_init_method: 'xavier' # IGNORED if linear_adapter is used, options: xavier, zero or normal
row_init_method: 'zero' # IGNORED if linear_adapter is used, options: xavier, zero or normal
norm_type: 'mixedfusedlayernorm' # IGNORED if layer_adapter is used, options are ['layernorm', 'mixedfusedlayernorm']
data:
train_ds: "nemo_data/json_files/train.jsonl" # expects a list of paths to training data files
validation_ds: "nemo_data/json_files/valid.jsonl" # expects a paths to validation data files
add_eos: True
shuffle: True
num_workers: 24
pin_memory: True
optim:
name: fused_adam
lr: 1e-4
weight_decay: 0.01
betas:
- 0.9
- 0.98
sched:
name: CosineAnnealing
warmup_steps: 50
constant_steps: 0 # Constant steps should also be 0 when min_lr=0
min_lr: 0.0 # min_lr must be 0.0 for prompt learning
monitor: val_loss
reduce_on_plateau: falsePlease let me know in case I am using the wrong config or pertaining script.
MaximumEntropy
commented
1 year ago Regarding your first two questions
data_prefix - That is the correct format to provide the data prefix however it is not meant to take train/val/test separately. It is meant to take a dataset and internally create train/val/test splits from it. So if you specify something like 0.3,"/path/to/data_1",0.4,"/path/to/data_2",0.3, "/path/to/data_3"] it will sample from these three different datasets with ratio 0.3,0.4,0.3 respectively.@arendu do you have any idea about the adapter error?
mayank-nference
commented
1 year ago Thanks for the clear explanation, @MaximumEntropy.
mayank-nference
commented
1 year ago @MaximumEntropy @arendu, it's been 5 days can someone please help me with the adapter issue?
github-actions[bot]
commented
1 year ago This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
github-actions[bot]
commented
1 year ago This issue was closed because it has been inactive for 7 days since being marked as stale.
Describe the bug
Steps/Code to reproduce bug I am using scrip borrowed from - [https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/megatron_gpt_pretraining.py]()
Config File That I am Using I have borrowed config from here [https://github.com/NVIDIA/NeMo/blob/v1.12.0/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml]()
Inpu train,valid, test files I am Using Input small_train, small_valid, small_test files look like
Expected behaviour The train, valid and test files for gpt model should be processed correctly, as a similar file properly formatted for megatron-bert got correctly processed. for the megatron-bert model, the input train file looked something like this. It was a binary classification problem.
Environment overview (please complete the following information)
Linux nemo-training 4.19.0-22-cloud-amd64 #1 SMP Debian 4.19.260-1 (2022-09-29) x86_64 GNU/LinuxCloud(specify cloud provider - GCP)pip install nemo-toolkit["all"]andnemo-version: 1.12.0.Environment details
If an NVIDIA docker image is used, you don't need to specify these. Otherwise, please provide:
Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux1.13.01.7.6python3.8Additional context Please correct me if my input is formatted incorrectly for a gpt model. I could not find any suitable sample example for fine-tuning/pretraining the neo-megatron-20 billion models. Neither could I find any sample input files, or how they should be formatted to process correctly for GPT