Llama3 finetune error: ValueError(f"Invalid {class_type} class: '{component_name}'") from None

[l3utterfly]

My dataset config:

dataset:
  _component_: torchtune.datasets.chat_dataset
  train_on_input: True
  source: jsonl
  split: train
  data_files: train.jsonl
  conversation_style: sharegpt
  max_seq_len: 8192
  chat_format: None
seed: null
shuffle: True

From the tutorial here: https://pytorch.org/torchtune/main/tutorials/chat.html I see Llama3 requires chat format = None since the Tokenizer takes care of the special tokens

However, running: tune run --nproc_per_node 6 full_finetune_distributed --config ./llama3.yaml

I get this error:

Traceback (most recent call last):
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/recipes/full_finetune_distributed.py", line 520, in <module>
    sys.exit(recipe_main())
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/config/_parse.py", line 50, in wrapper
    sys.exit(recipe_main(conf))
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/recipes/full_finetune_distributed.py", line 514, in recipe_main
    recipe.setup(cfg=cfg)
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/recipes/full_finetune_distributed.py", line 206, in setup
    self._sampler, self._dataloader = self._setup_data(
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/recipes/full_finetune_distributed.py", line 337, in _setup_data
    ds = config.instantiate(
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/config/_instantiate.py", line 106, in instantiate
    return _instantiate_node(config, *args)
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/config/_instantiate.py", line 31, in _instantiate_node
    return _create_component(_component_, args, kwargs)
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/config/_instantiate.py", line 20, in _create_component
    return _component_(*args, **kwargs)
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/datasets/_chat.py", line 138, in chat_dataset
    chat_format=_get_chat_format(chat_format),
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/config/_utils.py", line 223, in _get_chat_format
    return _try_get_component("torchtune.data._chat_formats", chat_format, "ChatFormat")
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/config/_utils.py", line 195, in _try_get_component
    raise ValueError(f"Invalid {class_type} class: '{component_name}'") from None

Are there any example configs for using a custom dataset for finetuning llama3?

[RdoubleA]

When you want to set any yaml config field to None, you have to use null instead of None (like in JavaScript). Otherwise it will be interpreted as an object named None. For example, the seed field in your config is set to null instead of None.

Alternatively, you can just remove the chat_format field entirely and it will default to None correctly.

[l3utterfly]

@RdoubleA thank you for your reponse!

I tried removing it, I get this error: TypeError: chat_dataset() missing 1 required positional argument: 'chat_format'. Seems it is required?

I also tried setting it to "null", I got this error:

File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/config/_utils.py", line 223, in _get_chat_format
    return _try_get_component("torchtune.data._chat_formats", chat_format, "ChatFormat")
  File "/home/layla/src/Layla-datasets/.venv/lib/python3.10/site-packages/torchtune/config/_utils.py", line 193, in _try_get_component
    return _get_component_from_path(module_path + "." + component_name)
TypeError: can only concatenate str (not "NoneType") to str

[RdoubleA]

@l3utterfly How did you install torchtune, via pip? I would install the nightlies or from source, we've made chat format optional for llama3 and it's not available in stable yet :)

[l3utterfly]

@RdoubleA yes, I installed from pip. Thanks, I will try the nightly now

[l3utterfly]

Thank you, installing directly from source worked