Kipok / NeMo-Skills

A pipeline to improve skills of large language models
https://kipok.github.io/NeMo-Skills/
Apache License 2.0
185 stars 41 forks source link

Large scale training #225

Closed wedu-nvidia closed 6 days ago

wedu-nvidia commented 6 days ago

Added the cache folder as the parent folder of training data path and tested on cluster. It worked. f" ++model.data.train_ds.index_mapping_dir='{os.path.dirname(os.path.abspath(params.training_data))}' "