ludwig-ai / ludwig

Low-code framework for building custom LLMs, neural networks, and other AI models
http://ludwig.ai
Apache License 2.0
11.11k stars 1.19k forks source link

Add Ludwig config json to output directory containing model weights #3961

Closed sanjaydasgupta closed 6 months ago

sanjaydasgupta commented 6 months ago

Is your feature request related to a problem? Please describe. Reproducing other users' experiences is important in many contexts, so automatically publishing the Ludwig config along with a fine-tuned model's weights on huggingface will be a helpful step. This objective can be achieved by just adding a json file containing the ludwig configuration to the output directory that also contains the model weights.

Describe the use case The primary motivation is to facilitate reproduction of other users' experiments. Publishing the Ludwig configuration is a first step in this direction.

Describe the solution you'd like Modify the code of method Ludwig.api.train() as follows:

  1. Add a new parameter save_ludwig_config_with_weights: bool = False that the caller can use to indicate whether or not they want to upload the user-provided ludwig config to HF along with the model weights.
  2. After the lines that save the model weights to the output directory (line# 768) add code to save a json or yaml file containing the user-provided config to the directory containing the model weights.

No other change is necessary as the existing code that uploads model weights and other associated files to HF (in/under Ludwig.api.upload_to_hf_hub(...)) automatically picks up all json and yaml files from the output directory.

Describe alternatives you've considered ~Modifying the code under LudwigModel's upload_to_hf_hub(...) is the only option. The changes should add a file containing the configuration (either _ludwigconfig.yaml or _ludwigconfig.json -- to be decided) to the experiment's _model/modelweights directory. The implementation of upload_to_hf_hub already looks for and uploads all yaml and json files in the _model/modelweights directory to huggingface.~

~Users may not always want to upload the configuration, so an additional parameter upload_config: bool = False for upload_to_hf_hub(...) will be added to select the required behavior.~

Additional context This is similar to the feature described here.

sanjaydasgupta commented 6 months ago

self-assign

sanjaydasgupta commented 6 months ago

take

alexsherstinsky commented 6 months ago

Implemented in #3965