search

mosaicml / llm-foundry

LLM training code for Databricks foundation models
https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm
Apache License 2.0
3.99k stars 525 forks source link

Make `fc_type` a dict to pass fc kwargs through #1201

Closed snarayan21 closed 4 months ago

snarayan21 commented 4 months ago

Some custom FC layers will need custom kwargs. This PR enables that by changing fc_type from str to Union[str, Dict], and converting it to dict thereafter.

Default configs have also been moved to their own file, keeping those defaults easily accessible.

We will likely seek to deprecate fc_type down the line and replace it with fc_config instead.

snarayan21 commented 4 months ago

finally fixed the ci errors lfg