Transformer Day: Get miniGPT up and running - Githubissues

LxMLS / lxmls-toolkit

Machine Learning applied to Natural Language Processing Toolkit used in the Lisbon Machine Learning Summer School

Other

223 stars 215 forks source link

Transformer Day: Get miniGPT up and running #177

Open ramon-astudillo opened 1 year ago

ramon-astudillo commented 1 year ago

Objective: Integrate miniGPT https://github.com/karpathy/minGPT into the toolkit.

[ ] Add a basic notebook labs/notebooks/transformers.pynb that runs a miniGPT.from_pretrained decoding with an example
- [ ] Port only the minGPT code needed (e.g. under labs/lxmls/miniGPT).
  - Less code is better for clarity. Code is already pretty compact so it may be ok as is.
  - Add License and ensure we attribute properly
- [ ] Load a basic model (GPT-2 large? XL?)
- [ ] Check a basic text completion exercise works, where we can give different prefixes are completions make more or less sense
- [ ] Other proposals @venelink @gonmelo ?

Branch: https://github.com/LxMLS/lxmls-toolkit/tree/transformer-day

Expected Finishing date: Ideally before June 12 meeting. If not during that week.

More ambitious objectives would include

[ ] Add a fine-tuning example