Closed ofirpress closed 2 years ago
At first glance these results don't look unreasonable. This slice of metrics is clearly from late in training when you're well past the point of overfitting. Presumably the train/validation losses are much closer during the first epoch.
Note that, from the API docs, "we periodically calculate metrics on batches of validation data during training time". Since this is done at the batch level, the high variance in these validation metrics is not surprising -- they don't look crazy to me. In fact, based on this batch data, it looks like the overall validation accuracy would come out to something close to the ~5% reported in the paper. I recommend evaluating the accuracy across the whole validation set to see if this is the case.
I'm not sure what would cause the occasional discrepancy between validation sequence/token accuracy. You're correct that if you pruned to single token completions, they should agree.
Thanks!
Hi! I'm trying to replicate your results with the GPT-3 API. This is how I've preprocessed the train file:
{"prompt": "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\n###\n\n", "completion": " 72"}
And here's the first line of the valid file:{"prompt": "Janet\u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?\n\n###\n\n", "completion": " 18"}
I'm trying to replicate the result from the paper which doesn't use the intermediate steps:
I run this command:
openai api fine_tunes.create -t train.jsonl -v valid.jsonl -m curie --n_epochs 20
The training loss and accuracy looks great, but the validation accuracy just goes crazy throughout all of training:
I'm also not sure how in some iterations validation_sequence_accuracy and validation_token_accuracy even though all validation completions are of length 1 (I pruned all the prompts where the answer were longer).
I'd be grateful for any advice here. I've tried different LRs but that didn't seem to help.
Thanks!