fix maximum input token count exceeds limit of 4096

aws-samples / amazon-bedrock-workshop

This is a workshop designed for Amazon Bedrock a foundational model service.

https://catalog.us-east-1.prod.workshops.aws/workshops/a4bdb007-5600-4368-81c5-ff5b4154f518/en-US/20-intro

MIT No Attribution

1.27k stars 545 forks source link

fix maximum input token count exceeds limit of 4096 #244

Open FireballDWF opened 2 months ago

FireballDWF commented 2 months ago

Issue #, if available: 224

Description of changes: Fix Maximum input token count 4919 exceeds limit of 4096 for train data in 03_Model_customization/03_continued_pretraining_titan_text.ipynb

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.