[BUG] Lambda based Youtube video transcript download seems to be blocked

typex1 commented 1 month ago

Describe the bug

Filling out this field will help us investigate the issue efficiently. Providing detailed information allows us to set the appropriate priority. We appreciate your cooperation.
A clear and concise description of what the bug is.

Creating a new Bot including a knowledge base from Youtube transcriptions fails. Error in Frontend: "Failed to detect language: Could not retrieve a transcript for the video https://www.youtube.com/watch?v=Pv0cfsastFs"

This error message is misleading, because what seems to go wrong is not language detection specifically, but the whole transcript API seems to be not usable.

It took me quite some research to have good evidence that AWS owned IP addresses are (currently) blocked from Youtube transcription download. This applies at least to Lambda functions and Cloud9. Tested in us-east-1, eu-central-1 and ap-northeast-1.

To Reproduce

Filling out this field will help us investigate the issue efficiently. Providing detailed information allows us to set the appropriate priority. We appreciate your cooperation.
Steps to reproduce the behavior:

Create a new Bot, adding any arbitrary Youtube URL as part of the knowledge base. After the sync phase is done, it will show you the related error.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

typex1 commented 1 month ago

Error situation on the Bot overview list:

statefb commented 1 month ago

The problem is caused by youtube-transcript-api library. Issues:

We may remove the youtube feature in the future for KnowledgeBase integration. Thank you for your understanding.

aws-samples / bedrock-claude-chat