YoutubeLoader throws 'module not found' error

devbydaniel commented 10 months ago

System Info

Running langchain in Nextjs app

"langchain": "^0.0.175",
"next": "^13.5.2",
"react": "18.2.0",

Who can help?

@eyurtsev

Information

[X] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[X] Document Loaders
[ ] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction

I'm using the YoutubeLoader as described in the docs. The YoutubeLoader is part of a route handler in my NextJs application.

// in youtube service
export async function getYoutubeTranscript(youtubeUrl: string) {
  const loader = YoutubeLoader.createFromUrl(youtubeUrl, {
    language: 'en',
    addVideoInfo: true,
  })
  const docs = await loader.load()
  return docs
    .map((doc) => doc.pageContent)
    .join(' ')
    .replaceAll('\n', ' ')
}

// in route handler
const transcript = await getYoutubeTranscript(link)

When using the route, I get an error because of a missing dependency - on my machine the route still works though.

Manually installing canvas via npm i canvas fixes the issue, but when I deploy to vercel, the function exceeds the maximum size limit of 50 MB compressed, because the canvas package is 45 MB compressed...

Not manually installing the dependency causes the route on Vercel to always return a 405 because of the missing dependency:

I realize that this is not caused by the YoutubeLoader itself, but still hoping for any pointers or ideas how to fix this.

Expected behavior

YoutubeLoader works without throwing a 'module not found' error

dosubot[bot] commented 10 months ago

🤖

Hello @devbydaniel,

I'm here to assist you with your issue. I'm an AI developed to help users like you navigate through technical issues and questions.

From the context you've provided, it seems like you're encountering a 'module not found' error with the 'canvas' module when using the YoutubeLoader in the LangChain framework. However, based on the information available in the LangChain repository, the 'canvas' module is not a direct dependency of the YoutubeLoader or any other part of the LangChain codebase.

The YoutubeLoader does have several Python dependencies, including google.auth.transport.requests, google.oauth2.service_account, google.oauth2.credentials, google_auth_oauthlib.flow, youtube_transcript_api, pytube, and googleapiclient.discovery. These are all Python packages and are not related to the 'canvas' module.

The 'canvas' module is a Node.js package, and it seems like your project is a Next.js application, which is a JavaScript framework. It's possible that the 'canvas' module is a dependency of another part of your project, not the LangChain framework.

If the 'canvas' module is indeed necessary for your project, you might be encountering the 'module not found' error because the 'canvas' module is not installed in your project's node_modules directory. You can try installing it using npm or yarn.

However, as you've mentioned, the 'canvas' module is quite large and might cause your function to exceed the maximum size limit when deploying to Vercel. You might need to look into alternative solutions or workarounds for this issue.

I hope this information helps! If you have any more questions or need further clarification, feel free to ask.

Best, Dosu

Sources

libs/langchain/langchain/document_loaders/youtube.py

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

devbydaniel commented 10 months ago

Not helpful unfortunately

dosubot[bot] commented 10 months ago

@baskaryan, I could use your expertise to resolve this one.

dosubot[bot] commented 7 months ago

Hi, @devbydaniel,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. The issue involves a 'module not found' error when using the YoutubeLoader in a Next.js application. The error is caused by a missing dependency, and manually installing the dependency fixes the issue but causes the function to exceed the maximum size limit when deployed to Vercel. There have been attempts to provide assistance, but it seems that the issue remains unresolved.

Could you please let us know if this issue is still relevant to the latest version of the LangChain repository? If it is, please feel free to provide an update or any additional information. Otherwise, you are welcome to close the issue yourself, or it will be automatically closed in 7 days. Thank you for your understanding and contribution to LangChain!

langchain-ai / langchain