search

Jsiewierski11 / web_crawler

Simple examples of building web crawlers using Beautiful Soup
0 stars 0 forks source link

Enhance Web Crawler with LLM for Movie Information Retrieval #2

Open mentatbot[bot] opened 2 months ago

mentatbot[bot] commented 2 months ago

This pull request integrates an open-source Language Model (LLM) to enhance the existing web crawler for fetching detailed movie information. The following changes have been made:

  1. Integration of Open-Source LLM API:

    • Added functionality to use LLaMA or Mistral LLMs to fetch movie information.
    • Created a function get_movie_info_from_llm to query the chosen LLM with the movie name and retrieve details.

  2. User Input Interface:

    • Implemented a terminal-based input for users to enter the movie name and choose the LLM.
    • Added input validation to ensure it is not empty.

  3. Fetching Initial Data Using Web Crawler:

    • Utilized the existing web crawler to fetch initial data such as movie URLs, basic info, and reviews.
    • Passed this data as context to the LLM to enhance its response.

  4. Fetching and Displaying Movie Information:

    • Used the chosen LLM to fetch movie summary, reviews, runtime, and reasons to watch.
    • Displayed the fetched information in a user-friendly format.

  5. Surprise Enhancement: Movie Recommendations:

    • Used the LLM to generate a list of similar movies based on the user's input.
    • Displayed the recommended movies along with the fetched information.

  6. Updated requirements.txt:

    • Added the transformers library to requirements.txt.

  7. Created README.md:

    • Added setup and run instructions to a new README.md file.

These enhancements provide a more comprehensive and user-friendly experience for retrieving movie information.

Closes #1

Thanks for using MentatBot. Give comments a :+1: or :-1: to help me improve!