Website Scraper Plugin Only Scanning Homepage

[ashmilhussain]

The website scraper plugin is currently only scanning the homepage. Modify the to plugin scrap a single page or subpages based on a flag. This flag should control whether the crawler scans just the homepage or includes subpages as well, providing more flexibility in how pages are crawled.

Use flag : is_scan_child 'true' : scraper both main page and subpages 'false' : scrap only main page

[YashBaviskar1]

Hey there! I would love to try to solve this issue. Maybe i will try to recursively scan the page whenever the flag is true. thank you

[ashmilhussain]

Hey @YashBaviskar1 ,

Assigning this issue to you, happy coding

If any assistance required meet our team here : https://join.slack.com/t/theailounge/shared_invite/zt-2ogkrruyf-FPOHuPr5hdqXl34bDWjHjw

[agberoz]

@YashBaviskar1 any updates on this?

[YashBaviskar1]

Hey there Yes, I was looking through the code base , from what i understand i have to change url_reader.py I was just figuring out a way on how to use def load(self) in Class UrlReader(DocsReader) to recursively call itself, is this the right approach.

Sorry for the delay

[agberoz]

@YashBaviskar1, Recursion might work, but it's risky for web scrapers due to infinite loops or stack overflow from nested or circular links.

I would also suggest using a data structure like a queue. Here's how it can work:

1. Start with the base URL.
2. Add child URLs to the queue as you process each page.
3. Ensure that each URL is only visited once (you can use a set to track visited URLs).
4. Process each URL iteratively from the queue.

[YashBaviskar1]

Yup thank you! I will try to do what you have said, it is pretty intuitive

[YashBaviskar1]

@ashmilhussain @agberoz, Hello there so I tried to make amend on what you had said and this is what i have came up with :

• In Class Website I added is_scan_child flag in __init__ and changed the fetch_data accordingly

  class Website:
  """
  Website class for interacting with website data.
  """
  def __init__(self, website_url: str, is_scan_child : bool = False):
      self.connection = {}
      self.is_scan_child = is_scan_child #default is False 
      self.params = {
          'url': website_url,
      }
  
  def fetch_data(self):
      base_reader = UrlReader({
          "type": "url",
          "path": [self.params.get('url')],
          "is_scan_child" : self.is_scan_child
      })
      data = base_reader.load()
      return data

• Now in Class UrlReader I did as you instructed, initialized a url_queue queue and a visited_url set and used find_all to find all the href tags and append in queue if the is_scan_child is true. The process iterates through the entire queue, while the set keeps track of visited links.

  class UrlReader:
  def __init__(self, source):
      self.source = source
      self.visited_url = set()
  
  def load(self):
      out = []
      is_scan_child = self.source.get("is_scan_child", False)
      url_queue = [] #defining a queue to capture sub_urls 
      if "path" in self.source:
          urls = self.source["path"]
          for url in urls:
              url_queue.append(url)
          while url_queue : 
              url = url_queue.pop(0)
              if url in self.visited_url :
                  continue
              self.visited_url.add(url)
              try:
                  response = requests.get(url)
                  if response.status_code == 200:
                      soup = BeautifulSoup(response.content, 'html.parser')
                      if is_scan_child :
                          for a in soup.find_all('a', href = True) :
                              absolute_url = urljoin(url, a['href'])
                              if absolute_url not in self.visited_url :
                                  url_queue.append(absolute_url)
                          #print(f" Sub Urls are : {url_queue}")
                      tag = soup.body
                      text = ''.join(list(tag.strings)[:-1])
                      metadata = {
                          "path": url
                      }
                      out.append({"content": str(text), "metadata": metadata})
                  else:
                      logger.critical(f"Failed to retrieve content, status code: {response.status_code}")
              except Exception as e:
                  logger.error(e)
      return out

Since I thought we are dealing with subpages mostly I added this line and note the new import too:

from urllib.parse import urljoin

absolute_url = urljoin(url, a['href'])

I do not know if it is necessary or not.

[ashmilhussain]

@agberoz check this

[agberoz]

@YashBaviskar1 It's looking good!

A suggestion: When adding child URLs to the queue, please ensure they start with the base URL to prevent the crawler from navigating to other domains.

[YashBaviskar1]

@agberoz Sure, we can do that using .netloc to include and compare the domain of the urls

from urllib.parse import urlparse

#Base Domain of the FIRST(BASE URL)
base_domain = urlparse(urls[0]).netloc

and then add this condition when appending the child url

if is_scan_child :
    for a in soup.find_all('a', href = True) :
        absolute_url = urljoin(url, a['href'])
        if absolute_url not in self.visited_url and urlparse(absolute_url).netloc == base_domain :
            url_queue.append(absolute_url)

if the absolute_url is of different domain, it will not be added in the url_queue and hence the crawler will stay on the base domain.

[YashBaviskar1]

Ok i will soon send a PR if everything works! Thank you