0ptim / JellyChat

🪼 AI chatbot for the DeFiChain ecosystem.
https://defichainwiki.com/jellychat
MIT License
11 stars 2 forks source link

Use custom sitemap parser #53

Closed 0ptim closed 1 year ago

0ptim commented 1 year ago

To no longer depend on from usp.tree import sitemap_tree_for_homepage.

_sitemapparser.py

import requests
from lxml import etree

def get_urls(url):
    loc_list = []

    # Fetch the XML content
    response = requests.get(url)

    if response.status_code == 200:
        xml_content = response.content

        # Parse the XML content
        root = etree.fromstring(xml_content)

        # Extract the <loc> values
        loc_tags = root.findall(
            ".//{http://www.sitemaps.org/schemas/sitemap/0.9}loc")
        for tag in loc_tags:
            loc_list.append(tag.text)

    return loc_list
0ptim commented 1 year ago

https://github.com/0ptim/JellyChat/commit/21339cb42deb53e723331774e739eb10d542e39a