Django check seo search in wrong content

corentinbettiol commented 4 years ago

Currently, django-check-seo is searching the content in the <div class="container"></div> tag.

It could be a problem if the main content of the pages of the crawled website is in another tag (like <main> or <div class="cms_main"> ...).

We should provide a setting where we could select a tag/class/list of tags/classes to search content in.

corentinbettiol commented 4 years ago

Possible solution:

Add a new parameter in settings.py, e.g:

DJANGO_CHECK_SEO_SEARCH_IN = "#wrap > .container"

This css selector will only search the elements with class container directly inside elements with id wrap:

So the #wrap > header > .container and footer > .container elements will not be selected.

The hard part is to have a consistent layout on each page to always select the main content.

corentinbettiol commented 4 years ago

Using a list of selectors and the fabulous extract() function of BeautifulSoup4, we can extract the selected html nodes from our main soup, and only keep what we want in the soup (the main content).

This method will certainly be chosen, because headers, navigation and footers tend to have the same class names & id for each page of the site (content class names may vary if we use different page templates in our website).

Ex:

Soup:

<!doctype html>
<html>
<head>
<title>Home</title>
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="description" content="None">
<meta name="keywords" content="kw1,  kw2,  kw3">
</head>
<body>
<div class="container">
    <ul class="nav">
        <li class="child selected">
            <a href="/fr/">Home</a>
            <ul>
                <li class="child descendant">
                    <a href="/fr/blog/">Blog</a>
                </li>
                <li class="child descendant">
                    <a href="/fr/page-3/">Page 3</a>
                </li>
            </ul>
        </li>
        <li class="child sibling">
            <a href="/fr/page-2/">Page 2</a>
        </li>
    </ul>
    <div class="darksky-small">
        <div class="darksky-icon darksky-rain"></div>
        <div class="darksky-temperature">9 °C</div>
        <div class="darksky-humidity">91 %</div>
        <div class="darksky-wind">3,6 km/h</div>
    </div>
    <section class="cover-section">
        <section class="container">Container content -1</section>
    </section>
    <section class="container">Container content 0</section>
    <div id="wrap">
        <section class="container">Container content 1</section>
        <section class="container">Container content 2</section>
        <div class="footer">
            <div class="containter">Footer content 1</div>
        </div>
    </div>
</div>
</body>
</html>

excluded_selectors = ".nav, .darksky-small, .cover-section, .footer"

for element in soup.select(excluded_selectors):
    element.extract()
    # [...]

print(soup)

<!doctype html>
<html>
<head>
    <title>Home</title>
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <meta name="description" content="None">
    <meta name="keywords" content="kw1,  kw2,  kw3">
</head>
<body>
    <div class="container">
        <section class="container">Container content 0</section>
        <div id="wrap">
            <section class="container">Container content 1</section>
            <section class="container">Container content 2</section>
        </div>
    </div>
</body>
</html>

Edit: Here's a screenshot of the previous code:

green background: what we want to keep
red background: what we don't want to keep
purple/blue/orange/ping tag hover: nodes to remove

kapt-labs / django-check-seo

Django check seo search in wrong content #35