Page crawl depth

[azdanov]

The issue is with crawling depth for blog and comment pages.

For example on a blog page blog/ there is a link to blog/page/2/ which works fine. But on blog/page/2/ there is a link to blog/page/3/ which is ignored during a crawl.

Same behavior happens with comments, e.g template-comments/#comments.

I know that it's possible to include a url manually, but it's not easy with pagination.

Is this a bug, or I'm missing some configuration?

[leonstafford]

@azdanov, which theme is this using? The plugin used to be more a traditional crawler, but now infers knowledge of URLs from WordPress for the first level and crawls one more level after that.

I'll look into the detection of the post/comment pagination URLs. We want to see them all in the initial crawl list preview on the Crawling tab

[fmt]

@azdanov meanwhile you could use something like:

function paginationHandler($query) {
  if(!is_admin() && $query->is_main_query()) {
    $offset = get_option('posts_per_page');
    if($query->is_paged) {
      $maxpostperpage = 1000;
      $postsperpage = $maxpostperpage - $offset;
      $query->set('posts_per_page', $postsperpage);
      $query->set('offset', $offset);
    } else {
      $query->set('posts_per_page', $offset);
    }
  }
  return;
}
add_action('pre_get_posts', 'paginationHandler');

[azdanov]

@leonstafford It's a custom theme based on Sage. Here is it's repo: azdanov/wordpress-first.

@fmt Thanks!

[leonstafford]

Thanks @azdanov, I'll use that to test.

[leonstafford]

I should have this working in first version tonight, with something like this in the FilesHelper class:

https://github.com/leonstafford/wp2static/issues/235#issuecomment-457904099

[leonstafford]

Remaining to fix in this issue:

• [x] category pagination URLs
• [x] comment pagination URLs

[azdanov]

Remaining to fix in this issue

I've also noticed:

• [x] tags pagination URLs

[leonstafford]

^ luckily, tags came for free with the category code (both taxonomy > term elements)

[leonstafford]

damn, one more:

Posts that split across multiple pages (will make new issue for that)

[leonstafford]

w00t, got it

/category/uncategorized/page/1
/my-lonely-post/comment-page-3
/my-oher-post/comment-page-3
/my-oher-post/comment-page-2
/my-lonely-post/comment-page-2
/my-lonely-post/comment-page-1
/my-oher-post/comment-page-1

[azdanov]

@leonstafford Good job!