search

destrex271 / pgweb-testing-harness

This is the official project repository for the PostgreSQL Website Testing Harness.
Other
5 stars 11 forks source link

[Feature]: Imrpove crawl capabilities for src/functional_tests/tests_homepage_links. py #18

Closed destrex271 closed 7 months ago

destrex271 commented 11 months ago

Currently the testing suite deploys a recursive crawler to check the URLs on the PostgreSQL website. We need a better and more customizable crawler with the ability to set depths. Here depths can be in the form of number of subpages scanned or number of subdomains scanned. Suggestion: Write a separate module for this instead of hardcoding it in the test file and refrain from using additional libraries.

Faakhir30 commented 7 months ago

Hi @destrex271 I would like to contribute to this issue. I'll be implementing separate module for the crawler with customized max depths and that crawler would be used in this test_homepage_links.py something like the following:

image

If you have any further suggestions, please share.

destrex271 commented 7 months ago

Hi @destrex271 I would like to contribute to this issue. I'll be implementing separate module for the crawler with customized max depths and that crawler would be used in this test_homepage_links.py something like the following:

image

If you have any further suggestions, please share.

Hi! Thanks for taking the initiative. Can you open this as a pull request and push the CustomCrawler code there? Will be a lot easier to review.

destrex271 commented 7 months ago

PR #31 Merged with main.