fix crawler issues - Githubissues

survos / survos

Example of a mono repo

MIT License

0 stars 1 forks source link

fix crawler issues #25

Open tacman opened 9 months ago

tacman commented 9 months ago

https://mus.wip/en/crawlerdata

Crawler should take an optional --locale argument that limits the pages being searched to only that locale. Otherwise, we're overwhelmed with duplicate links.

_profiler/ links should be excluded for the json -- they should never even get to the link list.

There are still pages missing, like https://www.mus.wip/en/mus/countries, which is in the footer. Maybe because of so many langugage and profile links.

tacman commented 7 months ago

Specifically, the DX for crawler should be:

Require the bundle
Configure survos_crawler.yaml (users: routes_to_ignore, paths_to_ignore)
Run the crawler against the dev environment, fix any errors. (Dynamically turn off the web profiler toolbar).
Configure the test environment and run the crawler with --env=test
run PHP unit with the generated routes from the --env=test run.
run code coverage (php-pcov seems to work in 8.3. This is already done in a few applications).

tacman commented 6 months ago

mailto: links should be ignored.
if a link has been seen, we don't need to write it again unless we have some new data, like # of times seen. Some 403's are wrong

Switch User should be excluded, and may be part of the problem above.