[Feature] Automation to scan for broken links

[siamonas]

Feature Description

Implementing a robust automation for scanning each topic's resources can help prevent errors in managing and maintaining these topics. By identifying and flagging broken links, the automation can ensure that all references are accurate and up-to-date.

What to consider:

• Redirects: The potential for links to redirect to other URLs, possibly causing incorrect flags for inaccessible files.
• HTTP Status Codes: Various codes (other than 200) that indicate different server responses and must be handled correctly by the automation.
• Rate Limiting: Avoiding hitting GitHub's rate limits, especially for unauthenticated users.
• Resource Availability: Addressing server-side issues that might prevent the automation from accessing links even when they're correct and accessible.
• File Format/Type Handling: Verifying accessibility of files directly pointed to by links, considering potential corruption or other file-specific issues.
• Caching Issues: Managing cached results efficiently to avoid missing new issues or changes in a repository's structure.
• Numeber of Files: Currently, the repository contains ~3600 references to external links, which may impact the automation's performance and require additional considerations for scalability.

[jdegand]

markdown-link-check could be a good candidate for this.

Updated: Actually, Markdown Link Check has been updated to linkspector, which is supposed to have less false positive results.

[pyoneerC]

Actions can solve this easily.

Or running

wget --spider --recursive --no-directories --no-verbose --output-file linktest.log https://roadmap.sh || true

[BeardedOwl1357]

Actions can solve this easily.

Or running

wget --spider --recursive --no-directories --no-verbose --output-file linktest.log https://roadmap.sh || true

When you run this command, this checks only the links which are of the domain roadmap.sh, not the youtube links that are associated with each learning roadmap.