[BUG] Validation of URLs Fails for Certain Status Codes

[Srikar04]

Bug Description: The URL validation logic in our project currently only considers a URL to be valid if it returns a 200 OK status code. However, some valid URLs, such as LinkedIn's homepage, return other status codes like 999, which causes our validation to incorrectly mark them as invalid. This issue affects the accuracy and reliability of the URL validation feature.

Steps to Reproduce:

Use the current URL validation logic to validate https://www.linkedin.com/in/srikarmellachervu/ Observe that the validation fails due to the status code 999 returned by LinkedIn. Expected Behavior: The URL validation should account for valid status codes other than 200 OK, including but not limited to 301 Moved Permanently, 302 Found, and specific custom status codes like LinkedIn's 999.

Actual Behavior: The URL validation fails for valid URLs that return status codes other than 200 OK which gives an output "Invalid URL"

Possible Solution: Update the validation logic to handle a broader range of valid HTTP status codes, such as 301 Moved Permanently, 302 Found, etc. Implement handling for specific known custom status codes, like LinkedIn's 999. Consider implementing a retry mechanism for temporary issues or status codes indicating the need for a second attempt.

Please issue this to come under gssoc 24

• OS: Linux
• Browser Brave

[github-actions[bot]]

Congratulations, @Srikar04! 🎉 Thank you for creating your issue. Your contribution is greatly appreciated and we look forward to working with you to resolve the issue. Keep up the great work!

We will promptly review your changes and offer feedback. Keep up the excellent work! Kindly remember to check our contributing guidelines

[abhijeetnishal]

Thanks for pointing it out this issue. Hey @RoshanAnsy, look into this issue.

[Srikar04]

I have a working solution to this please assign this to me so I can create a pull request

[abhijeetnishal]

Hey @Srikar04, could you explain your approach. If this is the case then we can't add every response code for every website in our code.

[Srikar04]

@abhijeetnishal I think we can fix by expanding the range of acceptable status codes such as 2xx and 3xx and also , implementing retries, and using configurable lists for non-standard codes. Codes such as 999 usually used for rate limiting and blocking bot requests We can solve this issue by making retries to the url so that atleast one of them hits.

[abhijeetnishal]

This is fine but are you sure it will work for all have you tested it because in later time what if someone else reports that some links are not working. Although I am assigning this issue to you.

Hey @RoshanAnsy, i think we missed these cases.

[abhijeetnishal]

Closing this issue as done.