Allow 127.0.0.1 as valid URL for scraping - Githubissues

Mintplex-Labs / anything-llm

The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more.

https://anythingllm.com

MIT License

27.48k stars 2.77k forks source link

Allow 127.0.0.1 as valid URL for scraping #2560

Closed shatfield4 closed 3 weeks ago

shatfield4 commented 3 weeks ago

Pull Request Type

[ ] ✨ feat
[x] 🐛 fix
[ ] ♻️ refactor
[ ] 💄 style
[ ] 🔨 chore
[ ] 📝 docs

Relevant Issues

resolves #2459

What is in this change?

Modify validURL to allow the localhost IP 127.0.0.1
Now allows this to be valid for bulk link scraper and single link scraper
Send 400 response on failed validURL check to fix frontend hanging on invalid URL

Additional Information

Developer Validations

[x] I ran yarn lint from the root of the repo & committed changes
[x] Relevant documentation has been updated
[x] I have tested my code functionality
[x] Docker build succeeds locally