[ ] 📖 Documentation (updates to the documentation or readme)
[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[x] 👌 Enhancement (improving an existing functionality)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠️ Breaking change (fix or feature that would cause existing functionality to change)
📚 Description
Currently, we have very basic path matching against the robots.txt to add the meta tag and header automatically, this is leading to inconsistent states.
In this PR we I'm to:
Match robots.txt using the same logic as Google
Apply rules for specific user agent
See in devtools which robots.txt rule is blocking current page from indexing
🔗 Linked issue
https://github.com/nuxt-modules/robots/issues/130
❓ Type of change
📚 Description
Currently, we have very basic path matching against the robots.txt to add the meta tag and header automatically, this is leading to inconsistent states.
In this PR we I'm to: