Remove URL fragment from crawler result

[lamaks]

Details

Remove URL fragment from crawler result to to avoid crawling the same resource more than once.

Pull request checklist

• [ ] Addresses an existing issue: Fixes #0000
• [x] Added relevant unit test for your changes. (yarn test)
• [ ] Verified code coverage for the changes made. Check coverage report at: <rootDir>/test-results/unit/coverage
• [ ] Ran precheckin (yarn precheckin)
• [x] Validated in an Azure resource group