Enable crawling for URLs having hash fragments for ai-scan cli

[v-viyada]

Details

Added new optional parameter keepUrlFragment to enable crawling for URLs having hash fragments for ai-scan cli and crawler packages. The default value of it is false so that existing scan work as is.

Motivation

Feature 2114928: ADO Extension should be able to crawl hash-routing SPA apps

Context

Pull request checklist

• [ ] Addresses an existing issue: Fixes #0000
• [x] Added relevant unit test for your changes. (yarn test)
• [x] Verified code coverage for the changes made. Check coverage report at: <rootDir>/test-results/unit/coverage
• [x] Ran precheckin (yarn precheckin)
• [n/a] Validated in an Azure resource group

[DaveTryon]

Running yarn test fails in 3 test files:

• packages/service-library/src/data-providers/privacy-scan-combined-report-provider.spec.ts
• packages/service-library/src/data-providers/page-scan-run-report-provider.spec.ts
• packages/service-library/src/data-providers/combined-scan-results-provider.spec.ts

However, since the main branch fails in the same places, it's not caused by this PR and is non-blocking