Support repository urls ending with .git

cmeessen commented 4 months ago

Fixes #1244

Changes proposed in this pull request:

scrapers recognise repository urls that end with .git and scrape them
adds a sentence to the UI stating which URLs are supported
update user documentation
adds tests for the scrapers (disabled)
rename GithubScaperTestIT to GithubScraperTestIT
fix bug where scraping_disabled_reason text field shows number of characters of repository_url

How to test:

docker compose down --volumes && docker compose build --parallel && docker compose up

create four software entries with the following repository URLs (or those of you choice) and set them visible:
- https://github.com/research-software-directory/RSD-as-a-service.git
- https://github.com/research-software-directory/RSD-as-a-service
- https://codebase.helmholtz.cloud/research-software-directory/RSD-as-a-service.git
- https://codebase.helmholtz.cloud/research-software-directory/RSD-as-a-service

wait for the scrapers to run or force them with:

docker compose run scrapers /opt/java/openjdk/bin/java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.git.MainCommits

PR Checklist:

sonarcloud[bot] commented 4 months ago

Quality Gate passed for 'scrapers'

sonarcloud[bot] commented 4 months ago