python linkchecker component

[pvgenuchten]

The python linkchecker is a basic checker which runs through a series of webpages identifies links and tries to resolve them, it reports in a number of potential output formats (xml, sitemap, sql)

The checker can run as a ci-cd script in a container

docker run --rm -it -u $(id -u):$(id -g) ghcr.io/linkchecker/linkchecker:latest --verbose https://www.example.com

results can be written to a location to be picked up by a process, for example a set of sql statements to be run against a DB

[pvgenuchten]

we already identified some challenges with this tool (and other tools):

• linkchecker can not a accept:html header, pycsw currently requires this header to return html, add ?f=html as a workaround
• pycsw maxpagesize (currently 20) is too low, better to use 100
• use --verbose and --check-extern to add full checking and logging

example

linkchecker https://soilwise-he.containers.wurnet.nl/cat/collections/metadata:main/items?f=html --verbose --check-extern

[pvgenuchten]

since the soilwise catalogue is not online yet, suggestion is to use a catalogue of ejpsoil

https://catalogue.ejpsoil.eu/collections/metadata:main/items?f=html

not sure if linkchecker properly fetches each result of the paginated search result, else maybe an option to check links per page

• https://catalogue.ejpsoil.eu/collections/metadata:main/items?f=html&0
• https://catalogue.ejpsoil.eu/collections/metadata:main/items?offset=50
• https://catalogue.ejpsoil.eu/collections/metadata:main/items?offset=100
• https://catalogue.ejpsoil.eu/collections/metadata:main/items?offset=150

etc...

Goal is to:

• [x] set up a ci-cd (in gitlab) which runs at weekly interval.
• [x] Results of the linkchecker can be saved as sql-inserts or xml
• [x] A next step in ci-cd picks up the content and pushes it to a database