Developers Italia provides a catalog of Free and Open Source software aimed to Public Administrations.
publiccode-crawler
retrieves the publiccode.yml
files from the
repositories of publishers found in the Developers Italia API.
publiccode-crawler
can either run manually on the target machine or it can be deployed
from a Docker container.
Rename config.toml.example
to config.toml
and set the variables
NOTE: The application also supports environment variables in substitution to config.toml file. Remember: "environment variables get higher priority than the ones in configuration file"
Build the binary with go build
You can build the Docker image using
docker build .
or use the image published to DockerHub:
docker run -it italia/publiccode-crawler
publiccode-crawler crawl
Gets the list of publishers from https://api.developers.italia.it/v1/publishers
and starts to crawl their repositories.
publiccode-crawler crawl publishers*.yml
Gets the list of publishers in publishers*.yml
and starts to crawl
their repositories.
publiccode-crawler crawl-software <software> <publisher>
Crawl just the software specified as parameter. It takes the software URL and its publisher id as parameters.
Ex. publiccode-crawler crawl-software https://api.developers.italia.it/v1/software/a2ea59b0-87cd-4419-b93f-00bed8a7b859 edb66b3d-3e36-4b69-aba9-b7c4661b3fdd
crawler download-publishers
downloads organizations and repositories from
the onboarding portal repository
and saves them to a publishers YAML file.Developers Italia is a project by AgID and the Italian Digital Team, which developed the crawler and maintains this repository.