The crawler - Githubissues

ministryofjustice / find-moj-data

Find MOJ data service • This repository is defined and managed in Terraform

MIT License

4 stars 0 forks source link

The crawler #105

Open seanprivett opened 3 months ago

seanprivett commented 3 months ago

Ideas:

Ask MP and CP if they're ok with us doing a scan of things on their platform. What does that look like? How would it work? Check in with DPR to see if they have any data products now in their repo Develop a script that we ask services to run themselves, which gives us the info we need Trawl Github

Questions:

If we want to ensure up-to-date metadata, do we need to consider implementing components of the 'embedded agent' solution (where an agent 'within' the system exports data to our aggregator service)?
- Is there a less invasive way of achieving this?

seanprivett commented 2 months ago

Questionnaire: what datasets do you have; where do they live; how might we get access to them; what data do they have that would be useful to us as metadata. And should this process be owned and initiated by DIP?