Mr0grog
commented
5 years ago I think we should define some scope & goals now… I'm on a personal mission to have all github issues I create & work on connect back to a repo's objective
❤️
We've stated from the get go that this project will support multiple organizations, which are in charge of defining their own missions. To me this means don't have to define a mission (that should be left to the various orgs), but instead we just need an objective for this repo.
I think it might be helpful to lay out what these orgs want to get out of this. That’s definitely more narrow than a high level objective, but I think we can (should?) drive upwards from these needs to a broader objective (then we can use the pyramid to drive down from that objective). At the very least, it’d be good to just have them articulated somewhere :)
I can speak for web monitoring:
- Regular (in time) and consistent (in data and quality) captures of large list of URLs (30k and growing).
- Proactive notification of captures that differ from the previous capture of the same URL (we don’t have this for any other data source, so it’s not a hard requirement)
- The outputted data structure should make it easy to understand and traverse redirects (if we can query for historic data, then we should be able to query on that information). This impacts the above point about notifying differences.
- “Regular” means >= 2/week (Versionista currently gets us every 2 days but occasionally fails; IA currently gets us every day.)
- Regular (in time) and complete (does a reasonably good job of getting all URLs) maps of all the pages in a site and the links between them.
- “Pages” means things you might browse directly to. It includes, for example, PDFs, but not JS files.
- “Regular” means >= 1/month.
- General a word corpus here would be great, but we can also build that as higher level tool.
- Data generated by this service should be public (or handed to an entity that can make it public).
- It’s a live, running service (not just code), so the actual scraping is de-duplicated across all our projects that have scraping needs.
In Slack @edsu and I had talked about the value of having a service like this for docnow and diffengine. If I understood his needs well:
- Regular (in time) captures of URLs mentioned in a curated set of RSS feeds
- Not sure on specific definitions of “regular” for this use-case.
- Proactive notification of captures that differ from the previous capture of the same URL
Draft Objective:
Define & maintain common, modular crawling infrastructure that multiple projects can depend on and share the results of.
I like this. 👍
Is there something worth adding in here about conceptualizing this as running infrastructure, not just code? (Is that a critical part of this project or is it something higher level, e.g. a goal specific to/owned by EDGI?)
Draft Near-Term Milestone:
Use walk to generate a sitemap…
- The person doing the deployment & the main code contributors should be different people, and only work through publicly-available documentation
I am worried this is not super feasible with the people-power we have, but it does sound like a nice way to validate how workable the software is. How can we make sure this happens?
b5
This is now the third direct iteration of this project. First it was sentry, then pieces of it were extracted into a sitemap CLI tool, and now it's here. There have also been lots of related projects along the way.
Pitfalls of past iterations of this project:
I think this is a sign that we've needed this infrastructure for some time, but have struggled to clearly define a manageable scope with clear milestones that keeps the project moving. Part of it is this kind of infrastructure is tough to build, and even harder to get right. The other part is until now many of our projects have been in a formative stage, and we couldn't get to this work until we had a clearer understanding of what we wanted.
I think we should define some scope & goals now. As a group we tend to get pretty bogged down in process, so I think we should do two things to make our lives easier:
By only requiring those two things to keep moving, but allowing for much more info to enter the conversation as food-for-thought. I think we can get to rough consensus and working code faster by just focusing on those two things.
Pyramid of Clarity
@joehand put me on to the pyramid of clarity a few weeks ago, and I must say it's proven a very effective tool for prioritizing work by connecting it to a mission. I'm on a personal mission to have all github issues I create & work on connect back to a repo's objective, and use how closely an issue matches the objective as the primary method of prioritizing work.
I'm not suggesting we need to follow this "pyramid of clarity" process, instead I think the pyramid shows the work we don't have to do, because it should be done elsewhere. We've stated from the get go that this project will support multiple organizations, which are in charge of defining their own missions. To me this means don't have to define a mission (that should be left to the various orgs), but instead we just need an objective for this repo.
I'd love to toss out a draft objective & milestone to work with:
Draft Objective:
Define & maintain common, modular crawling infrastructure that multiple projects can depend on and share the results of
Draft Near-Term Milestone:
Use walk to generate a sitemap:
These are meant to spur discussion 😄
Action Items:
roadmap.mdfile that maintains the project objective, stated goals & non goals, and info on how to track our progress