Evaluate GrimoireLab for open source community metrics

[jwflory]

Summary

Evaluate whether GrimoireLab is a useful metrics tool for LibreCorps to set up for managing current / future contracts

Background

Grimoire is a visual dashboard presenting open source community/people metrics. It integrates with existing open source tools and platforms to give an approximation of changes in an open source project. It includes monitoring areas like source code, issue/task management, mailing lists/forums, documentation, continuous integration (CI), and more.

This could be a useful service for LibreCorps to provide. If we set up Grimoire, we could create visual dashboards for each team and/or cohort. The dashboards would include graphs and charts for pieces of what we already evaluate on with our rubrics. We could provide links to the dashboards both for individual teams/cohorts and also the UNICEF fund managers to periodically check-in and quickly see the state of a team's open source engagement.

It also gives us a more helpful and intuitive way to monitor progress by an individual team in small, gradual steps versus a team that is not engaging with our resources at all.

Details

@Nolski and I didn't yet come to an agreement on what this looks like or how it should be presented, so we decided to start with an experiment and see where it goes. Steps to bring a proof-of-concept experiment might look like the following:

1. Spin up new DigitalOcean droplet under FOSS@MAGIC account
2. Install Grimoire onto droplet
3. Choose one team (preferably one doing well) to build a mock dashboard for, try out different data collection plugins to see what is useful or not
4. Tweak internally based on feedback
5. If we like it, present to UNICEF and get their feedback

Outcome

• Us:
  • Easier to track small gains / community changes in a team's project
  • Helps us work across multiple repos / platforms (i.e. for example, easy to combine two GitHub repos and a GitLab repo)
  • Monitor cohort progress in a convenient-to-use web interface (i.e. easier for us to on-board new students / people to help with LibreCorps program)
  • Helps make LibreCorps work more indispensable to evaluating success of an open source project
• Clients:
  • Move towards producing meaningful quantifiable metrics (these will get produced with or without us, do we want to influence these for the better?)
  • At-a-glance picture of how a project, team, or cohort is doing
  • Potential to reuse and repeat across future clients

---

cc: @itprofjacobs

[jwflory]

Next steps for this task is to set up a new DigitalOcean droplet in the FOSS@MAGIC account. @Nolski and I briefly chatted about trying to build a dashboard for the RIT FOSS community as a proof-of-concept (check out some of the available data sources for ideas on what we can represent).

Once we have a hosted instance up, we should start looking at the visualization tools at our disposal, do a cross-sectional review of our resources, and figure out ways we could build reproducible dashboards for visualizing the different key components we measure in our existing resources.

[Nolski]

One thing that has me a bit worried about whether we can host this on a free/cheap service is the reasonably high hardware requirements to run Grimore Lab.

Hardware: 2 CPUs, 8GB memory RAM and enough virtual memory for Elasticsearch

[itprofjacobs]

I can see about getting it on the MAGIC servers. Send me the specs

On Nov 7, 2019, at 3:42 AM, Mike Nolan notifications@github.com wrote:

 One thing that has me a bit worried about whether we can host this on a free/cheap service is the reasonably high hardware requirements to run Grimore Lab.

Hardware: 2 CPUs, 8GB memory RAM and enough virtual memory for Elasticsearch

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[Nolski]

@itprofjacobs If we could have a dedicated VM with 2 CPUs and 16GB of RAM that would be amazing :) Just FYI, that's quite a bit of pricy hardware. To get that on Digital Ocean, it would be around $90/month...

[itprofjacobs]

Have made a request with MAGIC director

On Nov 8, 2019, at 5:15 AM, Mike Nolan notifications@github.com wrote:

@itprofjacobs https://github.com/itprofjacobs If we could have a dedicated VM with 2 CPUs and 16GB of RAM that would be amazing :) Just FYI, that's quite a bit of pricy hardware. To get that on Digital Ocean, it would be around $90/month...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/librecorps/tasks/issues/22?email_source=notifications&email_token=AAII2PPCEK6RUWFD657H7GTQSU33RA5CNFSM4I74XR3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDPXWSY#issuecomment-551517003, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAII2PN44WMQWXHMHTUL5GLQSU33RANCNFSM4I74XR3A.

[jwflory]

@Nolski Do you have a source on those numbers? I was curious. I ran their full-stack container image and benchmarked it at about ~4GB of RAM usage with a fresh install.

References:

• Grimoire docs: Deploying from DockerHub
• Docker Hub: grimoirelab/full

[Nolski]

@jwflory I pulled those specs from the recommendations in their README. I suspect once you begin hooking it up and pulling data into it, it will begin using significantly more resources

[jwflory]

Discussed in a planning meeting with @tjzabel on 2019-11-14.

---

Next action steps

• Research-planning meeting: Wed, 20 Nov.

Notes

Some notes I typed up previously were lost, here was our whiteboard:

[jwflory]

_Discussed in a planning meeting with @Tjzabel and @Nolski on 2019-11-20._

---

Summary

Backtracked to requirements planning. We spent a lot of time back and forth on architecture and infrastructure. But before we spend time on technology, we haven't figured out how we will measure our success in whatever we do. So we spent some time to identify some goals, the success criteria for those goals, and what our next steps are to push things forward.

Grimoire components / architecture

Noting this here as a reference:

LibreCorps and Grimoire: What are the value-adds?

1. Tracking of partner projects through a visual dashboard.

• Grimoire allows us to do this in a pre-packaged way that is easy to manage.
• It is supported without a lot of manual work.

2. Evaluating change over time.

• Grimoire allows us to look at non-technical components of an open source community and monitor changes over time.
• This gives immediate feedback if our targeted efforts in supporting teams is being received.

3. Provide insight into the wider Humanitarian Open Source community (either internal or external to an organization).

4. Provide insight into the FOSS@RIT open source community.

What does success criteria look like for each of these value-adds?

1. Tracking of partner projects through a visual dashboard.

• Quick navigation across different organizations, filtered down to cohorts, filtered down to teams. Identify specific success stories.
• Compare specific teams (heros vs. slackers).

2. Evaluating change over time.

• Ability to see a range of different platforms used by a team and where they engage the most in the open.
• Provide specific references to what is working and what isn’t.
• Make better (personalized) predictions for project timelines.

3. Provide insight into the wider Humanitarian Open Source community (either internal or external to an organization).

• Easy understanding to see how different projects within an organization relate to each other.
• Discover new opportunities for collaboration across organizations or team in similar areas.

4. Provide insight into the FOSS@RIT open source community.

• Easy understanding to see intake of new community members (How many new members? Are we growing?)
• Identification of key contributors.
• Distinguish between multiple levels of engagement / participation.
• What do current interactions look like? (e.g. GitHub/GitLab, Slack, IRC, maybe Discourse)
• Monitoring of open source content tagged or affiliated with RIT on existing popular platforms (e.g. GitHub topic tags “RIT” “Rochester…”)
  • Helpful to discover where our FOSS support points on campus to help rep the Flag

Next steps

• @jwflory @tjzabel @Nolski Read up on the overall architecture / components of GrimoireLab over Thanksgiving
• Is it better to use a big image or split it up? This affects how we request resources and plan for the future.
  • Come ready to diagram architecture at next meeting (Wednesday, 4 December @ 10:00)

Other research questions to consider

1. Is the full Grimoire DockerHub image production-ready?
2. How do we manage backups?
3. Explore Ansible automation possibilities with FOSSRIT/infrastructure?
4. How viable is Podman? (rootless containers ftw)

[jwflory]

Included in 2020-03-24 issue triage and cleanup.

---

At CHAOSScon, I discovered Cauldron, a hosted Grimoire dashboard. This seems closely aligned to what we want. I posted a thread on the FOSS@RIT Discourse showing off ten years of FOSS@RIT:

https://fossrit.community/t/10-years-of-foss-rit-3m-lines-of-code-374-contributors-476-prs/190

Since I did that, I haven't had a chance to touch this again. It's on my radar as part of my independent study with @ritjoe though. Keeping this open for now.

[jwflory]

Cauldron is sufficient for most of our needs. I'm not going to have an opportunity to finish this before I officially depart RIT though. Closing as wontfix.