Project Health Reporting (take 2)

maoo commented 1 month ago

Follow up of https://github.com/finos/technical-oversight-committee/issues/125

The following health metrics have been built for the CDM project, using the following LFX Insights reports:

CDM (manual) report:

From Velocity (last quarter) - https://insights.lfx.linuxfoundation.org/foundation/finos/velocity?project=isda-cdm&routedFrom=Github&dateFilters=Last%2012%20Months&dateRange=2023-07-26%20to%202024-07-25&compare=PP&granularity=month&hideBots=true

Download Performance Metrics CSV
Download Lead Time CSV
Download Wait Time for 1st Review CSV

From Productivity (last quarter) - https://insights.lfx.linuxfoundation.org/foundation/finos/productivity?project=isda-cdm&routedFrom=Github&dateFilters=Last%20Quarter&dateRange=2024-04-01%20to%202024-06-30&compare=PP&granularity=week&hideBots=true&repository=all

Download Commits per Active Day CSV
Download New Contributors CSV

From Reports > Contributors (last quarter) - https://insights.lfx.linuxfoundation.org/foundation/finos/reports/contributors?project=isda-cdm&routedFrom=Github&dateFilters=Last%20Quarter&dateRange=2024-04-01%20to%202024-06-30&compare=PP&granularity=week&hideBots=true&repository=all

Download Total Contributors CSV
Download Monthly Active Contributors CSV

From Reports > Activities (last quarter) - https://insights.lfx.linuxfoundation.org/foundation/finos/reports/activities?project=isda-cdm&routedFrom=Github&dateFilters=Last%20Quarter&dateRange=2024-04-01%20to%202024-06-30&compare=PP&granularity=week&hideBots=true&repository=all

Download Total Activities CSV
Download Monthly Activities CSV (Warning! This returns an empty CSV)

Still unclear how to use Reports > Retention on https://insights.lfx.linuxfoundation.org/foundation/finos/reports/retention?project=isda-cdm&routedFrom=Github&dateFilters=Last%2012%20Months&dateRange=2023-07-26%20to%202024-07-25&compare=PP&granularity=month&hideBots=true

Attaching spreadsheet with one sheet per CSV export - CDM metrics (May-July 2024).xlsx

As next steps:

@maoo to work with LFX team to understand how to automate the download of all the CSV files for each project and how to iterate across all FINOS hosted projects
@finos/toc to review metrics above and provide feedback
@finos/toc to update the Jupyter Notebook accordingly and build the board report on project health

For all security aspects related with project healthcheck, we're considering to adopt/contribute to the OSPS Baseline, currently led by @eddie-knight

eddie-knight commented 1 month ago

During today's TOC call, @psmulovics commented that this is a great improvement from #125, but has distinct room for improvement.

@psmulovics Can you please elaborate on the gaps / room for improvement?

Both @chris-twiner and @caradelia have accepted the call for feedback, and intend to comment here before the next TOC call.

psmulovics commented 1 month ago

@psmulovics Can you please elaborate on the gaps / room for improvement?

Sure. Due to the data being 'human quantified' instead of 'scientific quantified', we do have entries saying '1.25K' instead of '1250' which makes it hard to display, needs manual intervention, etc. Same way, some of the data types, like dates, get mistakenly converted to other format, like '7/1/2024 12:00:00 AM' becoming '1.71T' in the report. Also, there are missing spots in the report, eg data that is missing. Also, we need to have better understanding for reports on what 'total activities', 'retention', etc actually means.

maoo commented 3 weeks ago

@psmulovics Can you please elaborate on the gaps / room for improvement?

Sure. Due to the data being 'human quantified' instead of 'scientific quantified', we do have entries saying '1.25K' instead of '1250' which makes it hard to display, needs manual intervention, etc. Same way, some of the data types, like dates, get mistakenly converted to other format, like '7/1/2024 12:00:00 AM' becoming '1.71T' in the report. Also, there are missing spots in the report, eg data that is missing. Also, we need to have better understanding for reports on what 'total activities', 'retention', etc actually means.

Thanks for the valuable feedback @psmulovics and thanks @eddie-knight for gathering feedback.

I agree on 'human' VS 'scientific' quantified data, especially on data exports.

Regarding the 'missing spots', it would be great to have more details on which metrics are missing.

Regarding the understanding for 'total activities', 'retention', etc , I'd ask you to use the LFX Insights UI tips, as shown below. Thank you!

mcderk commented 3 weeks ago

Thank you for the feedback. I work on LFX as head of product & design. I can add a backlog item to see if we can make our downloads more human friendly. Also, we can investigate the use case around automating the send of the report as a future feature.

Please see this documentation on reports - https://docs.linuxfoundation.org/lfx/insights/v3-beta-version-current/reports that should clarify to provide better understanding.

finos / technical-oversight-committee

Project Health Reporting (take 2) #135