search

ossf / Project-Security-Metrics

Collect, curate, and communicate relevant security metrics for open source projects.
https://openssf.org
Apache License 2.0
63 stars 24 forks source link

Bug: Metrics aren't being refreshed appropriately #51

Open scovetta opened 3 years ago

scovetta commented 3 years ago

Seems like the cron job isn't kicking off each day. Need to investigate.

iamamoose commented 3 years ago

I've noticed that many of the entries on metrics.openssf.org are many months out of date. In the meantime could you do a manual run (I've made a change to Apache default security files which should change every ASF project, for example)?

scovetta commented 3 years ago

@iamamoose This should be fixed now, would you mind taking a look for Apache to see if the updated data is available? IIRC, the latest scorecard snapshot was from July 27, so if you made the changes after that, it might not be in until that data refreshes first.

iamamoose commented 3 years ago

Am I looking in the right place?

I made the change https://github.com/apache/.github/commit/c5e16821126392a9613ee5def9d1cce56a1f64bf on Jul 19th which should cause all ASF scorecards to get a pass on security.md (by inheritance).

https://metrics.openssf.org/search?q=apache still shows all the entries as being updated 4 months ago, although the scorecard scores have changed upwards since last week (although the security.md tests are not showing passed)

scovetta commented 3 years ago

Ahh, yes, you're looking in the right place, but I'm not showing the right data. I'll get a fix out shortly, you'll be able to view within the Grafana UI on the right, "Recorded" boxes - for scorecard and criticality, those will show a date. image

Long, uninteresting story: A "project" has a last updated date, which is updated whenever that project is updated. A "metric" is the actual data, has a foreign key to the project, and also has a last updated date. When I reload data, I look for a project, then update the metrics. I don't update the project itself (because it only has a package url). So on that screen, I really want to show "last time metrics were refreshed" but it's instead showing "last time the project object was changed", which is effectively "when the project was first seen", hence 4 months ago when we last did a full wipe/refresh.

scovetta commented 3 years ago

Ok, I updated the recorded box - should show the latest date from when the scorecard metrics are refreshed.

david-a-wheeler commented 3 years ago

@scovetta - thanks so much for tracking that down!

iamamoose commented 3 years ago

Also the issue I was facing (scorecards not taking account of organisation wide policy files) was a bug so the Apache scorecards will all update appropriately once https://github.com/ossf/scorecard/pull/837 is merged and everything reruns