search

ossf / scorecard

OpenSSF Scorecard - Security health metrics for Open Source
https://scorecard.dev
Apache License 2.0
4.3k stars 470 forks source link

DISCUSSION: v4 milestone #1121

Closed laurentsimon closed 2 years ago

laurentsimon commented 2 years ago

To start thinking of our next step towards v4 release, let's write some ideas in this issue. We're thinking of v4 release for EOY 2021.

We can talk about them during next scorecard meeting, create issues and assign them to contributors, and then have them as milestones. Here a list to start with:

Please add what you think is worth discussing about. This will help for selection and prioritization.

Thanks everyone!

asraa commented 2 years ago

Random thought that might be nice to do for another release (not sure when v4 is on the timeline): scorecard as in-toto attestation? would mean defining a scorecard predicate and having an output format for that.

Higher levels could sign the envelope (e.g. a github-runner could run scorecard and then sign the in-toto proving that it ran scorecard.

laurentsimon commented 2 years ago

Random thought that might be nice to do for another release (not sure when v4 is on the timeline): scorecard as in-toto attestation? would mean defining a scorecard predicate and having an output format for that.

Higher levels could sign the envelope (e.g. a github-runner could run scorecard and then sign the in-toto proving that it ran scorecard.

Thanks for the suggestion, @asraa! We were thinking v4 release EOY.

naveensrinivasan commented 2 years ago

@laurentsimon Thanks, Probably add this as a milestone once we have consensus?

azeemshaikh38 commented 2 years ago

To expand on Laurent's comment, we are looking for contributors interested in owning some these KRs end-to-end.

  • Scorecard E2E tests

(i) enable e2e tests on ossf-test repos https://github.com/ossf/scorecard/issues/861 (ii) generalize e2e tests to run on RepoClient interface - https://github.com/ossf/scorecard/pull/1113#issuecomment-935432773

Also, the below KRs either require community inputs or a general helping hand:

Finally, items to help reduce technical debt. Not part of Milestone V4, more like ongoing KRs which help improve code quality:

@naveensrinivasan @chrismcgehee @david-a-wheeler FYI. Let us know if you would like to see anything else added here.

laurentsimon commented 2 years ago

@naveensrinivasan also proposed doing a scorecard demo/blog post on using scorecard to vet dependencies automatically.

naveensrinivasan commented 2 years ago

@naveensrinivasan also proposed doing a scorecard demo/blog post on using scorecard to vet dependencies automatically.

More understand the state of dependencies with scorecard data

laurentsimon commented 2 years ago

automatic documentation generation https://github.com/ossf/scorecard/issues/898

laurentsimon commented 2 years ago

GitHub action issue https://github.com/ossf/scorecard/issues/193. v4 milestone added.

laurentsimon commented 2 years ago

This issue https://github.com/ossf/scorecard/issues/426 is an important one, especially the pull_request_target trigger.

laurentsimon commented 2 years ago

Adding lines/filenames to our results https://github.com/ossf/scorecard/issues/725 is an important issue we should tackle for v4 since it improves the UX experience in the GitHub scanning dashboard

laurentsimon commented 2 years ago

@oliverchang will tackle https://github.com/ossf/scorecard/issues/1148. Thanks Oliver!

laurentsimon commented 2 years ago

FYI, Asra @asraa will tackle part of https://github.com/ossf/scorecard/issues/426. Thanks you Asra!

laurentsimon commented 2 years ago

This is also useful https://github.com/ossf/scorecard/issues/435#issuecomment-952446674

laurentsimon commented 2 years ago

Would love to have dangerous workflows in v4 https://github.com/ossf/scorecard/pull/1168, if possible Long-term, I think we'll merge Token-Permissions into it, as well as the GH workflow pinning that currently lives under pinned dependencies.

azeemshaikh38 commented 2 years ago

We seem to have a lot more v4 issues than initially discussed. Do we have the time commitment to complete all these extra items? Please note that we are aiming for a mid-Jan timeframe for a v4 release. And with a winter break, it does not give us a lot of time. @laurentsimon @naveensrinivasan

Extra issues I noted: #1174, #1196, #1038, #1260, #1270, #1275, #1238.

laurentsimon commented 2 years ago

I've removed the first one. The others are best effort. Many are simple enough that it's doable, and they improve the checks: I think it's god if we can fix those small issues before releasing. The ignore list for binary artifacts would be great, so I added in case I have time.

laurentsimon commented 2 years ago

Aiming for a release mid-January 2022. What's left:

  1. verify GH action for market place (WIP)
  2. documentation of the action
  3. submit starter workflows (WIP)
  4. Test end to end flow of installation
  5. Install action on a few repos OSSF owns and others
  6. Release notes
  7. Blog post prep

We have a v1 milestone on the action repo https://github.com/ossf/scorecard-action/issues?q=is%3Aopen+is%3Aissue+milestone%3Av1

georgettica commented 2 years ago

hey! I found this v4 when testing locally

docker run -v ${PWD}:/app -e SCORECARD_V4=y -e GITHUB_AUTH_TOKEN=token gcr.io/openssf/scorecard:stable --show-details --local=/app

makes it much faster to make my codebase comply with scorecard

laurentsimon commented 2 years ago

note that when running on a local repo --local=, not all the checks are run. Checks that are run are those that do not use GitHub APIs. Supported checks are indicated thru a repos: local in this file https://github.com/ossf/scorecard/blob/main/docs/checks/internal/checks.yaml#L47

FYI, this https://github.com/ossf/scorecard/pull/1405/files will remove the need for SCORECARD_V4 once merged.

laurentsimon commented 2 years ago

Closing since v4 is out.