search

NASA-AMMOS / slim

Software Lifecycle Improvement & Modernization
https://nasa-ammos.github.io/slim/
Apache License 2.0
24 stars 9 forks source link

[Improve Best Practice Guide]: Security scanning with SCRUB, et. al. #25

Open riverma opened 2 years ago

riverma commented 2 years ago

Checked for duplicates

Yes - I've already checked

Best Practice Guide

Continuous Testing

Best Practice Guide Sections

Describe the improvement

We'd like to integrate NASA's SCRUB into a continuous build system to make it easy for projects to automate security scanning, per NASA recommendations.

riverma commented 2 years ago

@jpl-jengelke - as @awdtinio mentioned, please chat with @jeffreypon as he's experimented with SCRUB.

Also - I think the security scanning with SCRUB ticket should really be delivered as a Jenkins plugin for people to quickly / easily adapt to their existing projects. Having a plug-and-play plugin would likely reduce the amount of time developers would have to spend getting SCRUB scanning initialized for their own CI. Take a look at this guide: https://www.jenkins.io/doc/developer/tutorial/create/

Thoughts?

jpl-jengelke commented 2 years ago

There's a Maven plugin that will integrate with the CodeSonar system, but of course Maven is mostly Java-specific. Developing a Jenkins plugin is a big task, so I don't think by any measure that it's doable with the resources at our disposal. However, we can offer a demonstration that is configurable and plug-and-play enough to cover all bases. That is what I am working on completing.

riverma commented 1 year ago

+1'd by @ramesh-maddegoda, @jeffreypon, @pymonger

jordanpadams commented 1 year ago

@riverma FYI, @lylebarner helped us get this integrated into our CodeQL execution using GitHub Actions. here is an example: https://github.com/NASA-PDS/validate/blob/main/.github/workflows/codeql-analysis.yml#L65

riverma commented 1 year ago

Hi @jordanpadams - thanks for pointing us to @lylebarner and the GH action you all have built for validate. This is pretty cool.

@jpl-jengelke - let's definitely work together and discuss with @jordanpadams and @lylebarner on synergizing their existing code with the goals of this ticket as we work on this during Q2.

jeffreypon commented 1 year ago

AMMOS currently setup to use Codesonar and sonarqube. Scrub outputs a .csv report in a specific format that the cybersecurity team can ingest into their VATT tool to perform adjudication.

@lylebarner If new tools are added to scrub (i.e. CodeQL), does the scrub tool output the .csv results in the specific format that can be imported into the cybersecurity's VATT tool without any software changes? Or does scrub have to be modified for each tool to get it in the specific format?

jeffreypon commented 1 year ago

@jpl-jengelke Make sure to grab the latest version of scrub. There was a major update this past week that changed some of the .csv output formats. The cybersecurity team is requesting all scans be performed using the latest version of scrub now. https://github.com/nasa/scrub

jpl-jengelke commented 1 year ago

Make sure to grab the latest version of scrub. Thank you!

riverma commented 1 year ago

There's another ticket we should probably merge into this one: https://github.com/NASA-AMMOS/slim/issues/51

riverma commented 1 year ago

Some ideas discussed with @jpl-jengelke for this ticket:

jpl-jengelke commented 1 year ago

Proposed implementation

  1. CI Server (Jenkins) Implementation (Java example)
    1. Execute inline as a post-build pipeline step
    2. Run against SonarQube
    3. Publish report in Jenkins workspace
    4. Publish artifact in Jenkins workspace
  2. GitHub Action Implementation (Python Example)
    1. Isolate execution in standalone GH Action (separate container)
    2. Run against CodeQL and PyLint
    3. Publish formatted report alongside Action Summary
flowchart LR
    subgraph CI["CI Server (Exa.: Java)"]
        C1[Compile]-->C2[Test]
        C2-->C3[Package]
        C3-->C4["Publish\n(Artifactory)"]
        C2-.-> |"Success, Monthly"| C5[SCRUB]
    end
    subgraph GA["GitHub Action (Exa.: Python)"]
        A1[Compile]-->A2[Test]
        A2-->A3[Package]
        A3-->A4[SCRUB]
        A4-->A5["Release\n(PyPi)"]
    end
riverma commented 1 year ago

Thanks for the write-up diagram @jpl-jengelke! Some questions:

riverma commented 1 year ago

@lylebarner - had a few questions for you:

lylebarner commented 1 year ago
  • Related to the question above, do you have recommendations for how to filter for a smaller sample of SCRUB scanning results such that SCRUB can run tools like SonarQube / CodeQL quickly enough on individual Git commits?

There is an institutional Secure Coding Guideline that can be used for this. Generally this is considered a good starting place for identifying high priority security issues. This guide can be used to tailor down to a smaller set of CWEs. I can provide a link via email.

  • Are there projects you can point us to that have successfully integrated SCRUB with Jenkins, e.g. for Java minimally?

Florin Tudor has set up several other repos with SCRUB-based scanning on Jenkins. He should be able to provide you with more information or at least point you in the right direction.

jpl-jengelke commented 1 year ago
  • What's preventing, if anything, SCRUB to execute at the test stage instead of post build?

Timing. We can't add anything to extend the build times for VICAR products since the timeline is so unpredictable in my experience. Potentially this could eat 30+ minutes, and that would not meet users' level of service expectations. In current planning we wanted this to occur once a month, maybe quarterly, at most, just to avoid long build times.

Now for a POC in SLIM we could do that, but we wouldn't want someone implementing it in a VICAR product right now.

jpl-jengelke commented 1 year ago

Thanks @lylebarner I'll check in with Florin about it.

lylebarner commented 1 year ago

@jpl-jengelke would it be possible to perform SCRUB analysis in parallel with test activities? From a technical point of view, SCRUB only needs the source code itself (and compiled outputs such as .class/.jar files) to perform analysis. Not sure of the system resources that are required to run the test suite though.

jpl-jengelke commented 1 year ago

@jpl-jengelke would it be possible to perform SCRUB analysis in parallel with test activities? From a technical point of view, SCRUB only needs the source code itself (and compiled outputs such as .class/.jar files) to perform analysis. Not sure of the system resources that are required to run the test suite though.

Yes, it is possible, though some accommodations must be made. One point of the test phase in the Maven lifecycle is to provide an exit point if tests fail, before packaging. So the SCRUB operations would be run against .class files but not packaged .jar files, which are actually the final deliverable.

The tasks would be run in containers to provide a cleanroom environment that isolates them from the CI server setup.

Another option is to run in a parallel job but that would likely require a second compilation.

Thanks for the write-up diagram @jpl-jengelke! Some questions:

  • What's preventing, if anything, SCRUB to execute at the test stage instead of post build? I think that would be a great value-add for teams to be more preventative with security scanning as early as possible than to possibly have public builds out there with security issues.

Costs versus value. During a release cycle, code should be turned around fast. If we execute SCRUB at the test stage then potentially tens of minutes are added to what must be a fast-turnaround build-compile-release cycle. The artifact is not generated until the test cycle completes (not sure if I'd add a pass/fail test). There are options but for a process that generates a report intermittently, the accommodation is to run it at the end of packaging and delivery.

I think we can more readily apply this to the testing stage for a Python deliverable that is completely generated and released using GitHub Actions.

(Note: (rough definitions) packaging == convert .class files into JARs; delivery == push compiled binaries (JARs) to an artifact repository; release == pushing the product out to servers or for public distribution.)

  • From @ramesh-maddegoda's question: can we justify why SonarQube and CodeQL / PyLint are the right tools to run via SCRUB? Why not other tools that are institutionally or open-source-environment supported by SCRUB's wrapping framework?

Part of the charge of this project is to make tools that support Lab processes and requirements. Because CodeQL is the successor to Semmle, and the Lab security reports are predicated on Semmle, we do need to support it. Otherwise, these are soft recommendations based on conventions (as understood) that we can re-evaluate with trade studies. For instance, SonarQube is a very Java-centric tool that is highly adopted and more critically implements an interface that combines a number of underlying code analytic tools, such as PMD, Jacoco, etc.

jpl-jengelke commented 1 year ago

Note, preliminarily modified design flow (above) to accommodate comments after our meeting. I'll be sending out a follow up invitation today. Thanks everyone!

riverma commented 1 year ago

Thanks for the feedback and responses @jpl-jengelke!

I think @lylebarner's thoughts on a potentially running scans in parallel to builds is really interesting. Perhaps it's better to run scans nightly / on a schedule as opposed to tying to build triggers? The length of the code scan issue you mentioned is the same problem that comes up often for running system regression tests - which can take hours. Thoughts?

sharimayer commented 1 year ago

Thinking about our Vicar builds in particular and knowing that only the source code is needed to run Scrub, it might make sense to run Scrub before the nightly builds (like 7 p.m.) as the nightly builds kick off at 9 p.m.

riverma commented 1 year ago

Thanks for that @sharimayer. Quick question: if no source code has changed, do the nightly builds still get triggered? If not - what mechanism is used to know nothing has changed?

sharimayer commented 1 year ago

Yes, the nightly builds get triggered even if no source code has changed.

-- Shari C. Mayer IDS CM

From: Rishi Verma @.> Date: Wednesday, March 8, 2023 at 11:55 PM To: NASA-AMMOS/slim @.> Cc: Mayer, Shari C (US 398G) @.>, Mention @.> Subject: [EXTERNAL] Re: [NASA-AMMOS/slim] [Improve Best Practice Guide]: Security scanning with SCRUB, et. al. (Issue #25)

Thanks for that @sharimayerhttps://urldefense.us/v3/__https:/github.com/sharimayer__;!!PvBDto6Hs4WbVuu7!N-3XbT8BcKx28EF1DesE1MXtuG8N4FzNTqcclJgoXCLfcL56KgWBP8OFt5jKZQMCtu7k2CK3XsBl05UR7rwU0BLWxno99_U$. Quick question: if no source code has changed, do the nightly builds still get triggered? If not - what mechanism is used to know nothing has changed?

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https:/github.com/NASA-AMMOS/slim/issues/25*issuecomment-1461506381__;Iw!!PvBDto6Hs4WbVuu7!N-3XbT8BcKx28EF1DesE1MXtuG8N4FzNTqcclJgoXCLfcL56KgWBP8OFt5jKZQMCtu7k2CK3XsBl05UR7rwU0BLW710VFJU$, or unsubscribehttps://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AKQIMYHRTEHXG6GMZVTLC6DW3GEF5ANCNFSM5UTJYRDA__;!!PvBDto6Hs4WbVuu7!N-3XbT8BcKx28EF1DesE1MXtuG8N4FzNTqcclJgoXCLfcL56KgWBP8OFt5jKZQMCtu7k2CK3XsBl05UR7rwU0BLWw50jmn0$. You are receiving this because you were mentioned.Message ID: @.***>

jpl-jengelke commented 1 year ago

From @riverma: Quick question: if no source code has changed, do the nightly builds still get triggered? If not - what mechanism is used to know nothing has changed?

From @sharimayer: Yes, the nightly builds get triggered even if no source code has changed.

@sharimayer is correct.

However, in the Java VICAR ecosystem the SCRUB scans will take place at compile/build/publish time on the CI server -- which is much earlier when changes in code are pushed to Git. It's a lean system that is set right now to compile on demand when it sees changes. As a result Java scans will take place before the VICAR nightly builds. (Nightly builds pull previously compiled JARs from the artifact repository, e.g. Artifactory.)

For VICAR C builds, a compile/re-compile also takes place every night, whether something changes or not, and that is the point at which SCRUB scans are expected to take place in the C world.

We haven't touched much on the C world yet and this task is focused mainly on automating the Java and Python side by augmenting the CI server processes, e.g. Jenkins, or the GitHub Actions side, e.g. tooling in the source code repository server.

jpl-jengelke commented 1 year ago

From @riverma: I think @lylebarner's thoughts on a potentially running scans in parallel to builds is really interesting. Perhaps it's better to run scans nightly / on a schedule as opposed to tying to build triggers? The length of the code scan issue you mentioned is the same problem that comes up often for running system regression tests - which can take hours. Thoughts?

We'll surely discuss this at today's meeting but that is a legitimate solution to the conundrum of running the job after every build. It does, however, decouple it from code commits and compiles so feedback is not immediate. However, it's an open question whether the SCRUB scan process requires immediate intervention or feedback. I would argue that most similar processes, such as Dependabot security scanning, are run on timers. In many respects that's a policy question.

jpl-jengelke commented 1 year ago

Thanks Everyone for attending today's meeting. It was truly informative and inspirational to hear everyone's take on this task, and I believe we came to a consensus on how to proceed.

Takeaways:

After this meeting, I will re-draw the above workflow and proceed with developing the discussed solutions.

Proposed solution:

Please feel free to comment, ask questions or add additional information. Great meeting today!

riverma commented 1 year ago

@jpl-jengelke: recommendation next steps for this ticket (two separate PR's):

  1. A PR write-up containing directions for installing a GitHub Action to execute SCRUB automatically
  2. A PR write-up to execute SCRUB within a local Jenkins environment
riverma commented 11 months ago

See @lylebarner's ticket over at SCRUB to help with (ii): https://github.com/nasa/scrub/issues/99

riverma commented 6 months ago

@jpl-jengelke: recommendation next steps for this ticket (two separate PR's):

  1. A PR write-up containing directions for installing a GitHub Action to execute SCRUB automatically
  2. A PR write-up to execute SCRUB within a local Jenkins environment

@jpl-jengelke - I think we have (1) from @lylebarner write-up - we can point to this. Do you have a working, generic, solution to document for (2)? Having the two in place - we should be able to close this ticket soon.

jpl-jengelke commented 6 months ago

Yes I do. I will publish it shortly within the Java Starter Kit. That will close a number of tickets simultaneously.

riverma commented 6 months ago

That sounds great - thanks @jpl-jengelke!

ingyhere commented 3 months ago

I'm not sure we need to do a trade study nor a reference architecture in this ticket which appears to be dedicated to SCRUB itself. Those issues should be part of general security scanning tickets and are being handled in #104 and #110, so removing them from here.