Refactor templates repo for detect secrets and distribute to all EN repos

[jordanpadams]

💡 Description

https://nasa-ammos.github.io/slim/continuous-integration/starter-kits/#git-secrets

Probably need discussion with secrets/credentials provider: https://github.com/awslabs/git-secrets#id20

Should follow the guidelines and recommendations from here: https://github.com/NASA-AMMOS/slim/issues/89

[nutjob4life]

Hi @jordanpadams, in between the time this issue was created (Jan 17) and the time it was put onto the backlog (Oct 17), a major thing has changed over at SLIM, that being: they no longer recommend git-secrets.

Instead, they've settled onto another tool, detect-secrets, which they've documented extensively. It looks like detect-secrets can do a whole lot more than git-secrets, which worked fine for AWS keys. Yet detect-secrets can also find absolute file paths (which you commented on), sensitive email addresses, hostnames—and even do entropy analysis (automatic detection of secrets based on their high degree of randomness).

We've already settled onto git-secrets for the template-repo-python and template-repo-java, and I can certainly spread git-secrets out to the rest of our repositories.

But, as the issue description says above ↑, do we want to follow the guidelines at NASA-AMMOS/slim#89—which would mean going to detect-secrets—or do what the issue title says—which would be propagating git-secrets?

[jordanpadams]

@nutjob4life feel free to refactor all this towards the best solution you deem for our projects. in this case, let's role with detect-secrets.

[nutjob4life]

To resolve this, I need to use the sandbox, however I have a prerequisite pull request that'll need to be merged so I can get the sandbox working again.

[nutjob4life]

Notes to self.

Gathering the Tools

All developers—whether they use Python or Java—will need Python. (Honestly, I'm not sure how I feel about this.)

One quickie idea:

$ cd $HOME
$ python3 -m venv bin
$ cat > bin/requirements.txt <<EOF
pre-commit ~= 3.5.0
detect-secrets @ git+https://github.com/NASA-AMMOS/slim-detect-secrets.git@exp
EOF
$ bin/pip3 install --requirement bin/requirements.txt

Establishing the Baseline

To generate the baseline of PDS repositories, we'll need to exclude far more than the SLIM folk think, as we don't commit generated files (I hope they aren't either).

For Python

detect-secrets scan . \
    --all-files \
    --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.secrets..*' \
    --exclude-files '\.git.*' \
    --exclude-files '\.mypy_cache' \
    --exclude-files '\.pytest_cache' \
    --exclude-files '\.tox' \
    --exclude-files '\.venv' \
    --exclude-files 'venv' \
    --exclude-files 'dist' \
    --exclude-files 'build' \
    --exclude-files '.*\.egg-info' > .secrets.baseline

For Maven

detect-secrets scan . \
    --all-files \
    --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.secrets..*' \
    --exclude-files '\.git.*' \
    --exclude-files 'target' > .secrets.baseline

[nutjob4life]

Progress: this is now functioning in the sandbox's pds-template-repo-python. Next step is to try it in the sandbox pds-template-repo-java and then see if we want to roll it out everywhere, and update our various instructions.

(Still have an open question about whether we want to force Java developers to install Python.)

[nutjob4life]

Progress: this is now functioning in the sandbox's pds-template-repo-java.

I've also updated the READMEs for both Java and Python with instructions on how to use detect-secrets. These instructions reference a wiki page that is yet to be written, so these are the next steps:

1. Write the wiki page so the links from the READMEs aren't 404
2. Create PRs for these changes from the sandbox → the real NASA-PDS organization
3. Once approved, roll out all these changes to the rest of the NASA-PDS repositories

[nutjob4life]

Suggested repositories:

• [x] archive-viewer
• [x] data-upload-manager
• [x] deep-archive
• [x] doi-service
• [x] doi-ui
• [x] ds-view
• [x] harvest
• [x] lasso-issues
• [x] mi-label
• [x] nucleus
• [x] pds-api
• [x] pds-api-client
• [x] pds3-product-tools
• [x] pds4-jparser
• [x] pdsen-maven-parent
• [x] registry
• [x] registry-api
• [x] registry-common
• [x] registry-crawler-service
• [x] registry-dashboards
• [x] registry-harvest-cli
• [x] registry-harvest-legacy
• [x] registry-harvest-service
• [x] registry-loader
• [x] registry-mgr
• [x] registry-mgr-legacy
• [x] registry-pds3-catalog
• [x] registry-sweepers
• [x] roundup-action
• [x] search-ui-legacy
• [x] transform
• [x] validate

[tloubrieu-jpl]

@nutjob4life is investigating 4 repos which do not behave as expected. The others are good.

[nutjob4life]

It turns out it was three repositories, not four, that had problems with server-side secrets-detection via GitHub Actions.

Only one of those repositories got merged into main, so I've filed a fresh pull request to fix the problem with the server-side secrets detection.

The other two repositories, doi-ui and registry, had their pull requests remain open. So I've pushed additional commits to those branches to address the problems with server-side secrets detection.

The doi-ui PR has already been merged, leaving (at the time of this writing), the PR for secrets on registry open.

In addition, these PRs for secrets detection remain open at the time of this writing—but had no issues with their secrets-detection status checks in GitHub Actions:

• https://github.com/NASA-PDS/roundup-action/pull/129
• https://github.com/NASA-PDS/search-ui-legacy/pull/4
• https://github.com/NASA-PDS/validate/pull/770

[jordanpadams]

@nutjob4life one more I found: https://github.com/NASA-PDS/wds-react-legacy/tree/main/.github/workflows

[nutjob4life]

@jordanpadams pull request created

[jordanpadams]

Closing this out since most repos have been retrofitted. Will add to additional repos as needed if any were missed.