FOIA, History, and Rewrites

[tomberek]

Continue from #62.

[rthbound]

Background for #67 (this issue) starts here: https://github.com/deptofdefense/code.mil/pull/62#issuecomment-282606545

Wait. Actually, does that bring up a good policy question (whether or not DoD employees should overwrite history at all)?

[fulldecent]

This issue seeks to discuss whether the US Department of Defense should interpret legal policy as allowing or not allowing using of git commands which cause a loss of historical record on the main repository.

This discussion appears to be off topic of this project's goals and I believe it is out of scope for this project to consider legal interpretations of git commands.

As such I recommend closing this issue.

[ckaran]

With all respect to @fulldecent, git history may be a record as defined here. That means those records may need to be kept, and before a rewrite to history is done, the appropriate legal counsel may need to be consulted.

[rthbound]

It has less to do with git, more to do with destruction of records -- who contributed to which projects, when... things like that I'd think would be subject to FOIA requests.

[ckaran]

@rthbound I agree, but that is the point; the git repository is the record in this case. Destroying that record can lead to serious legal issues. Rewriting history should not be done lightly, and should be done with legal counsel's advice. Ideally, we'd be able to say why the contents were scrubbed, but that introduces its own headache since history will have been changed at that point...

[fulldecent]

Technical perspective:

git history is not destroyed when it is rewritten. There are server backups on the git server also there are client backup on the committer which are usually only removed if you clean the git repo. Lastly, it may be preserver by any client that has synced the repository.

Practical perspective:

If this project believes that git policy is in scope, then should we also be considering if squashing commits during pull request review is legal? That was being facetious. No, we should not be considering the legal ramifications of pull request merge strategy.

Legal perspective:

General best practice is that you do not rewrite git history. However there are many reasonable exceptions: accidental commits with passwords, accidental commits of incomplete work to the wrong branch (grrr to git 2.0's new policy of push ALL branches by default). In summary because there are very reasonable and practical purposes for rewriting git history (WHICH HAVE NOTHING TO DO WITH DESTROYING RECORDS) then therefore it is unnecessary to create a policy of disallowing git history.

Trust me, if the maintainer of a project does inappropriately rewrite history with anything other than the best intentions then it will get on the front page of slashdot anyway. This is a self-regulating problem.

[rthbound]

@fulldecent from #62

Mistakes. If credentials are published, that's embarrassing. I'd argue that you should absolutely not overwrite that history, but rather change your credentials. Maybe I'm a bad guy subscribing to changes. I cache/persist all of them. You push credentials, I snag them. You wipe history. You just hid evidence showing you're to blame for my subsequent abuses.

Again, I'm just an enthusiastic open source contributor, but it sounds like y'all(?) have put too little thought into this.

[ckaran]

@fulldecent I see your points, and I'm glad you see mine. I agree that there are going to be cases where we need to rewrite history, however, those cases will need to be done with the advice of legal counsel. In addition, as you've pointed out it is going to be very difficult to destroy all history in some cases. For credentials and passwords, they'll also have to be changed. However, removing the history without keeping a backup of the original somewhere means that a record was destroyed, and it may have been a Federal Record.

So, how about this compromise: in general, it is very bad form to rewrite git history. If you must rewrite it, get the advice of competent legal counsel to determine if you're permitted to do so, and if you're required to keep a copy of the record somewhere safe before doing the rewrite.

@rthbound If you've pushed passwords/credentials, you'll have to change them anyways; there's just no other way around it from a security point of view.

[fulldecent]

Yes, of course you should change any compromised credentials.

Git has features to change history and they were not put there by accident. People do not need lawyer permissible to use other git features so they should not need lawyer permission to use git history rewriting features. Likewise government employees do not need permission to delete files on their computer. The same backup mechanisms that apply to those file will apply to government computers that have git installed.

Therefore, git policy is outside the scope of this project.

[rthbound]

So, how about this compromise: in general, it is very bad form to rewrite git history. If you must rewrite it, get the advice of competent legal counsel to determine if you're permitted to do so, and if you're required to keep a copy of the record somewhere safe before doing the rewrite.

This would address my concerns.

@rthbound If you've pushed passwords/credentials, you'll have to change them anyways; there's just no other way around it from a security point of view.

That's my point. If this occurs, change credentials immediately. Don't waste time amending commit history before doing so. In fact, don't amend it afterward either. I want my FOIA request to show mistakes made by my government.

[fulldecent]

And of course pull requests with squash will also "destroy" records. Again we should not mandate against pull requests because that is rediculous. If somebody wants to FOIA against the lost pull request commits then they can do that and they will get back the git objects from the tape backups of the computers.

[fulldecent]

The goal of this project is to allow government employees to happily and fearlessly use git/github just like the billions of other people that use it. When we start telling people you need lawyer permission to use the squash and merge feature for a PR then we are being counterproductive and it becomes another reason government employees will stick with what is safe rather than contributing to their maximum potential.

Again, nothing is ever lost, we have tape backups of everything. Therefore because data retention are not an issue then this discussion is moot.

[rthbound]

Again, nothing is ever lost, we have tape backups of everything. Therefore because data retention are not an issue then this discussion is moot.

If you don't create some record of your having amended the history, how will anyone know they need to go to the tapes?

[fulldecent]

Someone is free to constantly pull the repository if they want. And someone could pull all the repositories all the time to check for history changes if that's what they want to do.

There is no legal requirement that government needs to make it easy for people that want to file FOIA.

[rthbound]

But there is a legal requirement that you not destroy records.

Y'all are going to do what you're going to do anyway. I'm just trying to ask that you be responsible and ethical in doing it.

[fulldecent]

Yes. This record keeping compliance is met by tape backups. This is why government computers do not have the Recycle Bin disabled.

[ckaran]

@fulldecent Do you work for the Federal Government? If so, where? Not all computers are backed up, and even those that are backed up may only be backed up on a nightly basis. I average 10 commits/day on my projects. I could easily create & squash a commit before any backups occurred. @rthbound is correct; there is a legal requirement for not destroying records, and rewriting history is going to require some amount of involvement with the agency's lawyers.

[rthbound]

@ckaran I thought that smelled fishy. Thank you for addressing my concerns.

[fulldecent]

In the situation you are describing (nightly backups, 10 commits/day) then yes, you will very likely not lose any information and be in full technical compliance of records keeping requirements (if they actually do apply).

Documentation at https://git-scm.com/docs/git-gc

[tomberek]

I am hesitant to try to set an iron-clad policy. This can can unintended consequences and a chilling effect, where the very thing we'd like to encourage is participation. I like the thought behind

So, how about this compromise: in general, it is very bad form to rewrite git history. If you must rewrite it, get the advice of competent legal counsel to determine if you're permitted to do so, and if you're required to keep a copy of the record somewhere safe before doing the rewrite.

Should the CONTRIBUTORS.md discuss this policy? Should this be a recommendation to project managers? It doesn't seem far from standard OSS practices, so it may be best to just refer to established practice than try to set something new.

If nothing else, an addition to the FAQ might be in order.

[jbjonesjr]

Should the CONTRIBUTORS.md discuss this policy?

I would argue this is beyond the scope of this repository and its current focus on OSS licenses. While a very important question to ask about DoD Open Source repositories, it applies far beyond Open Source, and beyond just the DoD (for example, https://github.com/usepa, https://github.com/usds, and https://github.com/ngageoint are all using git and GitHub, and would likely have a stake in any decision, to name just a few out of many).

I would suggest, however, the Government Peer group (https://www.github.com/government/best-practices/issues) would be a good forum to continue this discussion with a wider audience.

[rthbound]

@jbjonesjr I do agree with you, 💯, both that a policy on the subject is outside this project's stated scope and that the subject reaches beyond the DoD and beyond OSS. Thank you also for linking to these additional resources.

I greatly appreciate all of you participating in this discussion, and the concern you've shown for preserving Federal Records.

I feel like my concerns have been addressed. I do think, @tomberek, that it would not be out of place to mention squashing in your contributions guidelines (see rails'), as I mentioned in #62:

Maybe in the contributing guidelines you say "before we merge, we may ask you to squash your commits into one so we can maintain a clean history."

Thanks y'all 👏

[BrandonBouier]

This is, in my opinion, the best discussion in this repo! I want to use it as an example of good, solid discourse & debate :)

For the record, I personally am not a fan of rewriting git history, much like @rthbound. That said, I'm also not a fan of trying (ultimately in vain) to restrict usage of tools like git in order to enforce a particular outcome. That tends to lead to systems that people will constantly have to work around or simply won't use. And requiring legal council for certain git actions would quickly make lawyers the bottleneck in the critical path of software development. I don't think anyone would want that.

So overall, I agree with @fulldecent.

However, as @ckaran strongly implied, in government there can often be a gap between what an org is required to do and what it is actually able to do. I've seen many a policy/regulation/law decreed with little to no resources (funding, equipment, staff, etc.) to make it happen.

At the end of the day though, @jbjonesjr is correct: this discussion goes way beyond the scope of this project.

So basically, a part of me agrees with all of you :)

[ckaran]

@BrandonBouier You're right about the discussion!

The reason I'm hammering on the records and rules and regulations isn't because they're fun to follow, it's because I know what happens if a) they are ignored and b) something 'bad' happens. The go-to solution is generally to curtail the activity that lead to the 'bad' thing happening... in this case that could mean Government forays into Open Source. This is why I, @storrgie, and others have been so adamant about what code.mil was up to. We really want Open Source to work within the Government, and don't want someone using a global ban hammer to end it all.

[fulldecent]

I am not exactly sure how this repo will end up being used. Maybe it will be a "copy-me-to-start-your-next-project" template like this https://github.com/fulldecent/swift3-module-template or maybe it will be a web page with a bunch of FAQs for people that want to start a project like https://opensource.guide/

Either way, this discussion is valuable and we may do well to include a FAQ section or wiki page that summarizes what we discussed. Such a section should be helpful and discuss the most common and practical situations first.

Here is a stab at it:

---

A note on git features and rewriting history

The git program and hosted services such as GitHub include the ability for you to permanently delete parts of your project. They also allow you to "rewrite history" in that you may modify entries that have been previously published. Examples include:

• Deleting a branch that has not been merged into a surviving branch
• Merging a branch using the --squash feature
• Using git push with the --force feature

Code.mil is aware of records keeping laws in 18 U.S.C. 2257. We have not studied these laws nor are we aware of any case law that prescribes whether records keeping requirements pertain to portions of your git repository. Additionally, we do not know about YOUR agency's data backup systems. When you use the three deletion/rewriting abilities above the old file versions may still be persisted on your computer (see git garbage collection) and captured by these data backup systems.

In summary, git is a powerful tool so make sure you understand it before you use it. Your agency may prescribe additional guidelines or restrictions for using git. However, code.mil has no advice to offer on setting such guidelines or restrictions.

[ckaran]

@fulldecent I love it! The DOD is big, so different components will be able to make their own decisions as they see fit. It will also be useful for when people use github in unexpected ways (e.g., for policy discussions like this one, which may be a Federal Record and therefore have to be preserved). Each project will then have to decide what to do on its own.

[fulldecent]

@ckaran Thank you!

Sorry, just realized I left your above question unanswered. I was previously an engineer with the US Navy.

[ckaran]

@fulldecent Got it, then you have some experience in dealing with the Government. Good to know!

[jordangov]

Hey folks, it's been a while, but I wanted to close this one out. Here is what we have implemented and our plan:

1. We have protected the master branch on this repository. That branch will be the guidance that DDS is actually offering and can only be updated via a Pull Request. There will be no deletions or force pushes to that branch, including administrators of this GitHub organization.
2. I think we do need a FAQ item, and I think @fulldecent's language is a good starting point. I'll open a separate issue for reviewing and then adding that language.

Thanks for all the participation folks! And keep the conversation coming!