Clarify ARL Form 1 process

[fulldecent]

The Major Review process includes "File an ARL Form 1". Instructions for filing the ARL form 1 include:

This information will be used both for publicity purposes, and by a supervisor and others to decide if it is time for a project to be subjected to another major review.

This is confusing because supervisors, in the process of a major review, will decide if a major review process is necessary.

Please clarify the intent of this passage.

[ckaran]

The issue is that the developers can decide on their own that they want a major review as this may boost their performance ratings. The supervisor can take a look at what they write and decide that they haven't really produced anything worth the effort of another major review, and so kills the major review. Any thoughts on how to express that?

[fulldecent]

The main vulnerability is in this sentence:

In the process described in this document, the primary purpose of the ARL Form 1 is for OPSEC review, but it also serves the secondary purposes of describing releases for publicity and metrics.

So that means developers are incentivised to pollute the major review pipeline with unnecessary changes to things which affect OPSEC (e.g. switch from SHA-192 to SHA-256 for something that does not matter.)

---

But, the ARL Software Release Process may already have the antidote:

The major review process MUST be followed if any of the following are true: ... It has been more than 1 year since the last time a major review has been done and the project is still active (material is being published to the public).

If ARL performs performance reviews only annually then gaming the system (unnecessary major reviews) would not be necessary since you already get a review once a year.

Maybe at the end of the year there will be a pile to do. But since

Publishing software as a regular business practice is new territory for ARL, and how to measure a project's impact on the general public is still a matter for debate.

we can probably cross that bridge when we come to it.

---

Recommendation

Change:

This information will be used both for publicity purposes, and by a supervisor and others to decide if it is time for a project to be subjected to another major review.

to:

A supervisor will use the filed ARL Form 1 to decide if the major review process is necessary or if, instead, the minor review process will be followed.

[ckaran]

OK, I see your concerns now. I'm not concerned because of how the incentives are structured; a developer only gets credit for the work they've generated since the last major review and its impact since the last major review. So if developer makes a meaningless change and tries to do another major review, their metrics won't improve at all for the second review.

That said, there is a constant (and moderately high) cost to performing a major review in the amount of time it takes. So my expectation is that while they can attempt to game the system, it will cost more than it will gain. Especially as the supervisors catch on, get annoyed, and start giving them less and less credit for poor work.

As for the wording, I can see your point about clarity. I do want to inform people that it can be used for publicity purposes as well as for evaluation purposes though; any thoughts on how to state all that clearly?

[fulldecent]

Who uses it for publicity purposes? I would put it like this:

A supervisor will use the filed ARL Form 1 to decide if the major review process is necessary or if, instead, the minor review process will be followed. Also, [THE PUBLICITY OFFICER] may use information from the ARL Form 1 for publicity purposes.

[ckaran]

I like it! The official name for [THE PUBLICITY OFFICER] at ARL is The Public Affairs Office (PAO).

Do you want to do a pull request (so you get credit), or do you want me to add it in directly? You know my preference is that you get credit for your work, but it's up to you...

[fulldecent]

PR coming.

[ckaran]

Just merged it in, I'm going to close this issue.