Make SECRET_DATA no longer a secret

[robredpath]

Make SECRET_DATA in CoVE no longer secret so that external parties can open PRs and run tests

[odscjames]

I've been talking to others about the OCDS part of this who have confirmed a worry I have.

There are several issues here that makes me think actually we should just delete the OCDS data.

As far as I've been told, there were 2 reasons why data was put in secret data as opposed to a public test.

1) the data was pre-release data from a publisher that was not public yet. 2) the data was real public data but changed to trigger an edge case that could be tested for, and we don't want this confused for real data.

If we were just anonymising to prevent problems associated with the second reason, it would be easy. Change the OCID prefix, change Org Id's, change the publisher meta - yes, the actual data is still mostly real but it was already public so who cares.

But anonymising for the first reason is, without more info that has been lost over time, impossible. While it's true the sources listed in the secret repo (Paraguay, Canada, Zambia) are all now public, we don't know if there was an change between the pre release data and what was actually published - it's possible our pre-release data has things in that were removed from the real data that should never be made public. We could try and scramble every field, but at that point any representation to what we are actually trying to test for is starting to be lost.

Other issues:

• All the data is V1.0, so it's value in testing later functionality is also diminished.
• There may be other reasons data was put into secret data that I don't know about, and thus other problems in releasing it.
• The tests contain tests of the CLI - that should really be in lib-cove-OCDS now.

There is one last complication; we are no longer sure why secret tests were added; as in, we aren't sure what bit of functionality they are actually trying to test. There is therefore a slight risk that we remove a test and later something breaks that a secret test would have caught. But also honestly, because of this reason when the secret tests fail and a dev goes to check if the difference in output is expected or a failure it is hard to know. And thus we currently have a risk the tests aren't doing their job anyway.

So I think, at this stage, I would just recommend removing OCDS COVE secret tests entirely.

I would then recommend a new ticket to generally identify functionality in COVE and Lib-Cove-OCDS we want tested, and then to work through and make sure things are tested in the relevant repo, making up new fake public data as we need.

Note there is also 360 secret test data, and some of the reasons above will apply to that too, but I don't work on 360 so don't know enough to make a recommendation about that.

[odscjames]

@jpmckinney, any thoughts on the OCDS side of this?

[jpmckinney]

Sure, either delete it or have the tests skipped if they are run in the Travis environment.

[jpmckinney]

Closed by #1205 ?