search

openzipkin-attic / apache-release-verification

Apache License 2.0
3 stars 1 forks source link

Separate ASF-required and non-ASF-required checks in summary #29

Open abesto opened 5 years ago

abesto commented 5 years ago

@adriancole in #27:

probably it would help my brain a lot to classify what are actually ASF requirements from what are not. There's no requirement to have the files in the source dist match the repo. The first couple repos we've done matched so mentally we are anchored to this concept. Since we are testing things that are not requirements and not necessarily even valid. It would be cool to order the ones that are concrete at the top, then a line or something, then the extra checks we made up which may help, but are not actually requirements at the bottom.

codefromthecrypt commented 5 years ago

Yep! what I meant is that there's no requirement for a match between the repo structure and the source distribution. Common problems in the last few releases are focus on what's in the repo vs what's actually required (ex NOTICE, DEPENDENCIES etc)

codefromthecrypt commented 5 years ago

most obvious point for us is several hours of discussion on the zipkin-api repo which has an incidental deployment and files around that. Most of this time was lost due to the assumption and mental anchoring folks besides me have.

That would certainly not be the only case. I think if we spot checked a few apache projects, we'd find out quickly there's a certain amount of fuzz in the assumption what's in the repo must all be in the source dist.

codefromthecrypt commented 5 years ago

also I would note that almost or perhaps exclusively, we have no IPMC folks verifying anything we do here is congruent. This lack of guidance is probably what leads to very quick anchoring to limited experience.

abesto commented 5 years ago

Gotcha. I think it's important to distinguish between:

  1. the contents of the source files (and number of them) must match – I just checked, and this is not explicitly called out anywhere in the ASF policy that I can see, it seems implicit in everything. The trick here is that what constitutes a "source file" is not trivial, see zipkin-api.
  2. the folder structure in which the source files live in the release should I guess bear some resemblance to the folder structure in the repo, but this seems to be totally not a requirement.

I'm unsure how to approach the first one (a band-aid for the second one will be #20). Maybe the most relevant part of the policy is https://www.apache.org/legal/release-policy.html#source-packages:

Every ASF release MUST contain one or more source packages, which MUST be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools.

We can verify this by automatically running a build-and-test, when we know how, and when this is relevant. However, this won't tell us if, for example, a source file containing tests is missing from the source archive. If a source file containing tests is missing from the source archive, then I'd say we don't satisfy the above requirement, in that users can test something, but it's not the version of the software the release pretends to be. This goes double for a situation where a source file is missing, but that doesn't break the build-and-test process for whatever reason, but it does cause a difference in runtime behavior.

Let's get some of our mentors involved?

codefromthecrypt commented 5 years ago

maybe you can do next release then decide?

codefromthecrypt commented 5 years ago

I think if you personally had to experience the impact of this tool, it would help shape it. Right now, there's a zipkin-reporter-java repo to release once zipkin is out. Sound fair?

codefromthecrypt commented 5 years ago

that means you do the recutting of tags explanations etc when people get scared etc. it is this, and the hours lost on this thrashing, that have cost me the most unnecessary time.

codefromthecrypt commented 5 years ago

alternatively you can defer first to your users about how the experience is, what is desired and not desired. The push to add more errors about source congruence is from you personally, and not a match to verification errors encountered recently, if except for adding scary messages which make getting people to vote a difficult nagging task that I've been doing in ways you can see and also not (ex direct messaging and begging)

abesto commented 5 years ago

Adrian, if you think using this tool is detrimental to Zipkin, then let's not use it. Re. thrashing, you're quite welcome to step out of this thrashing about the direction of the tool, and ask mentors to take over.

Re. user experience, let's not change the output based on what's convenient, but on actual ASF requirements, otherwise there's no point to this whole thing. All my comments are about making sure we understand what are the actual ASF requirements.

Overall, at this point I'm happy to step back and start doing release reviews manually, until someone else commits to arbitrating arguments like this. You don't have the bandwidth, and I believe they're crucial, so we're at an impasse.

(I'm about to head out, silence from here is not passive-aggressive, I'll just be not available to respond)

codefromthecrypt commented 5 years ago

I think what's frustrating to me is that you ask for things to be ERROR which are not in the docs, nor in recent problems in source verification votes. In fact you are doubling down on a problem (linty files being now error unless otherwise noted)

It would be less frustrating if you documented why you feel strongly about this with something.. like an ASF requirement or a release failure on account of something before trying to push it as mandatory.

Tools we have even without mentors includes the actual requirements, http://www.apache.org/legal/release-policy.html#what-must-every-release-contain

and also advice we got for source verification votes ex. https://lists.apache.org/list.html?general@incubator.apache.org

Anything beyond that is simply not something we need to deal with. We had a huge ordeal trying to get IPMC to be lenient and not interfere or cause problems. I think it is draconian to suggest my way or highway on either what you merge here is strict or we don't use the tool. I don't think you mean that. I am just hoping you can add at least a temporary preference to not introduce FUD (ex ERROR by default on things not ASF, or cite why they are ASF with actual facts and email .. and if not.. figure that out BEFORE assuming something)

basvanbeek commented 5 years ago

I understand the desire to have no grey areas of concern but can we indeed only implement the checks that actually are based on the explicit requirements. If we have additional desires or grey areas let's discuss them but only if a proper case for why we want it is provided.

The worst thing to do is to get into a mindset where we try to think about what the ASF would want. It is either explicit already, there exists "jurisprudence" or it will be raised by IPMC. Until then I seriously opt for "no action required"

abesto commented 5 years ago

Thank you for the thoughtful comments! Bottom line, I agree with everything in these last two comments, so that must mean I'm not very clear about where my head is. I apologize for this, and the frustration it caused. Side-note: I don't intend this to be my project, I very intentionally created it under openzipkin-contrib. Worst case, I'm happy to step back and let others drive it forward, with zero bad feelings, if we can't align our thinking.

Which seems to be not the case, I think we're pretty much aligned on goals. Once again I apologize for causing frustration with my lack of clarity. Lemme try to summarize where my head is:

On top of all this, and I think this is where things went south: when development on this tool reaches gray areas and edge cases, I tend to also bring in the additional idea of "release correctness", and catch mistakes with this automation. Having these in a single tool is convenient probably, but there must be a clear delineation between the "ASF requirements checker" and "release correctness helper" (which is what this very issue is tracking)

Finally, one learning for me here is to start discussions around better understanding ASF requirements on the dev list, or possibly the wider incubator list, looking for input, as opposed to handling them here on issues. For instance, I think that a content mismatch in a source file between the source archive and the tagged commit in the repo is an explicit violation of ASF requirements, but the right way to approach that is to get confirmation or correction from ASF people, and then put up a PR with appropriate context. This should eliminate time wastage on our end, and ensure we make the right calls. Starting such discussions on issues on this tool, I now understand, gives the impression that I already have a strong stance I'm unprepared to reconsider, which is not the case.

Sound good?

Love you all!