Improvement for the package triage process

[TomOne]

Some time ago I shared my experiences with trying to fix or update packages of other maintainers (https://groups.google.com/forum/#!topic/chocolatey/YyQn2Q2xmn0). I mentioned that this process is very time-consuming, especially when the maintainers don’t answer or ignore multiple requests, which happens occasionally. Browsing through the forum revealed that I’m not the only one who is frustrated by that.

Therefore I suggested a package process with fixed deadlines and designed a flowchart for it.

@ferventcoder already approved to make this process official and to add it to the wiki. Here is a draft for the page that would replace the current package triage process page: https://github.com/TomOne/nugetpackages/wiki/Package-Triage-Process

To apply this process it’s also needed to change the “Package Issues?” section on http://chocolatey.org/ and the permeable in the forum (https://groups.google.com/forum/#!forum/chocolatey). I included the suggested changes into my draft wiki page.

I’m really looking forward when this new triage process gets applied. I’m sure it will save time of package contributors and prevent annoyances.

[Redsandro]

I experience the same, it's okay if some maintainers are occupied, but often there is just no response.

There is also a massive amount of package trash where there are duplicate packages, personalized packages, personalized package names, and all kinds of other against-guidlines things.

Some of the packages I've seen on the gallery are prefixed with these names:

• Ed.Evans
• EthanBrown
• Fschwiet
• PProvost
• TimRayburn
• AlanStevens
• TheGiant
• secondstrategy
• kareemsultan
• Jivkok
• kellyelton
• fruin
• gb
• msanico (new 2014-07)
  • No icons
  • Bad pakage names
  • Separate packages for 32 and 64 bit
• jbatte47 (new 2014-07)
  • Naming guidelines
  • Personalized package

I'd like to see the gallery free of duplicates and personalized packages.

Also, new maintainers are not even encouraged to read the rules. Here is the registration verification mail:

Thank you for registering with the Chocolatey Gallery. We can't wait to see what packages you'll upload.

I would strongly advise to put the entire packaging guidelines text in the confirmation mail.

[TomOne]

Yes, I agree with you. But remember that many of these name prefixed and duplicate packages were created before I suggested the package rules and added them to the wiki after Rob had approved them. Sending the package rules with the confirmation e-mail would be a good idea, but in this way we won’t reach already registered maintainers.

A major problem is that also if new maintainers read the wiki once, they probably don’t check the wiki every week/month for updates. The only solution to let them know about important wiki and guideline updates would be a kind of newsletter that is sent to every single maintainer. These e-mails should then contain a link to the updated wiki sections and an invitation to read them.

But I see that we are moving a bit offtopic here. If you would like to discuss this aspect further, please open a new thread in the forum.

This issue is about an improved package triage process. There will always be package maintainers that ignore contributions and pull requests from others. The reasons for that are varied:

• Not even time for a couple of seconds for a short reply to the person who contacted the maintainer
• The desire to destroy or to harm the Chocolatey community
• Disinterest, laziness or stubbornness All these behaviours are unacceptable, so the main purpose of this improved triage process is to reduce the amount of time, effort and the vexation when people make pull requests to package maintainers.

[gep13]

With regard to the personal packages, I would like to see these moved off the main chocolatey feed onto a Community Chocolatey Feed on MyGet.Org, or encourage people to create their own MyGet feeds to store the packages.

[mwrock]

I agree with @gep13 that encouraging users to use a myget feed for personal packages is the way to go. As far as the educational process, certainly including the guidelines in the confirmation email or at least a link is a great idea, but might also consider checking conventions at push time. While we could not know for sure if a user is pushing a personalized package, we could look for dot notation that does not include the standard prefixes and then prompt the user with something like, "This might not be a package intended for public assumption, personal packages are best kept on other feds like myget. Are you sure you want to push this package?"

[ferventcoder]

Now you are talking more about the moderation rules engine I have been thinking about for awhile. :)

On Saturday, December 7, 2013, Matt Wrock wrote:

I agree with @gep13 https://github.com/gep13 that encouraging users to use a myget feed for personal packages is the way to go. As far as the educational process, certainly including the guidelines in the confirmation email or at least a link is a great idea, but might also consider checking conventions at push time. While we could not know for sure if a user is pushing a personalized package, we could look for dot notation that does not include the standard prefixes and then prompt the user with something like, "This might not be a package intended for public assumption, personal packages are best kept on other feds like myget. Are you sure you want to push this package?"

— Reply to this email directly or view it on GitHubhttps://github.com/chocolatey/chocolatey/issues/342#issuecomment-30058155 .

---

Rob "Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds http://ferventcoder.com http://twitter.com/ferventcoder

[gep13]

Does the Chocolatey Gallery, which comes from the Nuget Gallery, provide a lot of moderation capabilities? Or is this something that will have to be built in?

[ferventcoder]

That's funny right there...

On Saturday, December 7, 2013, Gary Ewan Park wrote:

Does the Chocolatey Gallery, which comes from the Nuget Gallery, provide a lot of moderation capabilities? Or is this something that will have to be built in?

— Reply to this email directly or view it on GitHubhttps://github.com/chocolatey/chocolatey/issues/342#issuecomment-30065185 .

---

Rob "Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds http://ferventcoder.com http://twitter.com/ferventcoder

[rismoney]

not really relevant to .org but for my internal repo I git back all my packages and use https://github.com/rismoney/chocolatey-git

I started making trivial tests for recognizing busted pkgs.

[gep13]

This is very cool! :-)

[rismoney]

thanks... So here is how we manage the package repo internally.

so we start with our git repo, setup very much like https://github.com/ferventcoder/chocolateyautomaticpackages

the difference being that each time we create or bump a version of the package we git tag it, and push to our internal gitrepo. This allows for peer review. The build.ps1 script is then able to iterate through every tag in the repo, and spit out the nupkg file into a dir. We also run a bunch of validation scripts on the package as a form of package unit testing using pester. We are getting to a point where the cpush can only be done by the bot running all this. We don't want to let users upload to the repo anymore, and feel the git review process is ideal for vetting the package before upload.

Now the chocolatey.org site has different intentions, but perhaps a CI validation process can be instituted. After all I don't think its practical to host every package in a single gitrepo, and gitsubmodules are nontrivial for the layman.

[gep13]

I really think that we can start to use this as a starter for ten. If nothing else, I think I am going to try to lever this for the packages that I maintain, and also at work (assuming you don't mind that is :-))

[ferventcoder]

There is nothing that keeps chocolatey from inspecting an already built package and attempting to do many of these things before it allows a package to be available to consumers.

[ferventcoder]

Nothing but rolling up some sleeves and getting to work that is.

[gep13]

Would this be a case of executing some checks during the cpack command, and then have it fail if something goes wrong, or would this be done on the Chocolatey.org server once the package has been uploaded to the server? Or both?

[rismoney]

imho, its overreaching for cpack. the receiver should have a triager (along the lines of a CI model)

[mwrock]

True. A CI model would be better. The question is what would a reliable CI model look like for chocolatey packages. This is something I have been thinking about lately. Ideally, it would spin up a VM of a vanilla OS install, run the package and validate that there were no errors (you could get fancier and have package authors include a validation scriptblock).

With the Boxstarter work that I hope to release this week this is simple:

$result = win7VM,win8VM,server2012VM,server2k8r2VM | 
    Enable-BoxstarterVM -Credential $cred -CheckpointName Fresh | 
    Install-BoxstarterPackage -PackageName myPackage -Force
$passed=$true
$result | ? { ($_.Errors.Count -gt 0) -or ($_.completed -eq $false)  } | % { $passed = $false }
return $passed

However this requires Hyper-V to be accessible. I'll be adding support to azure and AWS next but compute time is not free unless we could get a cloud provider to sponsor this. I think this could help in raising the credibility of the community feed which i think is one of the biggest road blocks to chocolatey adoption. Imagine adding some anti malware checking on top of that.

[rismoney]

funny you should say that. i hacked something pretty crude through appveyor that runs the test suite i have. it doesn't have the ability to spin up vms... :( and they don't have PR hooks, only single branch hooks). I ended up ditching it.

but last I heard travis is getting windows support, so when that drops, we might have something pretty awesome. btw-travis is awesome.

[gep13]

The validation is the tricky part though. For instance, in a recent package there was a sneaky addition of an Google and Google Tooblar:

http://chocolatey.org/packages/avastfreeantivirus

Whatever system we create needs to handle, somehow, the checking of unwanted uninstallations. I was discussing this with Rob earlier this week. This could be done, if possible, by taking a snapshot of the registry before and after the install (say with regshot) and also doing a diff of the file system (a harder thing to achieve) and checking for things that shouldn't be there. If anything else does appear, this would flag the package as rogue, and need further manual validation.

Gary

[gep13]

side note... @mwrock wrt to that script block, which looks amazing, do those names assume that a Hyper-V VM exists with that name? If it is not there, what does it do? I think you mentioned before that the VM would be created, but how does it know which iso to use when creating it?

[gep13]

Also, I love the idea of BoxStarter being used to validate Chocolatey Packages! :-) Talk about dog fooding! If the Azure support lands, would it be within the realm of legality to use an MSDN Subscription for spinning up VM's, even as a Proof of Concept? Happy to put my MSDN Sub to use for this if it is allowed. The credits aren't being used for anything else just now.

[mwrock]

Currently it assumes the VM to exist and will throw an error if not. After I wrap up azure and AWS and make a few "engineering debt" adjustments. I'm gonna start the functionality for injecting installer parameters into an ISO. The MS Hyper-V module provides pretty straight forward means to creating a VM from an ISO. I might add a convenience command to wrap it.

[gep13]

Sweet, sounds like you have it all wrapped up! Is this your sole project just now, features seem to be coming thick and fast :-)

[ferventcoder]

It's actually a both kind of thing. There would be a certain set of elements that would be run locally. Then a much bigger set that would overlap the local set (in case of client funny business) and run on the server.

Another way to think about it is unit tests versus integration tests. Locally we run some smoke checks and some validation of common issues.

On the server we let the package through but hold it until it passes more extensive validation rules.

The whole idea is a business rules engine that is based on a validation library I've written in the past. I just have to dig it out, brush it off and ensure it's all up to date.

On Tue, Dec 10, 2013 at 11:57 AM, Gary Ewan Park notifications@github.comwrote:

Would this be a case of executing some checks during the cpack command, and then have it fail if something goes wrong, or would this be done on the Chocolatey.org server once the package has been uploaded to the server? Or both?

— Reply to this email directly or view it on GitHubhttps://github.com/chocolatey/chocolatey/issues/342#issuecomment-30251496 .

Rob "Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds http://ferventcoder.com http://twitter.com/ferventcoder

[ferventcoder]

The script looks cool as well, but I don't like that it starts out positive ($passed=$true) and only gets set $false if it finds issues. Usually I assume failure unless the scenario tells me otherwise.

On Tue, Dec 10, 2013 at 1:56 PM, Gary Ewan Park notifications@github.comwrote:

side note... @mwrock https://github.com/mwrock wrt to that script block, which looks amazing, do those names assume that a Hyper-V VM exists with that name? If it is not there, what does it do? I think you mentioned before that the VM would be created, but how does it know which iso to use when creating it?

— Reply to this email directly or view it on GitHubhttps://github.com/chocolatey/chocolatey/issues/342#issuecomment-30261960 .

Rob "Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds http://ferventcoder.com http://twitter.com/ferventcoder

[mwrock]

Totally agree @ferventcoder I was lazy here just throwing out a proof of concept. For a real test I too always like to assume false.

[mwrock]

@gep13: It is my sole "after hours" project :) The code i have been pushing is largely based on spikes done in September so it has just been a matter of getting it fairly clean and around tests. Thought I was gonna be able to release Sunday night but after running tests in a domain joined environment, enabling WMI is kicking my butt. :(

[gep13]

Ah, I see, that makes a bit more sense. Based on what you had been pushing, I figured you hadn't slept in the last month! :-) Nightware, hopefully you can get it sorted out soon. Look forward to checking it out!

[Redsandro]

How did this suddenly turn into something I don't quite follow? :smile:

It's actually possible to have the package repo fire up a VM and test the package?

What if thee of us upload a package at the same time, three VMs will launch and we need three MS licenses?

I think the registry snap compare to check for bs is a bit tough though, updated software often has new fingers in the registry, so it would need manual updating. Might as well do a manual check on bs (toolbars) then.

[Redsandro]

I'll just leave this here..

https://github.com/Genbox/VirusTotal.NET

[mwrock]

For those interested. I released the Hyper-V functionality last night - http://boxstarter.org/VMIntegration

[rismoney]

Thats so awesome @mwrock !! can i make a suggestion? if you are going to provide other hypervisor/cloud functions, you might have the start of something like a powershell fog (see fog.io)

you might have, and I haven't looked at the source yet, but I envision these as external componentry you can just bolt on and re-use outside of box-starter.

[mwrock]

Yeah. That is what I'm thinking to a degree. However, this starts to look like Vagrant and my goal is not to be another Vagrant. If I had more time, I'd love to learn some Ruby and create a Boxstarter provisioner and a Hyper-V and azure provider for Vagrant. Hopefully I or someone else will eventually but I have some other stuff I really want to get to first. But yeah. I think you will likely see a Boxstarter API where you use the same command and pass in a provider string or enum value.

[Redsandro]

Just a thought, this would also help prevent the necessity of triaging in certain cases. Why not have a VM build all some core team Automatic Packages? We wouldn't need to be running Windows and a scheduler. Give a virtual chocolatey user some pushing to do. :)

[gep13]

I was having the same thought. I wondered whether the choco update package could be extended to include a flag to run the package through Boxstarter, and have it report back on success or failure. What do you think @mwrock

[Chromebuster]

In the next month or two, I'm going to have my servers up and running again, and at least one of them will be a TFS build server on top of other things, so i'm glad to allow you guys to use it; I'll run stuff on that little guy, and you can have your validation; just tell me what you want set up, and the server will have your TFS CI agent running on it.

[rismoney]

Can you elaborate more on this @Redsandro -https://github.com/chocolatey/chocolatey/issues/342#issuecomment-31094523

I don't understand not running windows, virtual Choco users, where boxstarter fits...

[Redsandro]

Can you elaborate more on this @Redsandro

In this case I'm actually not talking about Boxstarter, but in the light of this being a Chocolatey topic, the talk about VMs gave me this idea. Automatic packages need to be automatically generated on computers running all the time (or at least frequently). I don't run Windows, so I cannot turn my packages into automatic packages.

Why not have a VM do this, and have the user Chocolatey user push them, as described in the community packages repo.

For every package that is stuck unupdated with an unresponsive maintainer, we (or someone who decides this) can just 'kidnap' the stuck package, turn it into an autopackage, and hand it over to the VM/Chocolatey user.

[Chromebuster]

A VM shouldn’t be required; I have one, going on two, computers (servers, actually), that run all the time, constantly, nonstop. Invite them into Chocolatey’s service, please.

From: Redsandro [mailto:notifications@github.com] Sent: Monday, December 23, 2013 8:49 AM To: chocolatey/chocolatey Cc: Katherine Moss Subject: Re: [chocolatey] Improvement for the package triage process (#342)

Can you elaborate more on this @Redsandrohttps://github.com/Redsandro

In this case I'm actually not talking about Boxstarter, but in the light of this being a Chocolatey topic, the talk about VMs gave me this idea. Automatic packages need to be automatically generated on computers running all the time (or at least frequently). I don't run Windows, so I cannot turn my packages into automatic packages.

Why not have a VM do this, and have the user Chocolateyhttp://chocolatey.org/profiles/Chocolatey user push them, as described in the community packages repohttps://github.com/chocolatey/chocolatey-coreteampackages.

For every package that is stuck unupdated with an unresponsive maintainer, we (or someone who decides this) can just 'kidnap' the stuck package, turn it into an autopackage, and hand it over to the VM/Chocolatey user.

— Reply to this email directly or view it on GitHubhttps://github.com/chocolatey/chocolatey/issues/342#issuecomment-31119912.

[Redsandro]

Idea 1

Concerning the packages that are similar because owners don't respond and someone else decides to upload a newer version under a different name:

Maybe there should be a small group of moderators with the power to take over packages, remove the install script, add a dependency to the new package, or add the new package owner to the old package (to keep the sensible name).

I wanted to install Visual Studio for someone the other day, and there are like 40 packages. No idea which one I need. Maybe it's a bad example, but I had something like this before with some SQL implementation, and I remember having to compare versions, and guess which one I should install.

Idea 2

Automation. Set a clock (for e.g. 30 days) when a package maintainer is contacted for an update, and if they didn't respond within that timeframe, automatically add core team members as package owners and remove the old owner.

[ferventcoder]

The plan is to start adding more moderators this year for fun stuff like this.

On Wed, Apr 16, 2014 at 5:42 PM, Sander notifications@github.com wrote:

Concerning the packages that are similar because owners don't respond and someone else decides to upload a newer version under a different name:

Maybe there should be a small group of moderators with the power to take over packages, remove the install script, add a dependency to the new package, or add the new package owner to the old package (to keep the sensible name).

I wanted to install Visual Studio for someone the other day, and there are like 40 packages. No idea which one I need. Maybe it's a bad example, but I had something like this before with some SQL implementation, and I remember having to compare versions, and guess which one I should install.

— Reply to this email directly or view it on GitHubhttps://github.com/chocolatey/chocolatey/issues/342?utm_campaign=website&utm_source=sendgrid.com&utm_medium=email#issuecomment-40661480 .

Rob "Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds http://ferventcoder.com http://twitter.com/ferventcoder

[Redsandro]

Is it possible to automate a system to move packages to MyGet, which I know nothing about, but @gep13 recommended for alternative package hosting? Would be cool if moderators could 'move' invalid packages to a separate 'stream'.

[ferventcoder]

It is possible to do so with myget (copy them anyway)

[Redsandro]

@TomOne said on Dec 7 2013:

Yes, I agree with you. But remember that many of these name prefixed and duplicate packages were created before I suggested the package rules and added them to the wiki after Rob had approved them.

Guidelines like https://github.com/chocolatey/chocolatey/issues/316 and https://github.com/chocolatey/chocolatey/issues/317 are still being ignored well over half a year after I raised the issue. Soon every popular package will have a funny name, and only future obscure ones will be named properly. I recently came across HitmanProPortable_x64 and HitmanProPortable_x86. Thee "violations" in one package name. This slowly growing library of bad names can be (partially) prevented using some relatively easy tricks.

1. Put the entire packaging guidelines text in the confirmation mail. (see comment 30056205)
2. Put the entire naming guidelines in the confirmation mail.
3. Have cpack run checks, especially on the package id.
   • Similar to @TomOne's idea to have cpack prevent users from messing up encodings in https://github.com/chocolatey/chocolatey/issues/294
   • If it matches /portable/i but not /\.portable$/, refuse to pack/push.
     • Friendly message why.
   • If there's a period . not followed by installer$ or portable$, refuse to pack/push.
     • Friendly suggest replacing dots by dashes with example. E.g.: my.sweet.package -> my-sweet-package.
   • Same with underscores _.

[ferventcoder]

There is an easier fix, which is package moderation. Reject anything that doesn't meet the guidelines. Then like all of us are thinking, build it back into choco.exe during packaging and onto the server as package inspection rules to automatically reject packages that are not up to snuff.

[Redsandro]

That would help indeed.

But how would you typify "easier"? I think a moderation system could be in place a year from now. And as I said before, a poor solution in practice is infinitely better than a perfect solution in theory. (And I'm not even saying the solution is poor.)

The above 'trick' 1 and 2 could be in place today, because it doesn't require too much thinking or efford.

I can make a proposal with links instead of inline rules (because I cannot figure out how html markup works).

[Redsandro]

Here you go: https://github.com/chocolatey/chocolatey.org/pull/87

Feel free to word it any way you see fit.

[ferventcoder]

:)

[ferventcoder]

And as I said before, a poor solution in practice is infinitely better than a perfect solution in theory.

I've said something very similar to this many a time. :+1: A working solution is always better than a perfect solution that is not ready. (I mean, just look at chocolatey itself as an example of this in practice :/)

[rismoney]

I thought chocolatey goodness was as good as it gets.

[ferventcoder]

@rismoney The working implementation of chocolatey vPosh is a working solution, nowhere near perfect. I mean just look at all the issues, man! :)

The really good solution of chocolatey vNative Exe is going to be worlds closer to stable.