Is it possible to package this as docker image?

[ppanyukov]

Might be a long shot but how hard and possible at all to package this as a docker image and expose as a regular CLI with command line args?

Benefits this will have:

• No need to install this as an extension in VSTS account. We (and many otheres) cannot do this due to security policies.
• Can run locally or anywhere since it's a regular CLI.
• Docker images are obviously a standard de-facto these days -- we might want spin this up as a batch job in AKS you never know.

Everything should still work via volume mapping in docker. In VSTS we'd just run this with whatever args, no need for fancy UI really.

Also it would be nice to have the source code. It's a bit difficult to go through transpiled JS in VSIX package. I don't know if that is a too much of an ask though :)

[ReneSchumacher]

Hi Philip,

what exactly would you like to have as a CLI? Since we currently have three extensions in the marketplace, it's somewhat hard to guess :D

The biggest issues with converting build extensions to a CLI are their tight coupling to the agent infrastructure (we use lots of predefined variables, inherited proxy settings, etc.) and dealing with authentication and authorization. Pipeline tasks share the same security context as the rest of the pipeline so it's pretty easy to make API calls and retrieve data from Azure DevOps. In a CLI, you'd have to handle all that manually. In addition, some tasks (e.g., the Build Quality Checks) really are meant to be use in pipelines as they directly use pipeline information, set pipeline or pull request statuses, or otherwise make direct use of the pipeline context. In other words: it wouldn't be impossible to do this, but it's quite an effort.

I do understand the security concerns when it comes to marketplace extensions and I'm always happy to work with customers to resolve those. We've actually done a couple security reviews internally, since our extensions are used by many teams inside Microsoft as well.

I actually meant to make the code open source a long time ago, but I still haven't had the time to go through all the (lengthy) processes to do so. Still, we don't have anything to hide so I'm attaching the current version of the source code for Build Quality Checks, Create Work Item, and Post Build Cleanup here. If you have questions, feel free to ask :)

Ext-BuildQualityChecks.zip Ext-CreateWorkItem.zip Ext-PostBuildCleanup.zip Ext-Common.zip

Cheers, René

[ppanyukov]

It was about BuildQualityChecks, sorry I forgot there were three extensions here :)

Yes I see this might be quite a big effort to rip this out of VSTS, so maybe not a starter.

The biggest issue for us here is adding untrusted third-party extensions to VSTS. We don't fully understand the implications and risks and so we sit on the side of caution. I did read the FAQ here (https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/FAQ.md), and it's still not clear in my mind. If there are any additional docs or reading you could suggest it would be very helpful.

Also, if this extension was pubished by Microsoft proper, that would remove the problem, don't know if this is a feasible route to take :)

Thanks for the source code, will have a look!

[ReneSchumacher]

That is one of the discussions we keep having with customers and our FAQ is not really helpful in that regard. You can find our official docs about trusting marketplace extensions at https://docs.microsoft.com/en-us/azure/devops/marketplace/trust?view=azure-devops.

Since I wrote a longer email about extension trust to some internal folks, I'll just cite myself:

1. How can we ensure that an build task is not silently copying the sources somewhere else? Does the scope “vso.code” suffice or is the extension running inside a sandbox, where it can only communicate with the organizations base URLs?

   Extensions do not exactly run inside a sandbox. I.e., an extension can always communicate to external resources as long as there is nothing preventing the connection (e.g., a firewall in an on-prem Azure DevOps Server deployment). Scopes only limit an extension’s access to the server/service itself. Thus, if you don’t request access to code, the extension won’t be able to access code unless it provides a separate form of authentication (e.g., a PAT, service connection etc.). Without this separate form of authentication, the extension does always run in the current user’s scope. Thus, even if the extension asked for code access, it can only access code that the current user has permission to access. We do have mechanisms that prevent cross-side scripting. Thus, the extension cannot simply load script code from external sources and run that. At least I believe so.

   All of this is at least true for UI extensions. Pipeline tasks do always have the ability to get the pipeline’s authentication token and then have access to whatever the build identity (either Project Build Service Account or Project Collection Build Server Account) has access.

   To directly answer your question: you do not know if a build task silently copies sources to somewhere else, unless you look at the tasks source code. For tasks that can be installed on-premises, you can easily download the extension vsix file, extract it with a standard zip tool and look at the code that it contains. While this allows inspection of an extension prior to the first installation, it doesn’t help with automatic updates that run on Azure DevOps Services and even on Azure DevOps Server as long as the server has internet connectivity. Extensions are automatically updated as soon as a new version is published to the marketplace except when the extension owner requires additional scopes with a new version. Pipeline tasks are automatically updated to the highest minor and patch version; only major version updates require pipeline authors to explicitly update the pipeline definition.

2. How do we even know that an extension with a public repo is build from that repo?

   Short answer: you don’t. And I believe that is true for any extension, package, app, etc. in any marketplace, repository, or store. Afaik, every “hosting system” (e.g., Windows Store, npmjs.org, NuGet.org, Apple App Store, Play Store, etc.) for “packages” (i.e., NuGet, npm, bower, appx, vsix, etc.) only inspects the submitted packages an never build the packages from source code. Therefore, there is no way to ensure that an extension has been built from the public repo the author provides. Again, you could compare the code in the package to the code in the repo, but things like compilation or transpilation can make this a hard task.

3. Did we ever had a reported extension due to security violations?

   None that I recollect. We had one bad extension show up for the VS Code ecosystem though.

4. How do we at Microsoft consider an extension as “trusted” for our organizations? How do we validate them?

   Again, I don’t know. But here’s what I recommend my customers when it comes to extension:

   • Look at the overall appearance of the extension in the marketplace
     Does it have comprehensive documentation? Is there a link to licensing and support information. Is the code available? Is there a direct way to contact the extension author? Has it been published by a company or a “simple person”? How does the rating look like? Have there been questions and have they been answered in a timely manner? Is there a GitHub repo or other mechanism for tracking issue and are those issues actively worked on and solved? – These questions will help you get a feeling for an extension’s reliability and quality. If it has been published by a company, it should be easy to contact the company and ask for all the information you need. If it has been published by a simple person and the code is not publicly available, I expect the author to let me inspect the code upon request.
   • Inspect the extension code if you really want to know all the details of the extension
     Remember, this is no guarantee that there won’t ever be harmful code in an extension. et again, it helps forming an opinion about the extension and its author.
   • Pay close attention to a UI extensions requested scopes
     And don’t install extensions that ask for scopes for which you cannot figure out why they’re needed.
   • If you want to be sure for pipeline tasks, run them on a private agent with restricted internet connectivity
     That is the best way to prevent any outgoing traffic without you at least noticing it.

As I mentioned, our extensions had to go through a couple internal security reviews before some teams (Azure, Office) installed them. So by now I at least know what we do internally to ensure the quality of extensions we install from the marketplace.

Finally, here's an excerpt from another customer mail regarding the "third party" issue:

All Microsoft Premier Services extensions are not Microsoft “products” as there is no real product group behind the extensions. Premier Services is part of Microsoft’s support and services organization and helps customers get the most out of the Microsoft products they use. Part of that is that we develop extensions based on feedback we get during customer engagements. Whenever we have something that makes sense for a broader group of users, we publish those extensions on the marketplace.

Long story short:

• The extensions are not products in the sense that there is a full-blown product team behind it (we’re currently three people working on those extensions). • The extensions are indeed provided and supported by real Microsoft employees. • The extensions are not just experiments with limited support as is the case for the Microsoft DevLabs extensions.

You can rest assured that any publisher that uses the word Microsoft in their publisher name, has been tested and approved by Microsoft. Otherwise, those publishers are not approved and cannot publicly publish extensions on the marketplace.

I've talked to the marketplace team and tried to remove the "third party" label from our publisher but that doesn't seem to be that easy. They offered to switch to the Microsoft publisher, however, I'm unsure if this wouldn't be too disruptive. After all, extension are uniquely identified by publisher and ID so everyone would have to update all pipeline definitions to switch to the new publisher.

[ppanyukov]

Thanks for the email quotes, this is extremely helpful. I wish this area was not as complicated :(

They offered to switch to the Microsoft publisher, however

This would be super nice. Perhaps maybe rename these extensions with "mk2" suffix or something, i.e. essentially making a brand new extension?

[ReneSchumacher]

I'm gonna check my options here. I was already thinking about publishing the extensions under both publishers so people could switch from one to the other at their own pace.

[ppanyukov]

For extensions in general, one approach I'm thinking of is this:

• Fork the repo (provided licensing allows + source code available)
• Review and vet the code
• Publish as our own private extension
• Review and merge upstream changes when then happen.

Do you see anyone doing this a lot? Feels like a lot of effort for small org like ours.

I'm gonna check my options here. I was already thinking about publishing the extensions under both publishers so people could switch from one to the other at their own pace.

And thanks for this, I think we will wait, hopefully won't take too much time ! :)

[ReneSchumacher]

I haven't seen any customer doing something like this, not even the "large and paranoid" ones. ;-)

While this would work, it's probably way too much effort for most customers esp. if you have an actively engaged extension developer (like myself :-D). We sometimes push out two our three updates a week to fix bugs or other issues. Staying on top of all those incoming changes will definitely be hard.

[ReneSchumacher]

Hi @ppanyukov,

just wanted to let you know that we decided to switch to the Microsoft publisher. I'm not exactly sure how this will wok out but it should be non-disruptive so even existing installations of the extensions should continue to work (which is my biggest concern). The move should complete during the week and I'll update this issue accordingly.

René

[ppanyukov]

This is fantastic news, looking forward installing this :)

[ReneSchumacher]

Behold! We're now Microsoft :-D

That went far smoother (and much quicker) than I expected. I double checked to make sure that we're not labeled as third party anymore. If you have any other concerns regarding our extensions, please open another issue here or contact us at PSGerExtSupport@Microsoft.com.

Happy building! René

[ppanyukov]

Amazing work! many thanks for this. Lets close the issue, unless you want to keep it around?