Add additional metadata

[jessehouwing]

Currently the metadata only contains the versions for each task, but it would be relatively simple to add:

• Preview (isStable: false)
• Deprecated
• Repository.Uri (SourceUrl)
• Marketplace Uri (Homepage)

Since all this data is available.

This would look something like this:

@JamieMagee @viceice What do you think and am I correct in guessing this would need to fit the ReleaseResult structure?

Would I need to add the metadata at the Release level, or can I just populate the repo and homepage data at the ReleaseResult level?

[viceice]

normally it would be enough to have it on package level. but if it can change on different versions, it needs to be on the release.

i would prefer to have it on package level to reduce data size.

[jessehouwing]

I think I can normalize most tasks to have the metadata at the package level.

Some tasks however, have been forked or transferred to a different publisher, in which case the metadata would have changed at some point. The flutter task is an example of that. It started out from:

• https://marketplace.visualstudio.com/items?itemName=aloisdeniel.flutter

Then got forked and taken over by:

• https://marketplace.visualstudio.com/items?itemName=Hey24sheep.flutter

This means that while the taskname sometimes stays the same between these extensions, the taskid and longtaskname will be different between them. So the taskid and longtaskname would be straightforward and should be able to use the latest version metadata, the shortname can be problematic.

@viceice Can the SourceUrl or the Homepage be set at the package level and the only where it need to differ at the release level? Or, since this would be for older versions in almost all cases, simply follow the latest version?

eg:

    "flutter": {
      "sourceUrl": "https://github.com/Hey24Sheep/Flutter",
      "homepage": "https://marketplace.visualstudio.com/items?itemName=Hey24Sheep.Flutter"
      releases: [
         {
             "version":  "0.3.10"
         },
         ...
         ...
         {
             "version:"  "0.2.39"
             "sourceUrl": "https://github.com/aloisdeniel/Flutter"
         },
    }

[jessehouwing]

Here's a generated sample json file containing:

• releaseTimeStamp
• soucreUrl
• homepage
• isStable
• isDeprecated
• downloadUrl

This amounts to +- 50MB of json.

Without the downloadUrl it's about 20MB.

https://github.com/renovatebot/azure-devops-marketplace/tree/features/preview-and-deprecation

[viceice]

i think we don't need the download url

[viceice]

what about creating a JSON file per taskid and a meta file with mapping only. then the renovate datasources first fetches the metadata file and the taskid files on demand via GitHub CDN.

that way we don't need to update renovate to get newer data.

[jessehouwing]

Sure a possibility. We now have the data and can slice and dice it in any way you want :).

[viceice]

@JamieMagee @rarkins WDYT?

[jessehouwing]

An even better option would be to switch over top an API based approach, the TFS/Azure DevOps API has all the information needed for installed tasks:

• https://github.com/renovatebot/renovate/issues/19725

On top of the data the server already knows, the marketplace data can help people find newer versions that weren't installed into their server.

For the built-in tasks it usually doesn't make much sense to report availability of versions the server doesn't have, since they only get updated through server upgrades or Microsoft's provisioning process and there isn't much an admin or user can do unless they go the "I'll install them myself route". I have a another project to give admins of Azure DevOps Server the ability to overwrite tasks:

• https://github.com/jessehouwing/azure-pipelines-tasks-zips

[JamieMagee]

@viceice I like that idea. We use GitHub pages as a JSON API.

@jessehouwing we can still implement the API-based datasource, and prefer it whenever Renovate is running in an Azure DevOps context. For the case where code is hosted in GitHub, but builds are run on Azure DevOps (this is the case for almost all the Microsoft org on GitHub for compliance reasons) we can use the cached data from this repo.

[jessehouwing]

I was thinking we could add additional config to make the API work even in a GitHub context. The Docs for configuring renovateme on Azure DevOps also show how to add GitHub configuration to generate release notes, this would be similar but the other way round.

Eg

--host-rules="[{\"domainName\":\"dev.rzure.com/myorg\",\"token\":\"$(AZURE_DEVOPS_TOKEN)\"}]"

[JamieMagee]

Yeah, good point. That's another good way to provide an API token.

[jessehouwing]

So the idea is to generate a json file per identifier and stick the ReleaseRef object in that file, right? That should be relatively simple to do from where we are now.

[viceice]

yes please. than update the renovate datasource to fetch from github cdn instead of buildin big fat json.

[viceice]

I've created a feature issue on renovate side for required datasource changes.

• https://github.com/renovatebot/renovate/issues/20019

[jessehouwing]

@viceice check out https://github.com/renovatebot/azure-devops-marketplace/tree/features/preview-and-deprecation/_data

[jessehouwing]

There were a few illegal characters in the identifiers, I solved that with:

$identifier.ToLowerInvariant() -replace "[^a-z0-9_.-]", "_"

for now.

[jessehouwing]

@JamieMagee any news on the front-end implementation of this?