search

quarkusio / registry.quarkus.io

Quarkus Extension Registry application
https://registry.quarkus.io
Apache License 2.0
10 stars 11 forks source link

Serve platform extension catalogs #92

Closed gastaldi closed 1 year ago

gastaldi commented 2 years ago

This PR introduces some new admin endpoints (check OpenAPI for more details) and introduces some DB schema changes

Fixes #91

gastaldi commented 2 years ago

@aloubyansky please give this PR a try when you get a chance

aloubyansky commented 2 years ago

Trying it locally. Launched the registry in dev mode and populated it with the current extension catalog.

Issue 1. Navigating to http://localhost:8080/maven/io/quarkus/registry/quarkus-platforms/1.0-SNAPSHOT/quarkus-platforms-1.0-SNAPSHOT.json I see

{
  "platforms" : [ {
    "platform-key" : "io.quarkus.platform",
    "name" : "Quarkus Community Platform",
    "streams" : [ {
      "id" : "2.8",
      "releases" : [ {
        "version" : "2.8.0.Final",
        "member-boms" : [ "io.quarkus.platform:quarkus-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-optaplanner-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-kogito-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-qpid-jms-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-cassandra-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-amazon-services-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-camel-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-hazelcast-client::pom:2.8.0.Final", "io.quarkus.platform:quarkus-debezium-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-blaze-persistence-bom::pom:2.8.0.Final", "io.quarkus.platform:quarkus-google-cloud-services-bom::pom:2.8.0.Final" ],
        "quarkus-core-version" : "2.8.0.Final",
        "last-bom-update" : "io.quarkus.platform:quarkus-bom:2.8.0.Final",
        "platform-release" : {
          "platform-key" : "io.quarkus.platform",
          "stream" : "2.8",
          "version" : "2.8.0.Final",
          "members" : [ "io.quarkus.platform:quarkus-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-optaplanner-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-kogito-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-qpid-jms-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-cassandra-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-amazon-services-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-camel-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-hazelcast-client-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-debezium-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-blaze-persistence-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final", "io.quarkus.platform:quarkus-google-cloud-services-bom-quarkus-platform-descriptor:2.8.0.Final:json:2.8.0.Final" ]
        },
        "codestarts-artifacts" : [ "io.quarkus:quarkus-project-core-extension-codestarts::jar:2.8.0.Final" ],
        "project" : {
          "properties" : {
            "doc-root" : "https://quarkus.io",
            "rest-assured-version" : "${rest-assured.version}",
            "compiler-plugin-version" : "3.8.1",
            "surefire-plugin-version" : "3.0.0-M5",
            "kotlin-version" : "1.6.10",
            "scala-version" : "2.12.13",
            "scala-plugin-version" : "4.5.6",
            "quarkus-core-version" : "2.8.0.Final",
            "maven-plugin-groupId" : "io.quarkus.platform",
            "maven-plugin-artifactId" : "quarkus-maven-plugin",
            "maven-plugin-version" : "2.8.0.Final",
            "gradle-plugin-id" : "io.quarkus",
            "gradle-plugin-version" : "2.8.0.Final",
            "supported-maven-versions" : "[3.6.2,)",
            "proposed-maven-version" : "3.8.4",
            "maven-wrapper-version" : "3.1.0",
            "gradle-wrapper-version" : "7.4.1"
          }
        }
      } ]
    }, {
      "id" : "2.7",
      "releases" : [ {
        "version" : "2.7.5.Final",
        "member-boms" : [ "io.quarkus.platform:quarkus-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-optaplanner-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-kogito-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-qpid-jms-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-cassandra-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-amazon-services-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-camel-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-hazelcast-client::pom:2.7.5.Final", "io.quarkus.platform:quarkus-debezium-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-blaze-persistence-bom::pom:2.7.5.Final", "io.quarkus.platform:quarkus-google-cloud-services-bom::pom:2.7.5.Final" ],
        "quarkus-core-version" : "2.7.5.Final",
        "platform-release" : {
          "platform-key" : "io.quarkus.platform",
          "stream" : "2.7",
          "version" : "2.7.5.Final",
          "members" : [ "io.quarkus.platform:quarkus-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-optaplanner-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-kogito-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-qpid-jms-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-cassandra-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-amazon-services-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-camel-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-hazelcast-client-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-debezium-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-blaze-persistence-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final", "io.quarkus.platform:quarkus-google-cloud-services-bom-quarkus-platform-descriptor:2.7.5.Final:json:2.7.5.Final" ]
        },
        "last-bom-update" : "io.quarkus.platform:quarkus-bom:2.7.5.Final",
        "codestarts-artifacts" : [ "io.quarkus:quarkus-project-core-extension-codestarts::jar:2.7.5.Final" ],
        "project" : {
          "properties" : {
            "doc-root" : "https://quarkus.io",
            "rest-assured-version" : "${rest-assured.version}",
            "compiler-plugin-version" : "3.8.1",
            "surefire-plugin-version" : "3.0.0-M5",
            "kotlin-version" : "1.6.10",
            "scala-version" : "2.12.13",
            "scala-plugin-version" : "4.5.6",
            "quarkus-core-version" : "2.7.5.Final",
            "maven-plugin-groupId" : "io.quarkus.platform",
            "maven-plugin-artifactId" : "quarkus-maven-plugin",
            "maven-plugin-version" : "2.7.5.Final",
            "gradle-plugin-id" : "io.quarkus",
            "gradle-plugin-version" : "2.7.5.Final",
            "supported-maven-versions" : "[3.6.2,)",
            "proposed-maven-version" : "3.8.4",
            "maven-wrapper-version" : "3.1.0",
            "gradle-wrapper-version" : "7.3.3"
          }
        }
      } ]
    } ],
    "current-stream-id" : "2.8"
  } ]
}

codestarts-artifacts, project and even current-stream-id could be considered redundant in this content.

aloubyansky commented 2 years ago

To be able to test the CLI, I had to apply the following changes:

registry.quarkus.io:

diff --git a/src/main/java/io/quarkus/registry/app/maven/RegistryDescriptorContentProvider.java b/src/main/java/io/quarkus/registry/app/maven/RegistryDescriptorContentProvider.java
index cbce744..ee59ce0 100644
--- a/src/main/java/io/quarkus/registry/app/maven/RegistryDescriptorContentProvider.java
+++ b/src/main/java/io/quarkus/registry/app/maven/RegistryDescriptorContentProvider.java
@@ -72,6 +72,7 @@ public class RegistryDescriptorContentProvider implements ArtifactContentProvide
         platformsConfig.setArtifact(new ArtifactCoords(mavenConfig.getRegistryGroupId(),
                 Constants.DEFAULT_REGISTRY_PLATFORMS_CATALOG_ARTIFACT_ID, null, Constants.JSON,
                 Constants.DEFAULT_REGISTRY_ARTIFACT_VERSION));
+        platformsConfig.setExtensionCatalogsIncluded(true);
         qer.setPlatforms(platformsConfig.build());

quarkus:

diff --git a/independent-projects/tools/registry-client/src/main/java/io/quarkus/registry/client/maven/MavenRegistryClientFactory.java b/independent-projects/tools/registry-client/src/main/java/io/quarkus/registry/client/maven/MavenRegistryClientFactory.java
index ca5e8140fe..88f3c3ac59 100644
--- a/independent-projects/tools/registry-client/src/main/java/io/quarkus/registry/client/maven/MavenRegistryClientFactory.java
+++ b/independent-projects/tools/registry-client/src/main/java/io/quarkus/registry/client/maven/MavenRegistryClientFactory.java
@@ -159,7 +159,6 @@ public class MavenRegistryClientFactory implements RegistryClientFactory {
         } else {
             platformsResolver = new MavenPlatformsResolver(platformsConfig, defaultResolver, log);
         }
-
         return new RegistryClientDispatcher(config, platformsResolver,
                 Boolean.TRUE.equals(platformsConfig == null ? Boolean.FALSE : platformsConfig.getExtensionCatalogsIncluded())
                         ? new MavenPlatformExtensionsResolver(defaultResolver, log)
@@ -226,7 +225,7 @@ public class MavenRegistryClientFactory implements RegistryClientFactory {
         if (client == null) {
             return descriptor;
         }
-        if (isComplete(client)) {
+        if (isComplete(client, descriptor)) {
             return client;

.quarkus/config.yaml:

---
debug: true
registries:
- registry.quarkus.io:
    maven:
      repository:
        url: "http://localhost:8080/maven"

Running qs create --verbose I get

[aloubyansky@localhost playground]$ qs create --verbose
Creating an app (default project type, see --help).
[DEBUG] Creating a new project with initial parameters: CreateApp{gav=TargetGAVGroup [groupId=org.acme, artifactId=code-with-quarkus, version=1.0.0-SNAPSHOT, gav=null], quarkusVersion=TargetQuarkusVersionGroup{stream=null, platformBom=null}, targetBuildTool=TargetBuildToolGroup [gradle=false, gradleKotlinDsl=false, jbang=false, maven=false], targetLanguage=TargetLanguageGroup [java=<<detect java runtime version>>, kotlin=false, scala=false, sourceType=null], codeGeneration=CodeGenerationGroup [includeCode=true, includeWrapper=true, packageName=null], extensions=[], project=BaseCreateCommand [outputPath=null, registryClientRegistryClientMixin [useRegistryClient=true], projectDirName=null, projectRootPath=null, targetDirectory=null, values={}], properties={}}
[DEBUG] Creating an app using the following settings: {project_artifactId=code-with-quarkus, app-config={}, extensions=[], quarkus.create-project.no-buildtool-wrapper=false, project_groupId=org.acme, quarkus.create-project.no-code=false, project_version=1.0.0-SNAPSHOT, source_type=JAVA, java_target=11}
[DEBUG] Resolving Quarkus extension catalog for TargetQuarkusVersionGroup{stream=null, platformBom=null}
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/registry/quarkus-registry-descriptor/1.0-SNAPSHOT/maven-metadata.xml
Downloaded from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/registry/quarkus-registry-descriptor/1.0-SNAPSHOT/maven-metadata.xml (2.5 kB at 48 kB/s)
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/registry/quarkus-registry-descriptor/1.0-SNAPSHOT/quarkus-registry-descriptor-1.0-20220419.075311-1.json
Downloaded from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/registry/quarkus-registry-descriptor/1.0-SNAPSHOT/quarkus-registry-descriptor-1.0-20220419.075311-1.json (512 B at 57 kB/s)
[DEBUG] Resolved registry descriptor io.quarkus.registry:quarkus-registry-descriptor:json:1.0-SNAPSHOT from registry.quarkus.io
[DEBUG] Resolving platform catalog io.quarkus.registry:quarkus-platforms:json:1.0-SNAPSHOT
Looking for the newly published extensions in registry.quarkus.io
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/registry/quarkus-platforms/1.0-SNAPSHOT/maven-metadata.xml
Downloaded from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/registry/quarkus-platforms/1.0-SNAPSHOT/maven-metadata.xml (2.5 kB at 357 kB/s)
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/registry/quarkus-platforms/1.0-SNAPSHOT/quarkus-platforms-1.0-20220419.075311-1.json
Downloaded from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/registry/quarkus-platforms/1.0-SNAPSHOT/quarkus-platforms-1.0-20220419.075311-1.json (6.8 kB at 846 kB/s)
[DEBUG] Resolving platform extension catalog io.quarkus.platform:quarkus-bom-quarkus-platform-descriptor:json:2.8.0.Final:2.8.0.Final
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/2.8.0.Final/quarkus-bom-quarkus-platform-descriptor-2.8.0.Final-2.8.0.Final.json
Downloaded from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/2.8.0.Final/quarkus-bom-quarkus-platform-descriptor-2.8.0.Final-2.8.0.Final.json (196 kB at 2.6 MB/s)
[DEBUG] Resolving platform extension catalog io.quarkus.platform:quarkus-optaplanner-bom-quarkus-platform-descriptor:json:2.8.0.Final:2.8.0.Final
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-optaplanner-bom-quarkus-platform-descriptor/2.8.0.Final/quarkus-optaplanner-bom-quarkus-platform-descriptor-2.8.0.Final-2.8.0.Final.json
[ERROR] ❗  Unable to create project: Failed to resolve extension catalog of io.quarkus.platform:quarkus-optaplanner-bom:pom:2.8.0.Final from Maven repository registry.quarkus.io (http://localhost:8080/maven)

It looks like the registry is able to provide the quarkus-bom but not other members that are present in the platform release.

aloubyansky commented 2 years ago

Comparing the sizes of the quarkus-bom provided by the registry and the one in Maven Central, there seem to be a slight difference. The one provided by the registry: 195623 The one in Maven Central: 195781

I haven't compared the actual content though.

gastaldi commented 2 years ago

codestarts-artifacts, project and even current-stream-id could be considered redundant in this content.

Maybe, but since this is read directly from the database, I think I'd rather leave like this

gastaldi commented 2 years ago

To be able to test the CLI, I had to apply the following changes:

registry.quarkus.io:

diff --git a/src/main/java/io/quarkus/registry/app/maven/RegistryDescriptorContentProvider.java b/src/main/java/io/quarkus/registry/app/maven/RegistryDescriptorContentProvider.java
index cbce744..ee59ce0 100644
--- a/src/main/java/io/quarkus/registry/app/maven/RegistryDescriptorContentProvider.java
+++ b/src/main/java/io/quarkus/registry/app/maven/RegistryDescriptorContentProvider.java
@@ -72,6 +72,7 @@ public class RegistryDescriptorContentProvider implements ArtifactContentProvide
         platformsConfig.setArtifact(new ArtifactCoords(mavenConfig.getRegistryGroupId(),
                 Constants.DEFAULT_REGISTRY_PLATFORMS_CATALOG_ARTIFACT_ID, null, Constants.JSON,
                 Constants.DEFAULT_REGISTRY_ARTIFACT_VERSION));
+        platformsConfig.setExtensionCatalogsIncluded(true);
         qer.setPlatforms(platformsConfig.build());

I have introduced a QUARKUS_REGISTRY_PLATFORM_EXTENSION_CATALOG_INCLUDED env var to enable that. I'll make sure to include it in the production deployment once this percolates in

gastaldi commented 2 years ago

Comparing the sizes of the quarkus-bom provided by the registry and the one in Maven Central, there seem to be a slight difference. The one provided by the registry: 195623 The one in Maven Central: 195781

I haven't compared the actual content though.

There is one small inconsistency, which I fixed in https://github.com/quarkusio/quarkus/pull/25111

The JSON is identical when 2.8.1.Final is the last one included in the registry. Because other platforms are included later during the test, the extension name may be changed (because of fc98ccf8bd4fb55bf659876930951d7e7ea4d17b)

gastaldi commented 2 years ago

@aloubyansky let me know if there is anything missing before we merge this PR

aloubyansky commented 2 years ago

Thanks @gastaldi I'd really like to be able to test it with a client that supports snapshot platform overrides. I'll try to find time this and/or next week for this. But if you prefer to merge it before that, I won't mind.

gastaldi commented 2 years ago

No worries, I'd rather wait until the client is ready before merging then 👍🏻

aloubyansky commented 2 years ago

@gastaldi with the latest changes the quarkus-bom descriptor is resolvable from the registry but not the other members. Here is a log snippet

[DEBUG] Resolving platform extension catalog io.quarkus.platform:quarkus-bom-quarkus-platform-descriptor:json:2.9.2.Final:1.0-SNAPSHOT
Looking for the newly published extensions in registry.quarkus.io
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/maven-metadata.xml
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.9.2.Final.json
Downloaded from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.9.2.Final.json (198 kB at 4.7 MB/s)
[DEBUG] Resolving platform extension catalog io.quarkus.platform:quarkus-optaplanner-bom-quarkus-platform-descriptor:json:2.9.2.Final:1.0-SNAPSHOT
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-optaplanner-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/maven-metadata.xml
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-optaplanner-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-optaplanner-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.9.2.Final.json
[DEBUG] Failed to resolve extension catalog of io.quarkus.platform:quarkus-optaplanner-bom:pom:2.9.2.Final from Maven repository registry.quarkus.io (http://localhost:8080/maven)
[DEBUG] Resolving platform extension catalog io.quarkus.platform:quarkus-optaplanner-bom-quarkus-platform-descriptor:json:2.9.2.Final:2.9.2.Final
Downloading from central: https://repo.maven.apache.org/maven2/io/quarkus/platform/quarkus-optaplanner-bom-quarkus-platform-descriptor/2.9.2.Final/quarkus-optaplanner-bom-quarkus-platform-descriptor-2.9.2.Final-2.9.2.Final.json
Downloaded from central: https://repo.maven.apache.org/maven2/io/quarkus/platform/quarkus-optaplanner-bom-quarkus-platform-descriptor/2.9.2.Final/quarkus-optaplanner-bom-quarkus-platform-descriptor-2.9.2.Final-2.9.2.Final.json (5.6 kB at 19 kB/s)

As you can see, the optaplanner-bom descriptor couldn't be resolved and fell back to the Central repo.

gastaldi commented 2 years ago

@aloubyansky that's because the optaplanner-bom platform isn't included in the extension-catalog.

aloubyansky commented 2 years ago

Platform extensions should be registered though. IIUC, currently we registry the quarkus-bom ones and the non-platform / Quarkiverse ones? Not having Camel, Kogito and the rest of the platform doesn't seem fair. AFAIU, there are a couple of approaches:

  1. manually list the member BOMs under platforms in the extension-catalog
  2. pull in all the members from the specified quarkus-bom metadata [1]

It seems like 2. would be better, given that platform members may get enabled/disabled between releases, which would make it more difficult to maintain the extension catalog config.

[1] the metadata at the bottom in https://repo1.maven.org/maven2/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/2.9.2.Final/quarkus-bom-quarkus-platform-descriptor-2.9.2.Final-2.9.2.Final.json

gastaldi commented 2 years ago

@aloubyansky I agree that 2 is better, however, I'm afraid that will require a change in the DB model to support platform nesting (which I was considering as separate platforms in the extension catalog so far).

aloubyansky commented 2 years ago

I guess you could check the metadata and register the members as individual platforms for this to work. Although going forward we should adjust the data model to reflect the reality. We are not going back to a single BOM, that's for sure. The granularity of members and even their versioning could possibly change but the platform is going to remain a composition of members.

aloubyansky commented 2 years ago

Other members, besides the quarkus-bom, still aren't supported, afaics.

Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.11.0.CR1.json
Downloaded from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.11.0.CR1.json (197 kB at 1.8 MB/s)
Downloading from registry.quarkus.io: http://localhost:8080/maven/io/quarkus/platform/quarkus-optaplanner-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-optaplanner-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.11.0.CR1.json
Downloading from central: https://repo.maven.apache.org/maven2/io/quarkus/platform/quarkus-optaplanner-bom-quarkus-platform-descriptor/2.11.0.CR1/quarkus-optaplanner-bom-quarkus-platform-descriptor-2.11.0.CR1-2.11.0.CR1.json
Downloaded from central: https://repo.maven.apache.org/maven2/io/quarkus/platform/quarkus-optaplanner-bom-quarkus-platform-descriptor/2.11.0.CR1/quarkus-optaplanner-bom-quarkus-platform-descriptor-2.11.0.CR1-2.11.0.CR1.json (5.6 kB at 6.0 kB/s)

The general issue is actually not related to the feature we are introducing here. It has to be fixed either way.

gastaldi commented 2 years ago

@aloubyansky I noticed that the metadata.members can also contain a reference to itself (for example, the quarkus-debezium-bom-quarkus-platform-descriptor JSON has an entry to io.quarkus.platform:quarkus-debezium-bom-quarkus-platform-descriptor:2.9.2.Final:json:2.9.2.Final).

Is that intentional?

aloubyansky commented 2 years ago

Yes, it's a complete platform release info.

gastaldi commented 2 years ago

Yes, it's a complete platform release info.

Ok, so if I get this right, the metadata.platform-release content is the same in the JSON content from each member of the metadata.platform-release.members?

gastaldi commented 2 years ago

Also, I couldn't find a parent reference for each member (eg. something that indicates that the Debezium platform belongs to the quarkus-platform).

Is it because only the parent platform knows about their children (because the children may be referenced on other platforms)?

aloubyansky commented 2 years ago

The original impl allowed each member to keep its own versioning, instead of a common version for all the members. In fact, this could still be an option in the future. For example, when Camel needs to upgrade in the platform, only the Camel platform BOM version would change (and those member BOMs that happen to be affected by the Camel upgrade), the rest of the member BOMs would stay at their previous versions. As a consequence the same member BOM/descriptor version could appear in multiple releases. However, a given descriptor version would include only the release info it first appeared in. Which is OK. That's the reason there is no parent reference from a member to the platform release.

Terminology-wise, member BOMs aren't platforms but more like fragments of a platform.

gastaldi commented 2 years ago

@aloubyansky this PR should now work. The changes in https://github.com/quarkusio/jbang-catalog/pull/25 are needed after this PR is merged to publish platform members as separate platforms in the registry.

aloubyansky commented 2 years ago

Here is what I noticed comparing the current production platform catalog

{
  "platforms" : [ {
    "platform-key" : "io.quarkus.platform",
    "name" : "Quarkus Community Platform",
    "streams" : [ {
      "id" : "2.10",
      "releases" : [ {
        "version" : "2.10.2.Final",
        "member-boms" : [ "io.quarkus.platform:quarkus-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-optaplanner-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-kogito-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-qpid-jms-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-cassandra-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-amazon-services-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-camel-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-hazelcast-client::pom:2.10.2.Final", "io.quarkus.platform:quarkus-debezium-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-blaze-persistence-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-google-cloud-services-bom::pom:2.10.2.Final" ],
        "quarkus-core-version" : "2.10.2.Final"
      } ]
    }, {

and the one returned from this branch

{
  "platforms" : [ {
    "platform-key" : "io.quarkus.platform",
    "name" : "Quarkus Community Platform",
    "streams" : [ {
      "id" : "2.10",
      "releases" : [ {
        "version" : "2.10.2.Final",
        "member-boms" : [ "io.quarkus.platform:quarkus-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-optaplanner-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-kogito-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-qpid-jms-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-cassandra-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-amazon-services-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-camel-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-hazelcast-client::pom:2.10.2.Final", "io.quarkus.platform:quarkus-debezium-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-blaze-persistence-bom::pom:2.10.2.Final", "io.quarkus.platform:quarkus-google-cloud-services-bom::pom:2.10.2.Final" ],
        "quarkus-core-version" : "2.10.2.Final",
        "codestarts-artifacts" : [ "io.quarkus:quarkus-project-core-extension-codestarts::jar:2.10.2.Final" ],
        "last-bom-update" : "io.quarkus.platform:quarkus-bom:2.10.2.Final",
        "platform-release" : {
          "platform-key" : "io.quarkus.platform",
          "stream" : "2.10",
          "version" : "2.10.2.Final",
          "members" : [ "io.quarkus.platform:quarkus-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-optaplanner-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-kogito-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-qpid-jms-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-cassandra-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-amazon-services-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-camel-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-hazelcast-client-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-debezium-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-blaze-persistence-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final", "io.quarkus.platform:quarkus-google-cloud-services-bom-quarkus-platform-descriptor:2.10.2.Final:json:2.10.2.Final" ]
        },
        "project" : {
          "properties" : {
            "doc-root" : "https://quarkus.io",
            "rest-assured-version" : "${rest-assured.version}",
            "compiler-plugin-version" : "3.8.1",
            "surefire-plugin-version" : "3.0.0-M7",
            "kotlin-version" : "1.6.21",
            "scala-version" : "2.13.8",
            "scala-plugin-version" : "4.6.3",
            "quarkus-core-version" : "2.10.2.Final",
            "maven-plugin-groupId" : "io.quarkus.platform",
            "maven-plugin-artifactId" : "quarkus-maven-plugin",
            "maven-plugin-version" : "2.10.2.Final",
            "gradle-plugin-id" : "io.quarkus",
            "gradle-plugin-version" : "2.10.2.Final",
            "supported-maven-versions" : "[3.6.2,)",
            "proposed-maven-version" : "3.8.4",
            "maven-wrapper-version" : "3.1.0",
            "gradle-wrapper-version" : "7.4.2"
          }
        }
      } ]

Looks like there is platform metadata mixup from the actual platform catalog and the extension catalog.

gastaldi commented 2 years ago

@aloubyansky the platform release metadata is no longer returned to the client in https://github.com/quarkusio/registry.quarkus.io/pull/92/commits/669fe5ecdb25b80b4ba2b98c0dbc52521610ce50.

The Extension catalog metadata is stored in the metadata column in the platform_release table, because it's necessary for the descriptor JSON. eg.http://localhost:8080/maven/io/quarkus/platform/quarkus-debezium-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-debezium-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.10.2.Final.json

aloubyansky commented 1 year ago

@gastaldi does any of the below URLs work for you? http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/2.16.1.Final/quarkus-bom-quarkus-platform-descriptor-2.16.1.Final-2.16.1.Final.json http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-bom-quarkus-platform-descriptor-2.16.1.Final-1.0-SNAPSHOT.json They don't for me.

I tried it with

[aloubyansky@fedora resources]$ git diff
diff --git a/src/main/resources/application.properties b/src/main/resources/application.properties
index fcc5939..e65a7a9 100644
--- a/src/main/resources/application.properties
+++ b/src/main/resources/application.properties
@@ -57,4 +57,4 @@ quarkus.log.sentry.release=${quarkus.application.version}

 # Enable PlatformCatalogContentProvider endpoint during tests
-%test.quarkus.registry.platform.extension-catalog-included=true
+quarkus.registry.platform.extension-catalog-included=true
aloubyansky commented 1 year ago

That's a registry descriptor. The URLs are pasted are for a platform member.

gastaldi commented 1 year ago

@aloubyansky version needs to be 1.0-SNAPSHOT. Try with: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.16.1.Final.json

aloubyansky commented 1 year ago

Ah, ok. The original versions are not supported?

gastaldi commented 1 year ago

No, because the idea was that the JSON contents may change over time. But we can change that I think

aloubyansky commented 1 year ago

Yes, we considered both. Ok, thanks.

gastaldi commented 1 year ago

@aloubyansky try again, I implemented it in b370ea1 (#92)

aloubyansky commented 1 year ago

@aloubyansky version needs to be 1.0-SNAPSHOT. Try with: http://localhost:8080/maven/io/quarkus/platform/quarkus-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.16.1.Final.json

Still doesn't work for me. I guess I'm missing something. The descriptor says

  "platforms" : {
    "artifact" : "io.quarkus.registry:quarkus-platforms::json:1.0-SNAPSHOT",
    "extension-catalogs-included" : true
  },
gastaldi commented 1 year ago

@aloubyansky Mine works with the same descriptor. After starting the server, have you imported the data from the quarkus-extension-catalog repository?

aloubyansky commented 1 year ago

I thought I did, will try again.

aloubyansky commented 1 year ago

I can see it works but only for the currently promoted releases. For example it works for 2.16.2.Final and 2.15.3.Final. But not for 2.16.1.Final and 2.15.2.Final. Does it mean the previous versions aren't present in the registry database or simply not returned? We'd want all of them to be available.

aloubyansky commented 1 year ago

I'd expect the following two to work as well: http://localhost:8080/maven/io/quarkus/platform/quarkus-camel-bom-quarkus-platform-descriptor/2.15.3.Final/quarkus-camel-bom-quarkus-platform-descriptor-2.15.3.Final-2.15.3.Final.json http://localhost:8080/maven/io/quarkus/platform/quarkus-camel-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-camel-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.15.3.Final.json

Are Camel and others registered?

gastaldi commented 1 year ago

I can see it works but only for the currently promoted releases. For example it works for 2.16.2.Final and 2.15.3.Final. But not for 2.16.1.Final and 2.15.2.Final. Does it mean the previous versions aren't present in the registry database or simply not returned? We'd want all of them to be available.

That's because the descriptor in the quarkus-extension-catalog contains only the latest versions per major.minor, to avoid increasing indefinitely

gastaldi commented 1 year ago

I'd expect the following two to work as well: http://localhost:8080/maven/io/quarkus/platform/quarkus-camel-bom-quarkus-platform-descriptor/2.15.3.Final/quarkus-camel-bom-quarkus-platform-descriptor-2.15.3.Final-2.15.3.Final.json http://localhost:8080/maven/io/quarkus/platform/quarkus-camel-bom-quarkus-platform-descriptor/1.0-SNAPSHOT/quarkus-camel-bom-quarkus-platform-descriptor-1.0-SNAPSHOT-2.15.3.Final.json

Are Camel and others registered?

That would require running the catalog_publish from https://github.com/quarkusio/jbang-catalog/pull/25, I'll rebase and have a look tomorrow morning

aloubyansky commented 1 year ago

Good, thanks. Otherwise, it seems to work. Are there any DB indexes we want to review/adjust with this change? Any potential performance issues?

holly-cummins commented 1 year ago

Exciting!

gastaldi commented 1 year ago

@holly-cummins any comments before we merge this? Say it now or forever hold your peace 😄

holly-cummins commented 1 year ago

Looking forward to trying it out, @gastaldi . :)