Repo creation errors out with Ruby bindings

ianballou commented 6 days ago

Version

pulp-rpm 3.27.1 (client has same version)

      "versions": {
        "deb": "3.2.0",
        "rpm": "3.27.1",
        "core": "3.49.9",
        "file": "3.49.9",
        "ostree": "2.3.0",
        "python": "3.11.1",
        "ansible": "0.21.5",
        "certguard": "3.49.9",
        "container": "2.20.0"
      }

Tested with Katello 4.14.develop

Describe the bug Creating a repository returns a traceback. The repo does get created, but there's an error.

[11] pry(main)> repoconfig = PulpRpmClient::RpmRpmRepository.new(name: 'tes2')
=> #<PulpRpmClient::RpmRpmRepository:0x00007f497e8b0fa0 @autopublish=false, @name="tes2", @package_signing_fingerprint="">
[12] pry(main)> ::Katello::Repository.yum_type.first.backend_service(SmartProxy.pulp_primary).api.repositories_api.create(repoconfig)
2024-06-24T19:54:03 [D|kat|] Calling API: RepositoriesRpmApi.create ...
2024-06-24T19:54:03 [D|kat|] HTTP request body param ~BEGIN~
 | {"name":"tes2","autopublish":false,"package_signing_fingerprint":""}
 | ~END~
 | 
2024-06-24T19:54:03 [D|kat|] HTTP response body ~BEGIN~
 | {"pulp_href":"/pulp/api/v3/repositories/rpm/rpm/01904bd0-096a-7a74-8630-a4ed94672c82/","pulp_created":"2024-06-24T19:54:03.499852Z","pulp_last_updated":"2024-06-24T19:54:03.513474Z","versions_href":"/pulp/api/v3/repositories/rpm/rpm/01904bd0-096a-7a74-8630-a4ed94672c82/versions/","pulp_labels":{},"latest_version_href":"/pulp/api/v3/repositories/rpm/rpm/01904bd0-096a-7a74-8630-a4ed94672c82/versions/0/","name":"tes2","description":null,"retain_repo_versions":null,"remote":null,"autopublish":false,"metadata_signing_service":null,"package_signing_service":null,"package_signing_fingerprint":"","retain_package_versions":0,"checksum_type":null,"metadata_checksum_type":null,"package_checksum_type":null,"gpgcheck":null,"repo_gpgcheck":null,"sqlite_metadata":false,"repo_config":{},"compression_type":null}
 | ~END~
 | 
NameError: uninitialized constant PulpRpmClient::AnyType
from /home/vagrant/foreman/.vendor/ruby/3.0.0/gems/pulp_rpm_client-3.27.1/lib/pulp_rpm_client/models/rpm_rpm_repository_response.rb:491:in `const_get'

To Reproduce Try to create an RPM repo with the Ruby bindings

Expected behavior Repo is created without an error.

mdellweg commented 5 days ago

I suspect the pulpcore version at the time of creating the bindings may have been different from 3.49.9.

ianballou commented 5 days ago

If that's the cause, even more reason for us to start generating the bindings ourselves. I can try pulp-rpm 3.26 for now if that would be preferable.

ianballou commented 4 days ago

I've confirmed that regenerating the bindings myself fixes the issue. In comparing the generated bindings with the bindings on rubygems.org, I'm seeing references to AnyType that seem to be the cause of the problem:

diff -r /home/vagrant/foreman/.vendor/ruby/3.0.0/gems/pulp_rpm_client-3.27.1/lib/pulp_rpm_client/models/rpm_package_response.rb ../pulp-openapi-generator/pulp_rpm-client/lib/pulp_rpm_client/models/rpm_package_response.rb
230,239c230,239
<         :'changelogs' => :'AnyType',
<         :'files' => :'AnyType',
<         :'requires' => :'AnyType',
<         :'provides' => :'AnyType',
<         :'conflicts' => :'AnyType',
<         :'obsoletes' => :'AnyType',
<         :'suggests' => :'AnyType',
<         :'enhances' => :'AnyType',
<         :'recommends' => :'AnyType',
<         :'supplements' => :'AnyType',
---
>         :'changelogs' => :'Object',
>         :'files' => :'Object',
>         :'requires' => :'Object',
>         :'provides' => :'Object',
>         :'conflicts' => :'Object',
>         :'obsoletes' => :'Object',
>         :'suggests' => :'Object',
>         :'enhances' => :'Object',
>         :'recommends' => :'Object',
>         :'supplements' => :'Object',

and

diff -r /home/vagrant/foreman/.vendor/ruby/3.0.0/gems/pulp_rpm_client-3.27.1/lib/pulp_rpm_client/models/rpm_package_environment_response.rb ../pulp-openapi-generator/pulp_rpm-client/lib/pulp_rpm_client/models/rpm_package_environment_response.rb
81,84c81,84
<         :'group_ids' => :'AnyType',
<         :'option_ids' => :'AnyType',
<         :'desc_by_lang' => :'AnyType',
<         :'name_by_lang' => :'AnyType',
---
>         :'group_ids' => :'Object',
>         :'option_ids' => :'Object',
>         :'desc_by_lang' => :'Object',
>         :'name_by_lang' => :'Object',

for examples.

ianballou commented 4 days ago

We (Katello) still need to come up with a solid plan to create our own Ruby bindings. Building the bindings ourselves is easy as a one-off, but we'd need our self-generation process ironed out before we can start doing it reliably. Is there any way this can be handled without us having to monkey-patch things?

dralley commented 3 days ago

Is there any way this can be handled without us having to monkey-patch things?

@ianballou Are you talking about the short-term (handling this specific issue) or the long-term (working on the self-generation process)?

We'd be happy to work with you on getting the long-term process sorted out.

ianballou commented 3 days ago

@dralley I was talking about the short term. For our upcoming Katello 4.14, if we need to generate our own bindings, it would need to be done manually each time. It's not impossible, but there's always the risk of our packaging automation being used to bump the pulp_rpm_client version and having a new version pulled from rubygems.

pedro-psb commented 2 days ago

The problem

After some investigation and experimentation, these are my conclusions:

the upgrade to drf-spectacular 0.27.2 on the Y pulpcore release (3.55.0) caused this regression

the diff between the api.json files is that the regression is missing a type attribute on some fields:


diff --git a/tmp/wrong-api.json b/tmp/right-api.json
index fc0ae2d..5fd2c53 100644
--- a/tmp/wrong-api.json
+++ b/tmp/right-api.json
@@ -9066,6 +9066,7 @@
   "description": "A serializer for Content Copy API.",
   "properties": {
     "config": {

"type": "object", "description": "A JSON document describing sources, destinations, and content to be copied" }, "dependency_solving": { @@ -10104,6 +10105,7 @@ "minimum": 0 }, "repo_config": {
"type": "object", "description": "A JSON document describing config.repo file" },
...

The rest is very similar

The experiment was:

case: AnyTypes on the gem
- with pulpcore from main
- with rpm on main
case: no AnyTypes on the gem
- with pulpcore from main but with the drf-spectacular bump commit dropped via rebase
- with rpm on main

Ideas

I'm not sure what is the way to go, but some general approaches I see are:

try to get to the root cause of the problem
- couldn't find in the drf-spectacular changes any obvious culprit
do some hack on the workflow:
- patch the api.json or gem
- patch pulpcore, possibly only the binding publication part
Revert the bump

mdellweg commented 2 days ago

Actually not specifying the type: object is correct from the openapiv3 specification side. This is a bug that was fixed in spectacular. A JSONField can hold any type, including lists and strings, numbers and booleans. "object" from the JSON Schema Specification however only allows dictionaries. My conclusion is, that the ruby templates in openapi-generator (or at least the ancient version we currently employ) do not handle that valid case properly. But I would still suggest to go into the viewset and add @extend_schema decorators, defining what exactly the plugin expects to see in that json field.

https://github.com/tfranzel/drf-spectacular/issues/1095

dralley commented 1 day ago

@pedro-psb Wow. Great work!

pulp / pulp_rpm

Repo creation errors out with Ruby bindings #3639

The problem

...

The rest is very similar

Ideas