katello_manifest isn't idempotent

akofink commented 6 years ago

SUMMARY

Manifest upload should only happen once, regardless of the number of times a playbook is run.

ISSUE TYPE

Bug Report

ANSIBLE VERSION

ansible 2.6.2
  config file = /home/akofink/dev/nfs/centos7-hammer-devel/foreman-ansible-modules/ansible.cfg
  configured module search path = [u'/home/akofink/dev/nfs/centos7-hammer-devel/foreman-ansible-modules/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.15 (default, May 16 2018, 17:50:09) [GCC 8.1.1 20180502 (Red Hat 8.1.1-1)]

KATELLO/FOREMAN VERSION

tfm-rubygem-katello-3.7.0-1.el7.noarch
foreman-1.18.1-1.el7.noarch

NAILGUN VERSION

Version: 0.30.2

STEPS TO REPRODUCE

  - name: Upload a manifest
    katello_manifest:
      username: "{{ foreman_username }}"
      password: "{{ foreman_password }}"
      server_url: "{{ foreman_server_url }}"
      verify_ssl: "{{ foreman_verify_ssl }}"
      organization: Default Organization
      manifest_path: "{{ katello_manifest_path }}"
      state: present
  - name: Upload the same manifest again
    katello_manifest:
      username: "{{ foreman_username }}"
      password: "{{ foreman_password }}"
      server_url: "{{ foreman_server_url }}"
      verify_ssl: "{{ foreman_verify_ssl }}"
      organization: Default Organization
      manifest_path: "{{ katello_manifest_path }}"
      state: present

EXPECTED RESULTS

TASK [Upload a manifest] ******************************
changed: [localhost]

TASK [Upload the same manifest again] *****************************
ok: [localhost]

ACTUAL RESULTS

TASK [Upload a manifest] ******************************
changed: [localhost]

TASK [Upload the same manifest again] *****************************
changed: [localhost]

sean797 commented 6 years ago

I had a look at doing this a while ago, couldn't find a decent way to verify if we need to upload another one. IIRC there isn't a version number or anything similar that we could compare.

Any suggestions on how to do this?

akofink commented 6 years ago

I think we might need to modify the API to achieve this. There is the manifest_history api endpoint, but I don't see how we can get (i.e.) the upstreamConsumer info from the manifest to compare to this history output before we upload the manifest:

#<OpenStruct created="2018-08-10T04:07:59+0000", updated="2018-08-10T04:07:59+0000", id="4028f9ff6521eae1016522050634267d", status="SUCCESS", statusMessage="Default_Organization file imported successfully.", fileName="import_0067f831df87bd7f78ec.zip", generatedBy="rhn-engineering-akofink", generatedDate="2018-08-10T04:07:22+0000", upstreamConsumer={"created"=>"2018-08-10T04:07:59+0000", "updated"=>"2018-08-10T04:07:59+0000", "id"=>"4028f9ff6521eae1016522050634267e", "uuid"=>"a7c5b09d-ed99-412c-980c-7be32988ab2d", "name"=>"akofink", "type"=>{"created"=>"2018-08-10T04:07:47+0000", "updated"=>"2018-08-10T04:07:47+0000", "id"=>"4028f9ff6521eae101652204d7390001", "label"=>"satellite", "manifest"=>true}, "ownerId"=>"4028f9ff6521523801652160b2560001", "contentAccessMode"=>nil, "webUrl"=>"access.redhat.com/management/subscription_allocations/", "apiUrl"=>"https://subscription.rhsm.redhat.com/subscription/consumers/"}>

akofink commented 6 years ago

I've filed an issue to support this. https://projects.theforeman.org/issues/24669

sean797 commented 6 years ago

Inside a manifest, a lot of the attributes are the same. Take consumer_export.zip which is inside a manifest the only differences between them are the actual entitlements and entitlement_certificates, which would require us to look at each entitlements and the relevant attributes (quantity, pool_id, ect...) and compare it with Katello. IMO that's too much hard work and could be quite slow.

There could potentially be a better method.. Inside meta.json there is the created field, this seems to be the date/time when the manifest was created - you could create 2 manifests that have the same subs, so this isn't ideal, but it would work 90% of the time. But it doesn't seem like anything from this file (meta.json) is available via the candlepin API, let alone the Katello one.

TL;DR I think we're going to need candlepin changes.


[sokeeffe@sean manifest_tests]$ diff -qbur 1/ 2/
Only in 2/export/entitlement_certificates: 19701381444539885.pem
Only in 1/export/entitlement_certificates: 6062797014006410679.pem
Files 1/export/entitlements/d4811a09d51a4652bbd1bbe2e4efe942.json and 2/export/entitlements/d4811a09d51a4652bbd1bbe2e4efe942.json differ
Files 1/export/meta.json and 2/export/meta.json differ
[sokeeffe@sean manifest_tests]$
[sokeeffe@sean manifest_tests]$ diff <(jq -S . 1/export/meta.json) <(jq -S . 2/export/meta.json)
3c3
<   "created": "2018-08-21T12:03:23.079+0000",
---
>   "created": "2018-08-21T12:06:45.558+0000",
[sokeeffe@sean manifest_tests]$

akofink commented 6 years ago

For the scope of this repo and this issue, submitting the Katello issue should be enough, even if there are required Candlepin changes.

I'm envisioning an API endpoint that is fast (upload is super slow due to repo import, etc.), which takes the manifest zip just like the upload endpoint, but returns whether the manifest is there already or not. I have no plans to try to process the manifest in FAM.

evgeni commented 5 years ago

Actually, the module is semi-idempotent (lol):

https://github.com/theforeman/foreman-ansible-modules/blob/241d49fd27e7f9926373a442768f12cbc2273e4e/plugins/modules/katello_manifest.py#L129-L131

So the task is not marked as changed, but the upload does happen twice.

theforeman / foreman-ansible-modules