Some facts missing from foreman_facts when importing inventory (Satellite/Foreman)

[aquilinux]

ISSUE TYPE

• Bug

SUMMARY

• When running Inventory Sync with Satellite/Foreman plugin via foreman.py script some foreman_facts are missing on some hosts.

ENVIRONMENT

• AWX version: 7.0.0
• AWX install method: awx-rpm
• Ansible version: 2.8.6
• Operating System: CentOS 7.7.1908
• Web Browser: Chromium
• Foreman verision: 1.22.0

STEPS TO REPRODUCE

1) Run inventory sync via Satellite/Foreman plugin

2) Task ends successfully.

3) Check foreman fact on foreman:

4) Check foreman_fact on awx:

EXPECTED RESULTS

• All foreman_facts should be imported on AWX. Like this one:

ACTUAL RESULTS

• See 4) in steps to reproduce

ADDITIONAL INFORMATION

Same facts are present on Foreman for both hosts but some facts (like those in the picture) are missing in AWX only for some hosts).

Thanks!

[aquilinux]

I've noticed that all the Plesk on CentOS hosts we have are affected by this problem. This doesn't happen on plain CentOS hosts.

.. but shouldn't the foreman facts import be "host/node indipendent" ?

[wenottingham]

Please try the devel branch - we merged an updated foreman inventory script there.

[aquilinux]

thanks, i'll give it a try

[aquilinux]

Unfortunately i get the same result. Some custom facts present in foreman are not present in awx after sync with foreman.py plugin

[ryanpetrello]

@sjha4 @ares you have any ideas?

[ares]

This is still using the old way of syncing data. @aquilinux the foreman.py has been reworked and will change the way we fetch data, but you'll need Foreman 1.24 (GA imminent) and I suppose at this point nightly version of awx. Is it possible for you to upgrade? If not, to confirm it's foreman.py issue, do you see this fact in Foreman? It may be the old bug in ansible that we delete ansible facts - https://github.com/ansible/ansible/pull/51546

[aquilinux]

Thanks @ares i definetly see facts in foreman that are not imported to AWX host. In my screenshots is just one but those are many. I tried deleting the AWX host and re-running sync but nothing changed. Update options are flagged in source configuration as well. As far as i can see this https://github.com/ansible/ansible/pull/51546 is not my case. Some facts are missing from some hosts, i'd say they are missing on CentOS hosts (but not on all of the CentOS i have):

All hosts should have instead:

On foreman i have all facts for all hosts.

As for upgrading Foreman, is not that much. i can do it. But for AWX, i depend on @MrMEEE's builds at https://github.com/MrMEEE/awx-build

[cmasopust]

Same issue here with Foreman 2.1.2 and AWX 14.1.0, lots of facts are missing

[ryanpetrello]

@ares it sounds like we've still got some reports of this from people using the new collection - you have any ideas on what we could check next?

[ares]

I believe the problem is, when the collection was created, wrong froman.py script was taken (contrib vs plugin). Meaning all fixes we had simply vanished. @sjha4 is working on getting the new way of generating inventory back to the foreman.py that now lives in the collection.

[aquilinux]

After upgrading to Foreman 1.24 and AWX 17.0.0.1 i observe the same behavior. @ares can you better expain your previous comment? Any chance we get this working?

Thanks!

[ganniterix]

This is not a bug. You need to set the batch size. You're getting the default number of facts (250) as per documentation.

https://docs.ansible.com/ansible/latest/collections/theforeman/foreman/foreman_inventory.html#parameter-batch_size.

    def _get_facts_by_id(self, hid):
        url = "%s/api/v2/hosts/%s/facts" % (self.foreman_url, hid)
        return self._get_json(url)

and then ...

    def _get_json(self, url, ignore_errors=None, params=None):

        if not self.use_cache or url not in self._cache.get(self.cache_key, {}):
            if self.cache_key not in self._cache:
                self._cache[self.cache_key] = {url: ''}

            results = []
            s = self._get_session()
            if params is None:
                params = {}
            params['page'] = 1
            params['per_page'] = self.get_option('batch_size')

[aquilinux]

Hi Jean Paul, awesome! So i just need a file named xyz.foreman.yml containing:

batch_size: 800

along with foreman.py, right?

On Tue, Oct 5, 2021 at 11:34 PM Jean Paul Gatt @.***> wrote:

This is not a bug. You need to set the batch size. You're getting the default number of facts (250) as per documentation.

https://docs.ansible.com/ansible/latest/collections/theforeman/foreman/foreman_inventory.html#parameter-batch_size .

def _get_facts_by_id(self, hid):
    url = "%s/api/v2/hosts/%s/facts" % (self.foreman_url, hid)
    return self._get_json(url)

and then ...

def _get_json(self, url, ignore_errors=None, params=None):

    if not self.use_cache or url not in self._cache.get(self.cache_key, {}):
        if self.cache_key not in self._cache:
            self._cache[self.cache_key] = {url: ''}

        results = []
        s = self._get_session()
        if params is None:
            params = {}
        params['page'] = 1
        params['per_page'] = self.get_option('batch_size')

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ansible/awx/issues/5428#issuecomment-934878119, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGPM5MSKOQ564ZXPXIABMDUFNVMXANCNFSM4JUZCUKA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- "Madness, like small fish, runs in hosts, in vast numbers of instances."

Nessuno mi pettina bene come il vento.

[ganniterix]

I don't even have foreman.py in the folder. I am importing the ansible collection.

ansible-galaxy collection install theforeman.foreman

And then my config file (foreman.yaml) looks something like this.

---
cache: true
cache_connection: /tmp/foreman-dynamic-cache
cache_plugin: jsonfile
cache_plugin_prefix: ansible_facts
cache_timeout: 86400
plugin: theforeman.foreman.foreman
strict: false
validate_certs: false

url: "https://myurl.satellite.local"
user: "admin"
password: "xxxxxx"

want_facts: True
want_params: True
want_organization: True
want_smart_proxies: True
want_hostcollections: True
batch_size: 12345

host_filters: "xxxx"
keyed_groups:
- key: foreman_facts.abcde
  prefix: abcde
- key: foreman_facts.fghij
  prefix: fghij

This is how I call it from ansible cli. Inside awx I set the credential aside, and remove the cache stuff.

[aquilinux]

Thank you for your feedback, got it working here:

[image: image.png] and yes, it did the trick as i see facts being populated on hosts that were problematic. a bit misleading, though, since batch_size should set the batch PER page.. but looking at the code awx in getting only page 1, so.. it works. just a side note, using a batch of 800 to retrieve 800 nodes sometimes triggers 504 Gateway timeout on foreman's API.

By the way, your hint was correct. You solved me a 2yrs issue :D 5 stars!

On Wed, Oct 6, 2021 at 5:07 PM Jean Paul Gatt @.***> wrote:

I don't even have foreman.py in the folder. I am importing the ansible collection.

ansible-galaxy collection install theforeman.foreman

And then my config file (foreman.yaml) looks something like this.

---

cache: true cache_connection: /tmp/foreman-dynamic-cache cache_plugin: jsonfile cache_plugin_prefix: ansible_facts cache_timeout: 86400 plugin: theforeman.foreman.foreman strict: false validate_certs: false

url: "https://myurl.satellite.local" user: "admin" password: "xxxxxx"

want_facts: True want_params: True want_organization: True want_smart_proxies: True want_hostcollections: True batch_size: 12345

host_filters: "xxxx" keyed_groups:

• key: foreman_facts.abcde prefix: abcde
• key: foreman_facts.fghij prefix: fghij

This is how I call it from ansible cli. Inside awx I set the credential aside, and remove the cache stuff.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ansible/awx/issues/5428#issuecomment-936461215, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGPM5KEJHVV6N6ZK74G2BTUFRQ45ANCNFSM4JUZCUKA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- "Madness, like small fish, runs in hosts, in vast numbers of instances."

Nessuno mi pettina bene come il vento.

[ganniterix]

You might want to have a look at this ...

https://access.redhat.com/node/4224211/40/0/17083701

although in later versions of Satellite, the document is no longer present.

https://access.redhat.com/documentation/en-us/red_hat_satellite/6.9