search

RedHatInsights / insights-core

Insights Core is a data collection and processing framework used by Red Hat Insights
https://cloud.redhat.com/insights
Apache License 2.0
153 stars 183 forks source link

Environment variable INSIGHTS_FILTERS_ENABLED no longer works #4103

Closed bfahr closed 6 months ago

bfahr commented 6 months ago

The environment variable INSIGHTS_FILTERS_ENABLED can be used to turn off filtering when processing archives. Setting this env var to False turns off the filter checking and filtering of specs that are included in archives so that applications processing the archive using custom rules will see all of the data collected. After insights-core-3.3.21 release this env var is no longer working correctly. If it is set to False then filtered specs will not be loaded and if set to True they will be loaded but the default filters will be applied meaning that only data such as column headings will be present in the data. The True setting works as expected, but the False setting should enable loading all of the data from filtered specs in archives.

xiangce commented 6 months ago

Thanks, @bfahr, for reporting this issue, the root cause is that the original validation of the filterable specs didn't care about this environment https://github.com/RedHatInsights/insights-core/blob/de92cb5b6a156c5b451d964be8fc8888cec7aad0/insights/core/spec_factory.py#L195-L198

And in #4082, the getting filters are enabled for SerializedArchiveContext (insights-archive from core collection).

So, in runtime against the archive collected by core collection, the process of filterable specs will be terminated due to the above validation.

I raised #4104 to fix it, please have a look as well.

Once it gets approved, I will release it this Thu.

Sorry for bringing this trouble to you.

xiangce commented 6 months ago

Without the #4104, the result is:

~/work/insights/insights-core (master) ❯ INSIGHTS_FILTERS_ENABLED=False insights-cat insights.specs.Specs.insights_client_conf /tmp/insights-20240412012625
Traceback (most recent call last):
  File "/home/liuxc/work/insights/insights-core/insights/core/plugins.py", line 101, in invoke
    return self.component(broker)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuxc/work/insights/insights-core/insights/core/spec_factory.py", line 654, in __call__
    return self.kind(ctx.locate_path(self.path), root=ctx.root,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/liuxc/work/insights/insights-core/insights/core/spec_factory.py", line 180, in __init__
    self.validate()
  File "/home/liuxc/work/insights/insights-core/insights/core/spec_factory.py", line 198, in validate
    raise NoFilterException("Skipping %s due to no filters." % dr.get_name(self.ds))
insights.core.exceptions.NoFilterException: Skipping insights.specs.default.DefaultSpecs.insights_client_conf due to no filters.

Missing Dependencies:
    At Least One Of:
        insights.specs.default.DefaultSpecs.insights_client_conf

With the #4104 being patched, the result looks good

~/work/insights/insights-core (fix_4103) ❯ INSIGHTS_FILTERS_ENABLED=False insights-cat insights.specs.Specs.insights_client_conf /tmp/insights-20240412012625
SerializedOutputProvider("'/tmp/insights-20240412012625/data/etc/insights-client/insights-client.conf'")
[insights-client]
#authmethod=BASIC
# username to use when authmethod is BASIC
# password ******** use when authmethod is BASIC
#auto_update=True
# core_collect=True
core_collect=False
ros_collect=True
test=abc