analysis doesn't always fail when it should

[djdanielsson]

when analysis runs there is some cases that if an actor fails it creates the report but shows an error, the task should fail when this happens

[swapdisk]

@djdanielsson, to help with understanding the case for changing the behavior as proposed, can you share an example of an actor failure and the report showing the error?

I'm thinking if Leapp generates a complete pre-upgrade report, the task has done its job even though there is an error listed in the report. Maybe just output a warning about the error being reported similar to what the "Preupgrade analysis report is done" task of roles/analysis/handlers/main.yml is doing already in case of an inhibitor being reported?

[djdanielsson]

If an error occurred like what we were seeing it would mean leapp wasn't really finishing its analysis, IMO it should error because it isn't functioning correctly and likely if not guaranteed to cause issues when attempting to upgrade.

[swapdisk]

We have seen where a Leapp pre-upgrade report entry will have "flags": ["failure"]. Is this the same as what you are seeing, too? For example, in this "Excluded target system repositories" report entry...

{
  "title": "Excluded target system repositories",
  "timeStamp": "2023-09-13T21:09:33.215369Z",
  "hostname": "funny-moth.example.com",
  "detail": {
    "remediations": [
      {
        "type": "hint",
        "context": "If some of excluded repositories are still required to be used during the upgrade, execute leapp with the --enablerepo option with the repoid of the repository required to be enabled as an argument (the option can be used multiple times)."
      }
    ]
  },
  "actor": "repositories_blacklist",
  "summary": "The following repositories are not supported by Red Hat and are excluded from the list of repositories used during the upgrade.\n- (long list of repo redacted",
  "audience": "sysadmin",
  "flags": [
    "failure"
  ],
  "key": "1b9132cb2362ae7830e48eee7811be9527747de8",
  "id": "2250e3c15a0999b175fde8cfe54047da78d1594e81dd98d65312fa07facbe3bb",
  "tags": [
    "repository"
  ],
  "severity": "info"
},

I asked the Leapp team about this one. @pirat89 cited the doc here...

The failure Group [originally known as Flag] is recommended to be used when the report is related to a command or other action failure.

But he went on to say that it is kind of confusing to see this with the repositories_blacklist actor and it probably should not really be using the flag. He also said they don't typically use the failure flag with pre-upgrade actors, but rather just with actors in the later phases that only appear in the post-upgrade report.

With this understanding, we decided to ignore failure flags in the pre-upgrade reporting for the project I was working on. If this is the same as what you are seeing, I recommend ignoring it.

However, if the error you are seeing is different and especially if there is evidence that the generated report is incomplete because of it, we certainly should take a closer look.

[swapdisk]

@djdanielsson, is this issue still happening?

[djdanielsson]

I am not sure, I am not actively doing IPU's we can close it and if someone else notices it they can create or reopen this issue.