search

furlongm / patchman

Patchman is a Linux Patch Status Monitoring System
http://patchman.openbytes.ie
GNU General Public License v3.0
362 stars 78 forks source link

IndexError: list index out of range, When processing a report. #589

Closed Ezio1001 closed 5 months ago

Ezio1001 commented 6 months ago

I'm currently having an issue with processing a report specifically from a Rocky 8.7 host, when I run patchman -a I get this output:

root@host:/# patchman -a
Cleaning Reports for all Hosts
example.com
Processing Reports for all Hosts
Reverse DNS check disabled
example.com repos    100%|############################################################################################################################################|Time: 0:00:00
Traceback (most recent call last):   0%|                                                                                                                                           |ETA:  --:--:--
  File "/usr/bin/patchman", line 705, in <module>
    main()
  File "/usr/bin/patchman", line 699, in main
    showhelp = process_args(args)
  File "/usr/bin/patchman", line 659, in process_args
    process_reports(args.host, args.force)
  File "/usr/bin/patchman", line 490, in process_reports
    report.process(find_updates=False)
  File "/usr/lib/python3/dist-packages/reports/models.py", line 168, in process
    process_modules(report=self, host=host)
  File "/usr/lib/python3/dist-packages/reports/utils.py", line 80, in process_modules
    module = process_module(module_str)
  File "/usr/lib/python3/dist-packages/reports/utils.py", line 322, in process_module
    m_version = module_str[2]
IndexError: list index out of range

I dropped the mySQL DB and started all over again but this time I started by processing a report form an ubuntu server and it worked fine, then I added this Rocky 8.7 server and got this issue again.

Note: the Rocky 8.7 server in question does have enabled modules.

TW1920 commented 5 months ago

I have the same problem on Debian 12.5 (Patchman Core) - but only after added an additional host based on AlmaLinux (Client) - as far as I saw the report is much bigger than of our other servers (Clients). Maybe there's the reason, would explain the error message.

TW1920 commented 5 months ago

@Ezio1001 did you found any way to fix it?

@furlongm Any idea how to fix this? Would be great :)

Thank you for help

Ezio1001 commented 5 months ago

@TW1920 Sorry for not getting back any sooner. I actually found out how to make it work, or found a workaround. The issue as I deduced is with modules, I think the code needs some modification to be able to process enabled modules. I tried going through the code to attempt fixing the issue but it didn't work So what I did is I went to the patchman-client bash script you can find it at /usr/sbin/patchman-client If you're running the service over an Ubuntu server, extract the bash script and go to line 592 and change the {tmpfile_mod} to anything else, in my case I used {tmpfile_bug}. This workaround made patchman-client skip sending modules and I stopped getting the error. I know this workaround might look a bit tacky but packages and reboot status is what I really care about in a server atm so I had to do it this way. After that you can run the client as a bash script from whatever directory you extracted it to and you can use the -s flag to direct the report to your main server.

furlongm commented 5 months ago

@TW1920 @Ezio1001 does this issue only happen with Rocky/Alma 8? Not 9?

Do you have a list of available and enabled modules?

dnf module list and dnf module list --enabled

furlongm commented 5 months ago

Can you try commit 0404e57c8fc61ce33a88c3ccd7b672a08cdf1083 in PR #594 ? This should fix it but I want to confirm first.

Ezio1001 commented 4 months ago

@furlongm Good day. Sorry for the late response, I didn't get the Email notifications for some weird reason. Anyhow, the state which I'm running at currently is:

  1. After replacing the {tmpfile_mod} workaround things worked fine.
  2. I got some errors later after using the updated script which led me to revert to using the script from before two updates ago.
  3. The error I got after using the latest script was: 
    Repository 519 : Rocky / Red Hat Enterprise Linux 9 - atomic x86_64
No usable mirror found at https://updates.atomicorp.com/channels/mirrorlist/atomic/rocky-9-x86_64
Downloading repo info:             100%|############################################################################################################################################|Time: 0:00:00
Found yum rpm repo - https://updates.atomicorp.com/channels/atomic/rocky/9/x86_64/repodata/repomd.xml
Downloading package info:          100%|############################################################################################################################################|Time: 0:00:00
Traceback (most recent call last):
File "/usr/bin/patchman", line 705, in <module>
main()
File "/usr/bin/patchman", line 699, in main
showhelp = process_args(args)
File "/usr/bin/patchman", line 665, in process_args
refresh_repos(args.repo, args.force)
File "/usr/bin/patchman", line 122, in refresh_repos
repo.refresh(force)
File "/usr/lib/python3/dist-packages/repos/models.py", line 86, in refresh
refresh_rpm_repo(self)
File "/usr/lib/python3/dist-packages/repos/utils.py", line 716, in refresh_rpm_repo
refresh_yum_repo(mirror, data, mirror_url, ts)
File "/usr/lib/python3/dist-packages/repos/utils.py", line 582, in refresh_yum_repo
packages = extract_yum_packages(package_data, primary_url)
File "/usr/lib/python3/dist-packages/repos/utils.py", line 369, in extract_yum_packages
plen = int(next(m_context)[1].get('packages'))
File "src/lxml/iterparse.pxi", line 210, in lxml.etree.iterparse.__next__
File "src/lxml/iterparse.pxi", line 195, in lxml.etree.iterparse.__next__
File "src/lxml/iterparse.pxi", line 230, in lxml.etree.iterparse._read_more_events
File "src/lxml/parser.pxi", line 1376, in lxml.etree._FeedParser.feed
File "src/lxml/parser.pxi", line 606, in lxml.etree._ParserContext._handleParseResult
File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError
File "<string>", line 1
lxml.etree.XMLSyntaxError: Document is empty, line 1, column 1
    • I only mentioned it cause it seemed a new one, different than the other (out of index) ones.
  4. I'm still having some occasional issues with (rocky, centos, alma, and now rhel) repos, which I workedaround by deleting the faulty mirrors which usually are bad mirrors.
Ezio1001 commented 4 months ago

@furlongm I may suggest having a counter that after a set number of retries it starts ignoring the faulty mirrors automatically. I don't know If it already exists, but I did notice that there already was a (fail counter) for mirrors, so I assumed that can be utilized in a way to help with this matter. At the moment I have to go into each repo and delete faulty mirrors manually one by one. Also I wanted to ask about what (Disabling a repo) actually does because I tried it and it doesn't seem to ignore the repo when I'm running patchman -a (refreshing, processing, and doing all that cool stuff)

litebito commented 2 months ago

Hi,

I've the same error with the Atomicorp repo since I added it to one of my hosts

Repository 122 : Rocky / Red Hat Enterprise Linux 9 - atomic x86_64
No usable mirror found at https://updates.atomicorp.com/channels/mirrorlist/atomic/rocky-9-x86_64
Downloading repo info:             100%|#####################################################################################################################################################################|Time:  0:00:00
Found yum rpm repo - https://updates.atomicorp.com/channels/atomic/rocky/9/x86_64/repodata/repomd.xml
Downloading package info:          100%|#####################################################################################################################################################################|Time:  0:00:00
Traceback (most recent call last):
  File "/bin/patchman", line 705, in <module>
    main()
  File "/bin/patchman", line 699, in main
    showhelp = process_args(args)
  File "/bin/patchman", line 665, in process_args
    refresh_repos(args.repo, args.force)
  File "/bin/patchman", line 122, in refresh_repos
    repo.refresh(force)
  File "/usr/lib/python3.9/site-packages/repos/models.py", line 86, in refresh
    refresh_rpm_repo(self)
  File "/usr/lib/python3.9/site-packages/repos/utils.py", line 716, in refresh_rpm_repo
    refresh_yum_repo(mirror, data, mirror_url, ts)
  File "/usr/lib/python3.9/site-packages/repos/utils.py", line 582, in refresh_yum_repo
    packages = extract_yum_packages(package_data, primary_url)
  File "/usr/lib/python3.9/site-packages/repos/utils.py", line 369, in extract_yum_packages
    plen = int(next(m_context)[1].get('packages'))
  File "src/lxml/iterparse.pxi", line 209, in lxml.etree.iterparse.__next__
  File "src/lxml/iterparse.pxi", line 194, in lxml.etree.iterparse.__next__
  File "src/lxml/iterparse.pxi", line 229, in lxml.etree.iterparse._read_more_events
  File "src/lxml/parser.pxi", line 1387, in lxml.etree._FeedParser.feed
  File "src/lxml/parser.pxi", line 609, in lxml.etree._ParserContext._handleParseResult
  File "src/lxml/parser.pxi", line 618, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 728, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 657, in lxml.etree._raiseParseError
  File "<string>", line 1
lxml.etree.XMLSyntaxError: Document is empty, line 1, column 1

No issues with any other repo so far.... I guess there is something not right with the atomic repo?