AssertionError when running upd02_get_manifests_from_updates.py directly

[nimrodpar]

When i run (after running python3 upd01_get_list_of_updates.py and creating updates.json):

python3 upd02_get_manifests_from_updates.py

I get:

Processing Windows version 1507
[KB3074683] Downloading update
[KB3074683] ERROR: Failed to process update
[KB3074683]
Traceback (most recent call last):
  File "upd02_get_manifests_from_updates.py", line 172, in <module>
    main()
  File "upd02_get_manifests_from_updates.py", line 162, in main
    get_manifests_from_update(windows_version, update_kb)
  File "upd02_get_manifests_from_updates.py", line 132, in get_manifests_from_update

    download_url, local_dir, local_path = download_update(windows_version, update_kb)
  File "upd02_get_manifests_from_updates.py", line 47, in download_update
    found_updates = search_for_updates(f'{update_kb} {windows_version} x64')
  File "upd02_get_manifests_from_updates.py", line 20, in search_for_updates
    assert '(page 1 of 1)' in html  # we expect only one page of results
AssertionError

What am I doing wrong?

p.s. great work on this, and thanks in advance! 🙏

[m417z]

You got an assertion error because the first update that the script attempted to download, KB3074683, wasn't found in the update catalog. In fact, there are many such updates, as you can see in the published table, and as I also mentioned in the blog post. It seems that updates are removed from the update catalog with time - some updates were available when I first published Winbindex, but are no longer available.

Of course, an assertion error is not the best way for handling this case. I changed the script to print a warning if an update isn't found in the update catalog. Please try running it again. Since I currently use the scripts for downloading new updates, and not for building the database from scratch, you might experience more errors. Please report them if you encounter any, and I'll do my best to address them.

[nimrodpar]

Tried the new version. Now i'm getting:

Traceback (most recent call last):
  File "upd02_get_manifests_from_updates.py", line 190, in <module>
    main()
  File "upd02_get_manifests_from_updates.py", line 169, in main
    get_manifests_from_update(windows_version, update_kb)
  File "upd02_get_manifests_from_updates.py", line 139, in get_manifests_from_update
    download_url, local_dir, local_path = download_update(windows_version, update_kb)
  File "upd02_get_manifests_from_updates.py", line 86, in download_update
    subprocess.run(args, check=True, stdout=None if config.verbose_run else subprocess.DEVNULL)
  File "/usr/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'aria2c'

Which was solved by sudo apt install aria2. Probably wanna mention it's a dependency.

[m417z]

Right. You'll also need (reference):

sudo apt-get install cabextract
pip install orjson requests

And, for accessing VirusTotal via Tor: (reference):

pip install mitmproxy
sudo apt-get install -y tor
pip install git+git://github.com/m417z/pymultitor.git

[nimrodpar]

Installed all the deps, got a:

[KB4032695] ERROR: Failed to process update
[KB4032695]        Command '['aria2c', '-x4', '-o', PosixPath('manifests/1507/KB4032695/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu'), '--allow-overwrite=true', 'http://download.windowsupdate.com/c/msdownload/update/software/updt/2017/06/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu']' returned non-zero exit status 19.
Traceback (most recent call last):
  File "upd02_get_manifests_from_updates.py", line 190, in <module>
    main()
  File "upd02_get_manifests_from_updates.py", line 169, in main
    get_manifests_from_update(windows_version, update_kb)
  File "upd02_get_manifests_from_updates.py", line 139, in get_manifests_from_update
    download_url, local_dir, local_path = download_update(windows_version, update_kb)
  File "upd02_get_manifests_from_updates.py", line 86, in download_update
    subprocess.run(args, check=True, stdout=None if config.verbose_run else subprocess.DEVNULL)
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['aria2c', '-x4', '-o', PosixPath('manifests/1507/KB4032695/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu'), '--allow-overwrite=true', 'http://download.windowsupdate.com/c/msdownload/update/software/updt/2017/06/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu']' returned non-zero exit status 19.

So i tried running the command directly:

aria2c -x4 -o manifests/1507/KB4032695/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu --allow-overwrite=true http://download.windowsupdate.com/c/msdownload/update/software/updt/2017/06/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu

And saw this error:

Exception: [AbstractCommand.cc:351] errorCode=19 URI=http://download.windowsupdate.com/c/msdownload/update/software/updt/2017/06/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu
  -> [AbstractCommand.cc:789] errorCode=19 CUID#7 - Name resolution for download.windowsupdate.com failed:Timeout while contacting DNS servers

This is not surprising. They probably don't want automated tools downloading the updates. I'll try to find a workaround.

I'm running on Win10 WSL2 btw.

[m417z]

So i tried running the command directly:

aria2c -x4 -o manifests/1507/KB4032695/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu --allow-overwrite=true http://download.windowsupdate.com/c/msdownload/update/software/updt/2017/06/windows10.0-kb4032695-x64_ce02f93e314452c2dabd74bfbe536db170560ce6.msu

Works for me. Perhaps it's related to your internet connection or DNS configuration.

[m417z]

I'm running on Win10 WSL2 btw.

You can use Windows natively, without WSL. You'll have to unzip the tools.zip file, and install and run pymultitor according to the instructions.

[nimrodpar]

Okay I clicked the link directly, it tried to download, I canceled, then re-run the aria2 command and it worked.

Weird stuff, probably indeed DNS related.

Okay cheers I'm closing the issue.