Asset conversion stalls on media conversion

[brett-hamilton]

System Info

• OS: Windows 11

• Python: 3.12.2

  Following the build instructions from doc/build_instructions/windows_msvc.md, when running the first Python command the process stalled out for 30+ minutes before I manually stopped it.

Command

py -m openage --add-dll-search-path <path> --add-dll-search-path <path> main

Output Before Stopping

INFO [py] Looking for compatible games to convert...
INFO [py] Compatible game edition detected:
INFO [py]  * Age of Empires 2: The Conqueror's Trial Version
INFO [py] converting metadata
INFO [py] Finished metadata read (10.50 seconds)
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Finished data conversion (0.61 seconds)
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] -- Exporting graphics files...
-- Files done:   0/275

Error Output After Stopping

INFO [py] -- Exporting graphics files...
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<openage directory>\build\openage\__main__.py", line 178, in <module>
    sys.exit(main())
             ^^^^^^
  File "<openage directory>\build\openage\__main__.py", line 171, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<openage directory>\build\openage\main\main.py", line 82, in main
    convert_assets(asset_path, args)
  File "<openage directory>\build\openage\convert\main.py", line 119, in convert_assets
    convert(args)
  File "<openage directory>\build\openage\convert\tool\driver.py", line 36, in convert
    convert_metadata(args)
  File "<openage directory>\build\openage\convert\tool\driver.py", line 105, in convert_metadata
    ModpackExporter.export(modpack, args)
  File "<openage directory>\build\openage\convert\processor\export\modpack_exporter.py", line 60, in export
    MediaExporter.export(modpack.get_media_files(), sourcedir, modpack_dir, args)
rt
    MediaExporter._export_multithreaded(
  File "<openage directory>\build\openage\convert\processor\export\media_exporter.py", line 315, in _export_multithreaded
    while outqueue.qsize() < expected_size:
  File "<string>", line 2, in qsize
  File "<user>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\managers.py", line 821, in _callmethod
    kind, result = conn.recv()
                   ^^^^^^^^^^^
  File "<user>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "<user>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 321, in _recv_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
^C

Troubleshooting with the dev, I ran the following command and received a different error.

Command

py openage convert --jobs 1

Output

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<openage directory>\openage\build\openage\__main__.py", line 17, in <module>
    from .log import set_loglevel, verbosity_to_level, ENV_VERBOSITY
ImportError: attempted relative import with no known parent package

[brett-hamilton]

Adding -m to the py command made the conversion process execute properly. However, it still required the --jobs 1 argument, so the multiprocessing problem persists. Thanks!

Command

py -m openage --add-dll-search-path <path> --add-dll-search-path <path> convert --jobs 1

Output

INFO [py] Looking for compatible games to convert...
INFO [py] Compatible game edition detected:
INFO [py]  * Age of Empires 2: The Conqueror's Trial Version
INFO [py] converting metadata
INFO [py] using cached wrapper: C:\Users\brett\AppData\Local\Temp\AOC_empires2_x1.dat.pickle
INFO [py] Finished metadata read (1.06 seconds)
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Finished data conversion (1.49 seconds)
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] -- Exporting graphics files...
INFO [py] -- Exporting terrain files...
INFO [py] -- Exporting sound files...
INFO [py] Dumping metadata files...
INFO [py] Finished modpack export (23.45 seconds)
INFO [py] asset conversion complete; asset version: 7

[heinezen]

https://github.com/SFTtech/openage/pull/1625 added an error callback, so if you run the conversion now, you should see an error message. Could you try converting again to see what's the issue?

[brett-hamilton]

After updating with the error callback, here is the error message. Looks like a path issue:

INFO [py] -- Exporting graphics files...
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<openage directory>\build\openage\__main__.py", line 178, in <module>
    sys.exit(main())
             ^^^^^^
  File "<openage directory>\build\openage\__main__.py", line 171, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<openage directory>\build\openage\main\main.py", line 82, in main
    convert_assets(asset_path, args)
  File "<openage directory>\build\openage\convert\main.py", line 119, in convert_assets
    convert(args)
  File "<openage directory>\build\openage\convert\tool\driver.py", line 36, in convert
    convert_metadata(args)
  File "<openage directory>\build\openage\convert\tool\driver.py", line 105, in convert_metadata
    ModpackExporter.export(modpack, args)
  File "<openage directory>\build\openage\convert\processor\export\modpack_exporter.py", line 60, in export
    MediaExporter.export(modpack.get_media_files(), sourcedir, modpack_dir, args)
  File "<openage directory>\build\openage\convert\processor\export\media_exporter.py", line 120, in export
    MediaExporter._export_multithreaded(
  File "<openage directory>\build\openage\convert\processor\export\media_exporter.py", line 310, in _export_multithreaded
    MediaExporter.log_fileinfo(
  File "<openage directory>\build\openage\convert\processor\export\media_exporter.py", line 620, in log_fileinfo
    target_size = os.path.getsize(target_path)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 62, in getsize
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

[heinezen]

Interesting. I think the reaon is that the path could not be resolved, which is why it's None.

[heinezen]

Ok I think the issue is that when the the file info is logged, it's still being converted due to multiprocessing...

[heinezen]

I've fixed the logging but I think this has nothing to do with your original issue :D Could you retry with https://github.com/SFTtech/openage/pull/1627

[brett-hamilton]

Yes, after taking the changes from #1627, here is the output: The Keyboard Interrupt at line 251 is from me hitting ^C after it got stuck at the same point it did in my original issue (-- Files done: 0/275)

converting from 'C:\Program Files (x86)\Microsoft Games\Age of Empires II The Conquerors Expansion Trial'
INFO [py] Looking for compatible games to convert...
INFO [py] Compatible game edition detected:
INFO [py]  * Age of Empires 2: The Conqueror's Trial Version
INFO [py] converting metadata
INFO [py] using cached wrapper: <user directory>\AppData\Local\Temp\AOC_empires2_x1.dat.pickle
INFO [py] Finished metadata read (1.02 seconds)
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Finished data conversion (1.48 seconds)
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] -- Exporting graphics files...
Exception in thread Thread-3 (_handle_results):
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 789, in _export_texture
    from ...value_object.read.media.slp import SLP
ImportError: DLL load failed while importing slp: The specified module could not be found.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()   0/275
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 595, in _handle_results
    cache[job]._set(i, obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 781, in _set
    self._error_callback(self._value)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 263, in error_callback
    raise exception275
ImportError: DLL load failed while importing slp: The specified module could not be found.
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-6:
Process SpawnPoolWorker-4:
Process SpawnPoolWorker-5:
Process SpawnPoolWorker-7:
Process SpawnPoolWorker-11:
Process SpawnPoolWorker-8:
Process SpawnPoolWorker-12:
Process SpawnPoolWorker-13:
Process SpawnPoolWorker-9:
Process SpawnPoolWorker-10:
Traceback (most recent call last):
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 316, in _export_multithreaded
    MediaExporter._show_progress(outqueue.qsize(), expected_size)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 461, in _show_progress
    print(f"-- Files done: {format_progress(current_size, total_size)}",
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<user directory>\source\repos\openage\build\openage\__main__.py", line 178, in <module>
    sys.exit(main())
             ^^^^^^
  File "<user directory>\source\repos\openage\build\openage\__main__.py", line 171, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\source\repos\openage\build\openage\main\main.py", line 82, in main
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
    convert_assets(asset_path, args)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\source\repos\openage\build\openage\convert\main.py", line 119, in convert_assets
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    convert(args)
  File "<user directory>\source\repos\openage\build\openage\convert\tool\driver.py", line 36, in convert
    convert_metadata(args)
  File "<user directory>\source\repos\openage\build\openage\convert\tool\driver.py", line 105, in convert_metadata
    ModpackExporter.export(modpack, args)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\modpack_exporter.py", line 60, in export
    MediaExporter.export(modpack.get_media_files(), sourcedir, modpack_dir, args)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 120, in export
    MediaExporter._export_multithreaded(
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 277, in _export_multithreaded
    with multiprocessing.Pool(worker_count) as pool:
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 739, in __exit__
    self.terminate()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 657, in terminate
    self._terminate()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 698, in _terminate_pool
    raise AssertionError(
AssertionError: Cannot have cache with result_handler not alive

[heinezen]

Ah looks like it's the import error again.

https://github.com/SFTtech/openage/pull/1628 should produce less import errors, at least for fresh builds that run from the build directory.

[brett-hamilton]

I used the change in #1628 and re-built the project, here is that output:

converting from 'C:\Program Files (x86)\Microsoft Games\Age of Empires II The Conquerors Expansion Trial'
INFO [py] Looking for compatible games to convert...
INFO [py] Compatible game edition detected:
INFO [py]  * Age of Empires 2: The Conqueror's Trial Version
INFO [py] converting metadata
INFO [py] using cached wrapper: <user directory>\AppData\Local\Temp\AOC_empires2_x1.dat.pickle
INFO [py] Finished metadata read (1.27 seconds)
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Finished data conversion (1.48 seconds)
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] -- Exporting graphics files...
Exception in thread Thread-3 (_handle_results):
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 789, in _export_texture
    from ...value_object.read.media.slp import SLP
ImportError: DLL load failed while importing slp: The specified module could not be found.
"""-- Files done:   0/275

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()   0/275
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 595, in _handle_results
    cache[job]._set(i, obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 781, in _set
    self._error_callback(self._value)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 263, in error_callback
    raise exception275
ImportError: DLL load failed while importing slp: The specified module could not be found.
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-5:
Process SpawnPoolWorker-4:
Process SpawnPoolWorker-6:
Process SpawnPoolWorker-9:
Process SpawnPoolWorker-7:
Process SpawnPoolWorker-10:
Process SpawnPoolWorker-8:
Process SpawnPoolWorker-11:
Process SpawnPoolWorker-12:
Process SpawnPoolWorker-13:
Traceback (most recent call last):
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 315, in _export_multithreaded
    while outqueue.qsize() < expected_size:
          ^^^^^^^^^^^^^^^^
  File "<string>", line 2, in qsize
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\managers.py", line 820, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 301, in _send_bytes
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
    nwritten, err = ov.GetOverlappedResult(True)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)

  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
During handling of the above exception, another exception occurred:
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<user directory>\source\repos\openage\build\openage\__main__.py", line 183, in <module>
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
    sys.exit(main())
             ^^^^^^
  File "<user directory>\source\repos\openage\build\openage\__main__.py", line 176, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\source\repos\openage\build\openage\main\main.py", line 82, in main
    convert_assets(asset_path, args)
  File "<user directory>\source\repos\openage\build\openage\convert\main.py", line 119, in convert_assets
    convert(args)
  File "<user directory>\source\repos\openage\build\openage\convert\tool\driver.py", line 36, in convert
    convert_metadata(args)
  File "<user directory>\source\repos\openage\build\openage\convert\tool\driver.py", line 105, in convert_metadata
    ModpackExporter.export(modpack, args)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\modpack_exporter.py", line 60, in export
    MediaExporter.export(modpack.get_media_files(), sourcedir, modpack_dir, args)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 120, in export
    MediaExporter._export_multithreaded(
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 277, in _export_multithreaded
    with multiprocessing.Pool(worker_count) as pool:
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 739, in __exit__
    self.terminate()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 657, in terminate
    self._terminate()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 698, in _terminate_pool
    raise AssertionError(
AssertionError: Cannot have cache with result_handler not alive

[heinezen]

I think I have debug this myself on Windows.

[heinezen]

Hey, I have finally found the cause of the issue. The reason it didn't work was that the DLL paths weren't propagated to the multiprocessing subprocesses, which then made th import fail. https://github.com/SFTtech/openage/pull/1649 will fix the issue and add even more sane defaults.