Error during sync

[universalinterchange]

Get the following error after it sync's for a while.

Unexpected error occurred, stopping! Traceback (most recent call last): File "/home/mark/.local/bin/drpg", line 8, in sys.exit(run()) ^^^^^ File "/home/mark/.local/lib/python3.11/site-packages/drpg/cmd.py", line 32, in run DrpgSync(config).sync() File "/home/mark/.local/lib/python3.11/site-packages/drpg/sync.py", line 67, in sync pool.starmap(self._process_item, process_item_args) File "/usr/lib64/python3.11/multiprocessing/pool.py", line 375, in starmap return self._map_async(func, iterable, starmapstar, chunksize).get() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/multiprocessing/pool.py", line 774, in get raise self._value File "/usr/lib64/python3.11/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, *kwds)) ^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mark/.local/lib/python3.11/site-packages/drpg/sync.py", line 38, in wrapper return func(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mark/.local/lib/python3.11/site-packages/drpg/sync.py", line 76, in _process_item url_data = self._api.file_task(product["products_id"], item["bundle_id"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mark/.local/lib/python3.11/site-packages/drpg/api.py", line 94, in file_task while (data := resp.json()["message"])["progress"].startswith("Preparing"):

TypeError: string indices must be integers, not 'str'

[glujan]

That looks like an unhandled 400 response from DriveThruRPG API.

Thanks for reporting this! I'll try to reproduce and fix this.

[glujan]

It works for me 😕

I can prepare a branch for you with extra logging so we can understand what precisely is happening.

[glujan]

You can also try running it with --log-level=DEBUG and pasting the output here.

[universalinterchange]

Not sure if this makes a difference or not, but I have purchased a considerable amount of products from DTRPG. Something on the order of 50K+ files. It always dies after 33K files. I'm running this on Fedora 37 with python 3.11.1

How much of the debug log do you want? It's about 2.6M

[glujan]

@universalinterchange I need to prepare a new release for you to catch this exception and log more info, hope to do it this week.

[universalinterchange]

I threw in a little debug code and resp.json["message"] when it errors is: Connection refused by endpoint: 404 - {"error":"The requested product does not appear in your order history."}

But I don't know how to skip the product on that error

[glujan]

Thank you for digging into it. I'll try to prepare a new release based on what you've discovered.

Do you know what product is causing it - or rather why it is happening on this product? For example, if you go to My Library on your DriveThruRPG account, is it visible there?

[universalinterchange]

It appears to be a Product that was a Bundle from the WargamersVault that isn't visible on the DTRPG site. All of the bundle subproducts are still available. https://www.wargamevault.com/product/392924/OBSCURE-BATTLES-3--DRONE-COMMANDER-SQUADRON-BUNDLE. I wonder if it's a database integrity issue where the bundle was deleted from one table but still represented in another.

[glujan]

@universalinterchange Would you mind installing code from #35 (commit b0163b5085007993ce1728a3aaa7809d4b6c9a2f) ie by running: pip install https://codeload.github.com/glujan/drpg/zip/b0163b5085007993ce1728a3aaa7809d4b6c9a2f and trying out if this patch fixes the issue?

[universalinterchange]

That worked... thank you