Store photos and metadata in download directory

[esemwy]

I'd like to create the local web page equivalent of the desktop app. It would be helpful if the script could download and store images and json metadata along with the files.

[glujan]

That's a nice project idea!

While you can use drpg as a library, it currently does not support specyfying what fields to retrieve from the API. This method could be extended to do so: https://github.com/glujan/drpg/blob/ef8b5d3919ba9daea77664312bb146664c7f31eb/drpg/api.py#L74

[glujan]

Saving images and metadata is out of the scope of this library. But I'm good to extend drpg as described in the previous comment.

[esemwy]

The only thing missing is the actual product image. The "cover_url" just returns a slightly larger icon. "products_thumbnail100" of course, just returns the smaller icon. If I have time to reverse engineer some of the protocol next week, I'll see what I can find. If you have any notes to share, it'd be helpful.

[glujan]

@esemwy, you can have a look at #102 - it's not ready (I want to rewrite extra_fields to be a full list of fields to fetch, extend sync.py to allow easily selecting what fields to download, and allow post processing of API response), but maybe you find this usefull.

I don't think there was a field with a high-resolution cover, but it was a few years ago. Maybe something has changed since then. I remember using the official Windows app and mitmproxy to see what API looked like. Maybe the new website is not sever-side rendered as the old one, and uses the same API as the app?

[esemwy]

Looks like the client requests images that follow the pattern.

• https://api.drivethrurpg.com/images/publisherId/productId-thumb100.png
• https://api.drivethrurpg.com/images/publisherId/productId.png

We also know about "cover_url" which may or may not be vestigial.

• https://api.drivethrurpg.com/images/publisherId/productId-thumb140.png

The first being the thumbnail, and the second the large product image. Those, of course, redirect to the cloudfront URL where the content is actually stored. It's not clear how we derive the extension of the full size product image unless it's just expected to have the same format as the thumb100 or the thumb140. I'll write some test code later, and let you know.

[esemwy]

OK, so I got something working. I didn't actually need to download the cover images, although I can, but I did need the path, and the code I have seems to derive it OK:

        url = urlparse(product['cover_url'])
        filename = Path(url.path)
        newurl = f'{url.scheme}://api{url.netloc[3:]}{url.path[:-13]}{filename.suffix}'

The only thing that ends up a bit hacky is I need to instantiate DrpgSync so that I can call _file_path(). I guess I might suggest a more generic version that takes the path prefix and the compatibility_mode flag be added to the API.