Update download_all_assets.py

[craigvc]

Added Threading for concurrent downloads creation of subfolders for categories and category selection and download resumption

[WAUthethird]

Hey there, thanks for your PR! If resuming works as expected, then that's great.

This is a pretty comprehensive modification of this script though, so I have a few questions:

• What is your rationale for removing the bad zip file checker? I've found that even when a download completes, the download may still be corrupted in some way. I'm doubtful that resuming alone will ensure file integrity, and I wrote this script in order to create a long-term archive where that sort of thing is paramount.
• I was under the impression that Quixel throttled outgoing bandwidth across all concurrent downloads to a total of ~10-12MB/s (each download getting a subset of this), which is why I did not implement threading. Are you observing ~10-12MB/s on each concurrent download? If so, I'll definitely take threading into consideration, since that would be a huge improvement.
• This is not something I'm fully familiar with, but how does session.post handle resuming without the Quixel server handling a Range header? This is something I tested myself with requests.post, and it did not work. Also, does {'Accept-Encoding': 'gzip, deflate'} work to download smaller zips?
• Why did you switch to a far more complex and slower method of calculating checksums?
• Why did you remove this code? It is crucial to ensure that if downloads drop out partway, we retry the download. This happens a lot in my experience!

  if (asset_path / f"{asset}.zip").stat().st_size != asset_length:
  print(f"\nDownload for asset {asset} was incomplete!")
  print("Waiting 5 seconds and retrying.")
  time.sleep(5)

• Why did you remove the config options and the components section from this code? Both are very important for archival.

  data = {"asset": asset,
      "config": {"highpoly": True,
                 "lowerlod_meshes": True,
                 "lowerlod_normals": True,
                 "ztool": True,
                 "brushes": True,
                 "meshMimeType": "application/x-fbx",
                 "albedo_lods": True},
      "components": asset_components}

Also, I notice that you've limited retries now, but I would prefer that retries be unlimited. (again, for archival reasons)

[WAUthethird]

One other thing I'd like to note is that a "mega PR" like this that comprehensively modifies the most important file in this repository is more difficult to work with than a few smaller PRs that target different areas of improvement. For example, having threading split out into its own PR would make it so that I could review and test solely that functionality without worrying about all the other changes you made.

[craigvc]

hey sorry that was most definetely not ready to go as a pr!

i was still doing many things and had a brain fart, in terms of multithreading yes it works

the rest i am still addressing!

[WAUthethird]

Awesome!

[craigvc]

let me give this more of a sanity check for you - it may not sit exactly as a PR but you can take what you need

[craigvc]

I will send the file outside the pr for you