Few issues feedback

[DR-Mello]

Moving along downloading the surface assets about 500 now, using the earlier script it had the "KeyError: 'id' " see pic.

There are 4 id's that don't exist tried the "remove_asset_from_metadata" script but quit on error, missing but owned are; wfgpebvaw we2rbizaw vktudjsaw vktffapaw

Ran the "calculate_all_checksums" seemed to help with the ealier KeyError maybe but will run checksum again if needed.

The "MemoryError" has halted the download script few time seems no issues restarting, been using the new versions when they come.

Not sure Line 38 DownloadAllAssets type -issue or not?

Just given feedback, works for me, Thanks!

[WAUthethird]

Hey there, thanks for the feedback!

The place the KeyError happens in would suggest that the server returned a 200 (success) code but not a valid response, which is unusual behavior for a server. Anything besides 200 is caught and retried.

Since the latest update which switched away from recursive function calls in the download script, I've noticed a big decrease in weird, unexplainable issues that I've been unable to diagnose properly and that usually crashed the script after a while. Let me know if you're still experiencing the issue on the latest code!

And thanks for the IDs! I'll check those and add them to the readme. Also, you mentioned that the remove_asset_from_metadata script quit on error? Which error?

As for the MemoryError, that's not an issue I've encountered myself. The complete metadata requires quite a bit of RAM to be loaded in initially - this memory usage does go down substantially as downloads progress, but I've seen it go as high as 4-5GB. If you don't have enough free RAM when you run the script, then (I assume) that error occurs. Check and make sure your version of Python is not 32-bit, by the way!

[DR-Mello]

Sorry can't remember the error, sure it would reproduce all i did was input an id like vktffapaw and no go.

Have 32GB on this machine ram usage 40% with Python 3.12.6 64bit.

Just after posted the above issues resurfaced;

Using the script just before the most recent, token expired input new token got this like above;

Closed and started python ran script received the mem error again tried newest script same.

Now running checksum calc did this the other day when had these issues all was good whenever it halted closed and restarted no problem but not sure if it was checksum py that helped but will see.

[DR-Mello]

After I ran checksum py back to downloading, few hurdles all good, like you have a 10MB/s connection with the traffic now varies from half to full bandwidth lately.

All good...

[WAUthethird]

Huh, it would surprise me if the checksum script is able to fix the issues you're describing, since the download script only really (besides saving new checksums) uses the checksum file to figure out which assets have yet to be downloaded. Either way, if it works now, great!

[DR-Mello]

My thoughts too just coincidence but...

Got up this morning with clear head and...

Oops! The memory problem was user error, running in PyCharm, JM checked heap set 2048 and doubled it to 4GB.

[WAUthethird]

Ah, gotcha!

[DR-Mello]

Regarding for resuming "download_all_assets" for me have not had issue with that, shutting down when token expires or with the issues from my mem allocation problems, seems all is working great won't know if missing anything yet but looks good to me.

When do have problem with download script the calc-checksum and all good ...Thanks for the great work!!

Just wish new about your script earlier people are hitting the servers, was utilizing full download bandwidth now last few days 6MB/s with moments of top speed of 13MB/s. But been downloading "surface" material with no issues now and at about 1500 files at 2TB yah along way to got to 7352 files!!

Edited because project started with 3.11 corrected to the latest; Not worried now it doesn't effect the cat's interested in but took a break and tried the "remove_asset_from_metadata" again and still getting memory error 4GB allocated even tried 5;