search

darkdragn / party

A quick *.party downloader
93 stars 13 forks source link

JSONDecode error when pulling user DB #46

Open Hydronicx opened 1 month ago

Hydronicx commented 1 month ago

party coomer onlyfans icarlyreboot

gives JSONDecodeError: Unterminated string starting at: line 1 column 2603455 (char 2603454)

On Windows, latest version installed with pip command from Readme.

Would be awesome if you fix this. Thanks!

darkdragn commented 2 weeks ago

Yeah, I've been looking into it. Near as I can tell, sometimes cloudflare caches an incomplete copy of the artists index API call and returns it. And cloudflare keeps ignoring the no-cache headers. There's actually a funny comment from the devs on the endpoint ranting about cloudflare hating json.

I'll let you know if I get anywhere with it. Right now I usually just take a break from trying, wait five minutes or so then it starts working again.

Hoernchen commented 2 days ago

Some users have so many posts that the server returns 429 and it breaks trying to generate the posts, and then it fails if you try to pull the db again immediately because again, the json is not json, but a 429 response.

Hydronicx commented 2 days ago

Very interesting! I also noticed (unrelated) that some files (especially videos) frequently do not download completely anymore. Does the code check for file completion when it downloads? Thanks!

Hoernchen commented 2 days ago

The way downloads break is by stalling with no traffic for some time, and then the connection breaks/times out, so it's different. The code does notice that: DEBUG | party.posts:download:199 - {'error': ClientPayloadError('Response payload is not completed'), 'filename': ....

Hydronicx commented 2 days ago

Is this what the full_check flag is for? It seems like it checks if the file is complete.