[Feature Request}: Improve Error handling for IAFD headshot requests (handle 403 errors gracefully)

[fourstix]

Is your feature request related to a problem? Please describe. When the IAFD refuses requests from Cloudscrapper with a 403 error (Forbidden) it causes an agent like GEVI, to fail to match a headshot in IAFD and the headshot is replaced with the IAFD "No Photo Available" image. This happens even if there is a previously matched headshot, so when IAFD begins refusing requests. It causes existing headshots to be over-written with the No Photo Available image. This tends to wipe out previously correct headshot metadata from valid matches.

Describe the solution you'd like Catch the condition when the Cloudscrapper failure from requesting a headshot IAFD is refused with a 403 error, and in this case do not update the headshot metadata, instead of setting the headshot metadata to the "No Photo" image.

Additionally, maybe the 403 error handling code should short circuit and end the entire set of IAFD headshot requests, since IAFD seems to be using 403 to indicate that it is currently not serving data? I'm wondering that perhaps multiple 403 requests are exacerbating the situation with IAFD further?

Describe alternatives you've considered None. Except waiting a few hours and trying "Refreshing Metadata" when IAFD has returned to answering requests.

Additional context Here's a short demo to illustrate the issue. "Outta the Park!" is a movie that was successfully matched by the GEVI earlier. A second film "Overpowered" is being matched with the GEVI while IAFD is refusing requests.

Before searching on another film, this film "Outta the Park!" has the following cast headshots:

After IAFD refuses requests for headshots for "Refresh Metadata" on another film, "Overpowered", the cast list for "Outta the Park!" looks like this:

Notice that Jessie Colter's head shot image has now been over-written with the "No Photo Available" image.

The error in the GEVI log shows the request for Jessie Colter's headshot was refused from IAFD. 2023-05-19 13:00:58,595 (7fd1ce858b38) : INFO (sandbox:19) - GEVI - ---------------------------------------------------------------------------------------------------- 2023-05-19 13:00:58,596 (7fd1ce858b38) : INFO (sandbox:19) - GEVI - UTILS :: Unmatched Cast Name 5 - Jessie Colter/jessiecolter 2023-05-19 13:00:58,600 (7fd1ce858b38) : INFO (sandbox:19) - GEVI - UTILS :: CloudScraper Request: https://www.iafd.com/ramesearch.asp?searchtype=comprehensive&searchstring=Jessie+Colter 2023-05-19 13:01:02,753 (7fd1ce858b38) : ERROR (sandbox:19) - GEVI - UTILS :: Error: Cannot Process IAFD Cast Search Results: < Failed to read IAFD URL: ('< CloudScraper Failed: %s >', Exception(('< CloudScraper Failed Request Status Code: %s >', 403),)) - Processing Abandoned! > 2023-05-19 13:01:02,778 (7fd1ce858b38) : INFO (sandbox:19) - GEVI - ---------------------------------------------------------------------------------------------------- In fact, all of the requests for headshots from IAFD were refused with a 403 error.

[JPH71]

Good Catch

I will look into this and implement an exit from the scraping process when this happens... However, this does not seem to work always... for example you are using GEVI, then switch to using AEBN.... then click on a film previously scraped by GEVI, sometimes all the metadata disappears with the exception of the summary and poster... its like plex decides to refresh metadata, but no agent logs are produced... for me when IAFD starts submitting a 403 error... I first check the websote manually... usually it runs slow as well. I then reboot and wait 5-10 minutes or so... and this sorts the issue...

Jason xxx

On Fri, 19 May 2023 at 18:23, fourstix @.***> wrote:

Is your feature request related to a problem? Please describe. When the IAFD refuses requests from Cloudscrapper with a 403 error (Forbidden) it causes an agent like GEVI, to fail to match a headshot in IAFD and the headshot is replaced with the IAFD "No Photo Available" image. This happens even if there is a previously matched headshot, so when IAFD begins refusing requests. It causes existing headshots to be over-written with the No Photo Available image. This tends to wipe out previously correct headshot metadata from valid matches.

Describe the solution you'd like Catch the condition when the Cloudscrapper failure from requesting a headshot IAFD is refused with a 403 error, and in this case do not update the headshot metadata, instead of setting the headshot metadata to the "No Photo" image.

Additionally, maybe the 403 error handling code should short circuit and end the entire set of IAFD headshot requests, since IAFD seems to be using 403 to indicate that it is currently not serving data? I'm wondering that perhaps multiple 403 requests are exacerbating the situation with IAFD further?

Describe alternatives you've considered None. Except waiting a few hours and trying "Refreshing Metadata" when IAFD has returned to answering requests.

Additional context Here's a short demo to illustrate the issue. "Outta the Park!" is a movie that was successfully matched by the GEVI earlier. A second film "Overpowered" is being matched with the GEVI while IAFD is refusing requests.

Before searching on another film, this film "Outta the Park!" has the following cast headshots: [image: image] https://user-images.githubusercontent.com/5274281/239591469-42f9fbce-56bb-407b-86d8-14fd520036ae.png

After IAFD refuses requests for headshots for "Refresh Metadata" on another film, "Overpowered", the cast list for "Outta the Park!" looks like this: [image: image] https://user-images.githubusercontent.com/5274281/239593298-195cc40d-6730-4482-9ad2-23b95317348d.png

Notice that Jessie Colter's head shot image has now been over-written with the "No Photo Available" image.

The error in the GEVI log shows the request for Jessie Colter's headshot was refused from IAFD. 2023-05-19 13:00:58,595 (7fd1ce858b38) : INFO (sandbox:19) - GEVI -

2023-05-19 13:00:58,596 (7fd1ce858b38) : INFO (sandbox:19) - GEVI - UTILS :: Unmatched Cast Name 5 - Jessie Colter/jessiecolter 2023-05-19 13:00:58,600 (7fd1ce858b38) : INFO (sandbox:19) - GEVI - UTILS :: CloudScraper Request: https://www.iafd.com/ramesearch.asp?searchtype=comprehensive&searchstring=Jessie+Colter 2023-05-19 13:01:02,753 (7fd1ce858b38) : ERROR (sandbox:19) - GEVI - UTILS :: Error: Cannot Process IAFD Cast Search Results: < Failed to read IAFD URL: ('< CloudScraper Failed: %s >', Exception(('< CloudScraper Failed Request Status Code: %s >', 403),)) - Processing Abandoned! > 2023-05-19 13:01:02,778 (7fd1ce858b38) : INFO (sandbox:19) - GEVI -

In fact, all of the requests for headshots from IAFD were refused with a 403 error.

— Reply to this email directly, view it on GitHub https://github.com/CodyBerenson/PGMA-Modernized/issues/243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKI3AKPV3MGPDA7BJNJBE3LXG6T7TANCNFSM6AAAAAAYIBJXA4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[JPH71]

Or alternatively when a 403 occurs we will have a different picture uploaded... so that it is instantly recognisable that a picture could not be got from IAFD... Give me a number of days to think this through.... Jason

[fourstix]

Hi, My preference would be to change the ticky mark (green check or red x) to something that indicates the scrapper was denied the image requested. (Response 403 "Forbidden" means the request is valid, but is being denied for some other reason, while 404 means "Not Found".)

Just to pick on poor Jessie Colter. Here's an image showing what I am suggesting as if, all the other actors got images returned, but the request for Jessie's image from IAFD was denied by a 403 response. I would suggest not setting the cast image meta data at all, and instead just set the tick mark to something like "?" for an server request error.

Something like this:

[JPH71]

Good idea... I will look for an unicode symbol to signify this error

Jason xx

On Wed, 24 May 2023, 12:52 fourstix, @.***> wrote:

Hi, My preference would be to change the ticky mark (green check or red x) to something that indicates the scrapper was denied the image requested. (Response 403 "Forbidden" means the request is valid, but is being denied for some other reason, while 404 means "Not Found".)

Just to pick on poor Jessie Colter. Here's an image showing what I am suggesting as if, all the other actors got images returned, but the request for Jessie's image from IAFD was denied by a 403 response. I would suggest not setting the cast image meta data at all, and instead just set the tick mark to something like "?" for an server request error.

Something like this: [image: Suggestion] https://user-images.githubusercontent.com/5274281/240579928-fc5f4441-24e4-41fc-94b3-cde187519ba5.png

— Reply to this email directly, view it on GitHub https://github.com/CodyBerenson/PGMA-Modernized/issues/243#issuecomment-1560898840, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKI3AKOTC2T7TL2QC2IXHS3XHXR7LANCNFSM6AAAAAAYIBJXA4 . You are receiving this because you commented.Message ID: @.***>

[fourstix]

Hi Jason, There's the Unicode replacement character \uFFFD, which is a black diamond with a white question mark. It's defined to represent unmatched characters in text. It seems like a good fit. :-)

https://en.wikipedia.org/wiki/Specials_(Unicode_block)

Best regards, Gaston

[JPH71]

Star!

On Wed, 24 May 2023, 15:48 fourstix, @.***> wrote:

Hi Jason, There's the Unicode replacement character \uFFFD, which is a black diamond with a white question mark. It's defined to represent unmatched characters in text. It seems like a good fit. :-)

https://en.wikipedia.org/wiki/Specials_(Unicode_block)

Best regards, Gaston

— Reply to this email directly, view it on GitHub https://github.com/CodyBerenson/PGMA-Modernized/issues/243#issuecomment-1561188654, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKI3AKKYCPNSG34KYBAEA6TXHYGUJANCNFSM6AAAAAAYIBJXA4 . You are receiving this because you commented.Message ID: @.***>

[JPH71]

I am also going to create a collection for films who get an iafd error... it will be in the group like stacked, found on iafd etc...

One will then be able to keep track of all films and scenes... And the re scrape them when needed....

Jason.... thanks for pushing me down this line of thought... its been bopping up and down once or twice in the past but just didn't stick its head over the parapet...

On Wed, 24 May 2023, 17:12 Jason Hudson, @.***> wrote:

Star!

On Wed, 24 May 2023, 15:48 fourstix, @.***> wrote:

Hi Jason, There's the Unicode replacement character \uFFFD, which is a black diamond with a white question mark. It's defined to represent unmatched characters in text. It seems like a good fit. :-)

https://en.wikipedia.org/wiki/Specials_(Unicode_block)

Best regards, Gaston

— Reply to this email directly, view it on GitHub https://github.com/CodyBerenson/PGMA-Modernized/issues/243#issuecomment-1561188654, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKI3AKKYCPNSG34KYBAEA6TXHYGUJANCNFSM6AAAAAAYIBJXA4 . You are receiving this because you commented.Message ID: @.***>

[JPH71]

I did not create a 403 Error collection - mainly because I as off do not yet have the code for removing a film from a collection... But will think of this.....

However a White Question Mark signifying a 403 Error has been added.... Please look at this.... It could do with some improvement.....

Code is with Cody awaiting upload.... Agents converted so far are:

1. Best Exclusive Porn
2. Fagalicious
3. GEVI
4. GEVIScenes
5. QueerClick
6. WayBig

I will start getting the other agents converted.....

Jason

[fourstix]

The 403 Error collection is more of a nice to have thing for me. So not having that feature isn't a big deal for me.

[JPH71]

I will create a collection for it... I just realised that at each scrape all collections are reset...

By next week this should be in effect...

[fourstix]

Hi, It's gotten better. :-) But it seems like a 403 Error still wipes out the actor's image.

Here's the cast of Back to Black 2 before matching: Here's the result from matching Hairy Raw Tops:

Note that the Danny Lopez request was refused by IAFD and is now shown as DL in the cast for the previous movie

So a failure to match because of a 403 error is still wiping out images from previous matches. Is there a way to just not update the metadata for an actor that doesn't match because of a 403 error?

Here is the log file: com.plexapp.agents.GEVI.log

[fourstix]

In case you need it, the names of the movies matched above were: (USA Jock) - Back To Black 2 and (Raw Joxxx) - Hairy Raw Tops

If you need any more information please let me know.

Oh and many, many thanks for fixing the GEVI agent so quickly!

Best regards, Gaston

[JPH71]

Let me think the code..... I will see if there is a way to iterate through the metadata..... if there is then it would be possible to skip processing the actor.....

Jasonx

On Mon, 10 Jul 2023 at 16:21, fourstix @.***> wrote:

Hi, It's gotten better. :-) But it seems like a 403 Error still wipes out the actor's image.

Here's the cast of Back to Black 2 before matching: [image: Back to Black 2 Cast (Before)] https://user-images.githubusercontent.com/5274281/252384482-226376b7-75f0-406e-8ada-68ca57e865f0.png Here's the result from matching Hairy Raw Tops: [image: Hairy Raw Tops] https://user-images.githubusercontent.com/5274281/252384662-e59ea9fa-c729-48ec-a976-fbf02c77fcc2.png

Note that the Danny Lopez request was refused by IAFD and is now shown as DL in the cast for the previous movie [image: Back to Black 2 Cast (After)] https://user-images.githubusercontent.com/5274281/252385061-7f89679f-f5c6-4522-a28a-6c53ae3a28d4.png

So a failure to match because of a 403 error is still wiping out images from previous matches. Is there a way to just not update the metadata for an actor that doesn't match because of a 403 error?

Here is the log file: com.plexapp.agents.GEVI.log https://github.com/CodyBerenson/PGMA-Modernized/files/12003641/com.plexapp.agents.GEVI.log

— Reply to this email directly, view it on GitHub https://github.com/CodyBerenson/PGMA-Modernized/issues/243#issuecomment-1629183032, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKI3AKIVGJPBG4NZVRW4KQ3XPQMYTANCNFSM6AAAAAAYIBJXA4 . You are receiving this because you commented.Message ID: @.***>

[JPH71]

Ok, I will do - I was replying to your previous when this one came through.....

Cheers

Jason x

On Mon, 10 Jul 2023 at 16:38, fourstix @.***> wrote:

In case you need it, the names of the movies matched above were: (USA Jock) - Back To Black 2 and (Raw Joxxx) - Hairy Raw Tops

If you need any more information please let me know.

Oh and many, many thanks for fixing the GEVI agent so quickly!

Best regards, Gaston

— Reply to this email directly, view it on GitHub https://github.com/CodyBerenson/PGMA-Modernized/issues/243#issuecomment-1629213037, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKI3AKO3QU4HGNUSAAW5EZTXPQOWHANCNFSM6AAAAAAYIBJXA4 . You are receiving this because you commented.Message ID: @.***>

[fourstix]

Yikes. I just keep making things worse. Stopping/Restarting Plex and refreshing the metadata, I finally get the Danny Lopez head-shot to appear, but now Buster Sly and Tyler Reed are gone.

To be fair, this isn't really the agent's fault, IAFD keeps getting more and more cranky, and sporadically refusing more requests.

I'm wondering if it would be better to have the ability to put my own head-shot server or server image directory ahead of IAFD like one can do with the Gay Adult NFO agent.

[JPH71]

I had requested some code off a plex guru to do this exactly but as as of yet I have heard nowt from him...

I am going to try and find a way to detect if there is a photo already attached and then precent what is already there been overwritten...

On Mon, 10 Jul 2023, 19:42 fourstix, @.***> wrote:

Yikes. I just keep making things worse. Stopping/Restarting Plex and refreshing the metadata, I finally get the Danny Lopez head-shot to appear, but now Buster Sly and Tyler Reed are gone.

To be fair, this isn't really the agent's fault, IAFD keeps getting more and more cranky, and sporadically refusing more requests.

[image: image] https://user-images.githubusercontent.com/5274281/252418527-2ee65406-7b8c-4c93-a0ca-03641e20b32e.png

I'm wondering if it would be better to have the ability to put my own head-shot server or server image directory ahead of IAFD like one can do with the Gay Adult NFO agent.

— Reply to this email directly, view it on GitHub https://github.com/CodyBerenson/PGMA-Modernized/issues/243#issuecomment-1629424550, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKI3AKK6QGJXAQDR2YDCPD3XPQ5INANCNFSM6AAAAAAYIBJXA4 . You are receiving this because you commented.Message ID: @.***>

[JPH71]

Please close this issue as this has been implemented. For the head shot stuff put it in as a request

cheers Jason

[fourstix]

OK, I will close this and put in a separate request for the head shot. Thanks for fixing this issue.

[JPH71]

Cheers mate... then I can keep an eye on it...

On Thu, 3 Aug 2023, 20:40 fourstix, @.***> wrote:

OK, I will close this and put in a separate request for the head shot. Thanks for fixing this issue.

— Reply to this email directly, view it on GitHub https://github.com/CodyBerenson/PGMA-Modernized/issues/243#issuecomment-1664456447, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKI3AKKZ3JPM5S3S253AYSTXTPWBPANCNFSM6AAAAAAYIBJXA4 . You are receiving this because you commented.Message ID: @.***>