Issues in sync workflow

[CheraHamza]

Sync workflow kept failing after reaching the timeout. I set IMDB_TRACE to true, and it seems like some error is blocking the execution. I keep getting those couple of log messages repeatedly until the timeout is reached and the sync workflow fails.

[cecobask]

Hi @CheraHamza, your sync workflow is stuck at the stage after entering credentials and clicking the submit button. The reason is more than likely that the IMDb website has prompted you to enter a captcha. This usually happens after a couple of failed attempts to authenticate, or when you’ve logged in and out within a short time frame.

I recommend the following:

• Double check your IMDb credentials are correctly stored in the repository secrets
• Manually authenticate on the IMDb website which can help clear the captcha prompt
• Halt the sync workflow for a few hours before trying again

Let me know how it goes! I’m expecting some bugs might pop up here and there as I had to rewrite the IMDb implementation.

To prevent this issue from happening I will add a check for captchas after authentication has been attempted.

[paviem]

Hi @cecobask, with IMDB_TRACE set to false, I get the following error (set to true, I get the same authorization error as @CheraHamza):

[CheraHamza]

• Manually authenticate on the IMDb website which can help clear the captcha prompt

Hi @cecobask, How can I manually authenticate though? If you mean the browser on my computer, then I am already signed in on IMDB

[CheraHamza]

I have been emailed this by IMDB. I think this is the reason the workflow is stuck at authentication. Now the issue is where can I put the verification code I receive by email to verify the authentication.

[cecobask]

When the IMDB_TRACE configuration value is set to true, the syncer outputs extra information about the activities happening in the headless browser. @paviem, you're experiencing the same issue in both cases, but when tracing is disabled you don't get the detailed logs output.

The syncer was initially set up to wait on two conditions after the submit button on the authentication page is pressed:

• successful authentication
• failed authentication due to invalid credentials

I did not consider that captcha prompts would show up after a couple of failed authentication attempts. To address this issue, in the latest commit I added a safeguard which would report if a captcha prompt had hindered the authentication flow and cancel the sync early. Without the latest changes, the sync can get stuck waiting for one of the above two conditions until the timeout kicks in.

I also added bot detection prevention, but that unfortunately will not eliminate the possibility of captcha prompts appearing.

In summary, if anybody is experiencing this issue they can do the following:

• sync their fork with the latest changes
• ensure the IMDb credentials stored as repository secrets are valid (this is very important)
• run the syncer locally with IMDB_HEADLESS config value set to false