search

javinizer / Javinizer

(NSFW) Organize your local Japanese Adult Video (JAV) library
MIT License
599 stars 62 forks source link

WARNING: [Javinizer] Unable reach Javlibrary, enter websession/cookies to use the scraper #169

Open jvlflame opened 3 years ago

jvlflame commented 3 years ago

Updated June 16 2024

Join the discord server for support: https://discord.gg/Pds7xCpzpc

Download the updated files: >> 2.5.17_-_v1.01.zip <<

Instructions for both CLI and GUI

1. Open the javinizer powershell module directory with CLI command Javinizer -OpenModule 2. Download the updated script files in the attached 2.5.17.zip and extract the files DIRECTLY into the module folder and replace all existing files

3a. Install and run the flaresolverr application. If you modify the port or need to access the flaresolverr API from a different address, take note of it for the next step. https://github.com/FlareSolverr/FlareSolverr

3b. If you set flaresolverr to a different port or address from its default, open up the files Public/Get-JavlibraryUrl.ps1, Public/Get-JavlibraryData.ps1 and Public/Javinizer.ps1 and update the $FLARESOLVERR_URL variable to point to the correct URL and save.

  1. Check that the scraper.movie.javlibrary (or whichever language) setting is enabled and that javlibrary is added into your scraper metadata settings

  2. Set the throttlelimit setting to 1. The scraper may break often if it is greater than 1.

  3. Test that javlibrary scraper is working in the CLI. View your flaresolverr logs to check that it is solving the cloudflare challenges.

    Javinizer -Find SSNI-776 -Javlibrary

Instructions for GUI (Windows)

  1. Folllow the CLI and GUI instructions above to extract the updated files to your module directory

  2. Open PowerShell 7 in administrator mode and run the following to replace your current GUI installation Javinizer -InstallGUI -Force

  3. Open the GUI and confirm that scraping works Javinizer -OpenGUI

Instructions for GUI and CLI (Docker)

  1. SSH into your running container 
    
# Replace "<container_name>" with the name of your javinizer container

docker exec -it /bin/bash


2. Find the javinizer module path inside the container. In most cases it will be `/root/.local/share/powershell/Modules/Javinizer/2.5.17`. If not, enter PowerShell using `pwsh` and run `(Get-InstalledModule).InstalledLocation`. 

3. Use `curl` (or alternative) to download the `2.5.17.zip` file below into the container and copy the contents from `Private` and `Public` into their respective directories in the module folder.

4. If you want to update the GUI, copy and replace the `javinizergui.ps1` file from under `Universal/Repository` in the zip file into `home/data/Repository/javinizergui.ps1`

5. You may need to restart the container to propagate changes. (You will need to repeat these steps if you delete/reinstall the container)
cavalia88 commented 3 years ago

I have tried changing the javlibrary.baseurl setting to http://n53i.com, still getting the same error message.

Can you provide more detailed instructions on copying the cookie contents and user agent details? It's not very clear from your instructions

jvlflame commented 3 years ago

I have tried changing the javlibrary.baseurl setting to http://n53i.com, still getting the same error message.

Can you provide more detailed instructions on copying the cookie contents and user agent details? It's not very clear from your instructions

@cavalia88 Can you try setting the javlibrary.baseurl to http://www.n53i.com instead? Not sure if that will change anything but I'm not having any issues from that mirror right now.

Basically you want to access javlibrary.com (or whatever your javlibrary.baseurl is set as) via your browser and complete the cloudflare captcha when prompted. After doing so, check the site's browser cookies and copy each of the corresponding cookie content over to Javinizer when it prompts you.

image

Though I did just try it right now and javlibrary isn't giving me the proper cookies in the browser. It's possible that Cloudflare is now properly detecting legitimate (browser) vs bot (javinizer) traffic and isn't providing the cookies in the browser...

EDIT: If the proper cookies aren't showing up, try deleting the cf_clearance cookie from your browser and then refreshing. Rinse and repeat until all the proper cookies show up:

cavalia88 commented 3 years ago

Thank you for the clarification. I realized what the initial problem was on my side. I updated the Javinizer version to the latest version, but was amending the settings file of the earlier version. Changing the javlibrary.baseurl setting to http://n53i.com works now for me

When Javinizer updates to a new version, the old settings are not automatically migrated? We need to manually bring the settings over? I'm using the CLI version via Powershell on Windows 10

jvlflame commented 3 years ago

Glad it's working for you now!

If you use the native PowerShell method, Update-Module Javinizer it won't migrate settings. If you use the custom update function Javinizer -UpdateModule then it will.

damenootoko commented 3 years ago

It seems like n53i.com now have cloudflare as well, and I've been trying to fill the cookies values from javlibrary while on it's baseurl + cookies value from n53i while using it's baseurl as well but both has been failing consistently with errors that shows i need to input captcha

jvlflame commented 3 years ago

@damenootoko I'm using the default HTTPS javlibrary site to get the cookies and it's still working for me. Make sure when you paste your browser useragent it's all on a single line. I noticed when copying from google it gets separated into 2 lines so you'll need to paste it somewhere first and copy it again if you're using that method.

image

cavalia88 commented 3 years ago

Same here, n53i.com no longer works. Does Javinizer allow scraping from other websites? Or it currently only works with Javlibrary?

Since Javlibrary is causing so many issues, thought it may be better if i switch to another website.

jvlflame commented 3 years ago

@cavalia88 Yes, you can disable the javlibrary scraper completely and set your metadata priorities to any scraper of your choice.

Open the settings via the commandline by using Javinizer -OpenSettings.

jvlflame commented 3 years ago

I found an issue where not using the -Path parameter when running Javinizer would fail to append to necessary cookies onto the scraper.

E.g. running Javinizer -Recurse would fail, but Javinizer -Path . -Recurse would work. Going to push a new release with this fix which will hopefully resolve some people's issues.

yumesakiri commented 3 years ago

I ran into this issue because I was using https://www.b49t.com as a mirror (can't remember why I started using b49t). https://www.n53i.com wasn't working either, so I changed the base url back to https://www.javlibrary.com itself and it worked. I noticed I couldn't find any of the aforementioned cookies for any of the 3 sites on any of Chrome, Firefox, Safari, and I'm not sure why. It looked like I was failing n53i's cloudflare check when I looked. Anyway, hope this helps someone.

natsocenis commented 2 years ago

Looks like this one is happening again. Started a week ago.

Have tried the options but my cookies for the JAVLibrary site do not match what the tool needs, and the mirrors either ask for cookie https://www.n53i.com info or dont seem to work (https://www.b49t.com)

Has anyone got this to work?

Flute7 commented 2 years ago

Looks like this one is happening again. Started a week ago.

Have tried the options but my cookies for the JAVLibrary site do not match what the tool needs, and the mirrors either ask for cookie https://www.n53i.com info or dont seem to work (https://www.b49t.com)

Has anyone got this to work?

Load the site (www.o58c.com) in an incognito browser window (chrome) to generate the full list of cookies. Even after clearing cookies and site data in a normal tab, I couldn't get all the cookies until I did this.

Should say there are 9 cookies. You will need to select cookies under o58c.com and www.o58c.com directories as the required info is spread across both entries.

chronotank79 commented 1 year ago

Having this issue again. I've tried all the different solutions in this thread but I cannot get any of the various mirrors suggested here to give me the cookies that Javinizer wants. Another thing that is strange is that if I run the CLI Javinizer multiple times, it will intermittently sometimes be able to scrape one or two movies (that it had given an error for previously.)

chronotank79 commented 1 year ago

UPDATE: using a VPN while scraping seems to have fixed the issue. Turn off VPN, error again. Turn on VPN, works again.

I think if you are sorting a large number of files, cloudflare may eventually decide to require additional security from your IP address. What is strange about this is that Javlibrary works perfectly normally in my normal web browsers.

S1lv3rL10n commented 7 months ago

Cannot locate all cookie values. Have tried Firefox, Chrome and Edge. Are there other workarounds?

ssssionsimon commented 3 months ago

Having this issue in the current version; none of the backup websites seem to be online. Javinizer asks for __cf_bm, cf_clearance and UserAgent. I do see the cf_clearance cookie, but the __cf_bm cookie seems not to be visible. Any ideas?

jvlflame commented 3 months ago

@ssssionsimon I updated the post to provide the latest instructions on how to fix the javlibrary scraper.

androo00 commented 3 months ago

@jvlflame i have followed your instructions and have flaresolverr running in windows. I have specified the right host in the 3 public powershell scripts.

If i run the powershell command on the flaresolverr github to test, it works fine and solves the challenge. But when i try to sort it cant seem to solve the challenge.

2024-06-23 14:06:45 INFO Incoming request => POST /v1 body: {'session': 'javinizer', 'cmd': 'request.get', 'maxTimeout': 60000, 'url': 'http://www.javlibrary.com/en/vl_searchbyid.php?keyword=SSNI-776'} 2024-06-23 14:06:50 INFO Challenge detected. Title found: Just a moment... 2024-06-23 14:07:45 ERROR Error: Error solving the challenge. Timeout after 60.0 seconds. 2024-06-23 14:07:45 INFO Response in 60.115 s 2024-06-23 14:07:45 INFO 10.10.1.60 POST http://10.10.1.60:8191/v1 500 Internal Server Error

Command line sort: Write-Error: /root/.local/share/powershell/Modules/Javinizer/2.5.17/Public/Get-JavlibraryUrl.ps1:38 Line | 38 | Write-JVLog -Write:$script:JVLogWrite -LogPath $script:JV … | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | [SSNI-776] [Get-JavlibraryUrl] Error occured on [GET] on URL [http://www.javlibrary.com/en/vl_searchbyid.php?keyword=SSNI-776]: {"status": "error", "message": "Error: | Error solving the challenge. Timeout after 60.0 seconds.", "startTimestamp": 1719115605104, "endTimestamp": 1719115665219, "version": "3.3.19"} WARNING: [SSNI-776] [Get-JavlibraryUrl] not matched on JavLibrary

Running test from same machine in powershell using the following command block works fine and shows results. `$body = @{ cmd = "request.get" url = "http://www.javlibrary.com/en/vl_searchbyid.php?keyword=SSNI-776" maxTimeout = 60000 } | ConvertTo-Json

irm -UseBasicParsing 'http://10.10.1.60:8191/v1' -Headers @{"Content-Type"="application/json"} -Method Post -Body $body`

2024-06-23 14:14:56 INFO Incoming request => POST /v1 body: {'url': 'http://www.javlibrary.com/en/vl_searchbyid.php?keyword=SSNI-776', 'cmd': 'request.get', 'maxTimeout': 60000} 2024-06-23 14:14:58 INFO Challenge detected. Title found: Just a moment... 2024-06-23 14:15:09 INFO Challenge solved! 2024-06-23 14:15:11 INFO Response in 15.676 s 2024-06-23 14:15:11 INFO 10.10.1.60 POST http://10.10.1.60:8191/v1 200 OK

jvlflame commented 3 months ago

@androo00 Did you update to the latest flaresolverr version v3.3.20? You could also try removing all flaresolverr instances where it's setting session = javinizer from all the updated script files so that it'll recreate a new session on every fetch (though I'm not sure that would be very performant).

androo00 commented 3 months ago

@jvlflame weird.... it works now. Im guessing you wrote that up when it was version 3.3.19. Updated to 3.3.20 which was released today and it works.

Thanks for bringing it to my attention. And thanks for all the work you put into this

kraussian commented 2 months ago

@jvlflame I'm getting the same error as @androo00 and I'm using the latest FlareSolverr which is v3.3.21.

From FlareSolverr:

2024-07-26 12:31:22 INFO     Serving on http://0.0.0.0:8191
2024-07-26 12:35:50 INFO     Incoming request => POST /v1 body: {'session': 'javinizer', 'cmd': 'request.get', 'maxTimeout': 60000, 'url': 'https://www.javlibrary.com/en/vl_searchbyid.php?keyword=SSNI-776'}
2024-07-26 12:35:52 INFO     Challenge detected. Title found: Just a moment...
2024-07-26 12:36:51 ERROR    Error: Error solving the challenge. Timeout after 60.0 seconds.
2024-07-26 12:36:51 INFO     Response in 61.304 s
2024-07-26 12:36:51 INFO     127.0.0.1 POST http://localhost:8191/v1 500 Internal Server Error

From Javinizer:

Javinizer -Find SSNI-776 -Javlibrary
Write-Error: C:\Users\xyz\PowerShell\Modules\Javinizer\2.5.17\Public\Get-JavlibraryUrl.ps1:38
Line |
  38 |              Write-JVLog -Write:$script:JVLogWrite -LogPath $script:JV …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | [SSNI-776] [Get-JavlibraryUrl] Error occured on [GET] on URL
     | [https://www.javlibrary.com/en/vl_searchbyid.php?keyword=SSNI-776]:  {   "status": "error",   "message": "Error:
     | Error solving the challenge. Timeout after 60.0 seconds.",   "startTimestamp": 1721968550334,   "endTimestamp":
     | 1721968611638,   "version": "3.3.21" }
WARNING: [SSNI-776] [Get-JavlibraryUrl] not matched on JavLibrary

Any suggestions on how I could fix this?

RabidWolf commented 2 months ago

FlareSolverr doesn't work for anyone at the minute as cloudfare has pushed another update but FlareSolverr team are working on it so it'll take sometime.