search

KevCui / soaper-dl

:tv: Soaper downloader: download TV series and movies in your terminal
Do What The F*ck You Want To Public License
70 stars 19 forks source link

Wait's forever for cookie #22

Closed EricMarcantonio closed 1 year ago

EricMarcantonio commented 2 years ago
Screen Shot 2022-08-01 at 12 44 54 PM

Won't load.

KevCui commented 2 years ago

Hey @EricMarcantonio, it might be caused by the invalid user agent. Could you please try to remove user-agent file in the script directory and run the script again? It will generate a new file with new user agent inside it. If it's still not working, please send me the generated user agent value and I will take another look.

valtov commented 2 years ago

I tried running the script on Ubuntu server with headless chrome

Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4237.100 Safari/537.36

Same issue as above, although I'm guessing for headless the issue can't be helped

KevCui commented 2 years ago

Hey @valtov, thanks for the info. I tried the same user agent in your comment and it appears to be working for me.

Could you please check if you are able to visit the site https://soap2day.ac and access to any video content from Chrome browser on your Ubuntu machine? If yes, then please try to edit bin/getCookie.js file line 12: headless: true to headless: false. Save the change and run the script, it will start Chrome without headless and the cookies should be fetched.

valtov commented 2 years ago

Like I said I'm on Ubuntu server, all I see is a command line. I ran this google-chrome-stable --headless --disable-gpu --screenshot --run-all-compositor-stages-before-draw --virtual-time-budget=10000 https://soap2day.ac to see what's happening and its asking to click to 'verify human' which I don't think I can do. I'm guessing the only way forward (assuming I want it to work from my server) is to steal the cookies from my regular browser and paste them in somehow in your script.

KevCui commented 2 years ago

Hey @valtov, the script is using Puppeteer to run headless Chrome, and it should work on Ubuntu server as well, even there is only CLI.

You could run this command bin/getCookie.js "$(which google-chrome-stable)" https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4237.100 Safari/537.36' to see if it returns an error when fetching cookie.

If you'd like to investigate further, you could add await page.screenshot({ path: 'screenshot.png' }); in bin/getCookie.js file to capture a screenshot and see what's blocking the process.

azeem39 commented 2 years ago

I am having the same issue on ubunut VPS and after i run "bin/getCookie.js "$(which google-chrome-stable)" https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4237.100 Safari/537.36'" I gee this :

internal/modules/cjs/loader.js:638 throw err; ^

Error: Cannot find module 'puppeteer-extra' at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) at Function.Module._load (internal/modules/cjs/loader.js:562:25) at Module.require (internal/modules/cjs/loader.js:692:17) at require (internal/modules/cjs/helpers.js:25:18) at Object. (/home/soap2day-dl/bin/getCookie.js:4:19) at Module._compile (internal/modules/cjs/loader.js:778:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10) at Module.load (internal/modules/cjs/loader.js:653:32) at tryModuleLoad (internal/modules/cjs/loader.js:593:12) at Function.Module._load (internal/modules/cjs/loader.js:585:3)

KevCui commented 2 years ago

Hey @azeem39, from the error message in your comment Cannot find module 'puppeteer-extra', you are having a different issue. Please make sure the required npm packages are installed on your VPS: cd bin; npm i puppeteer-extra puppeteer-extra-plugin-stealth

azeem39 commented 2 years ago

Thank you for that, when i run ./soap2day-dl.sh -p /tv_aTo2OTs.html -e 1.1 -d i get this

cyber-hunt7 commented 2 years ago

Good day,

If I may ask where do you get this or how do you get this.

aTo2OTs.html

From: azeem39 @.> Sent: Friday, August 19, 2022 6:58 PM To: KevCui/soap2day-dl @.> Cc: Subscribed @.***> Subject: Re: [KevCui/soap2day-dl] Wait's forever for cookie (Issue #22)

Thank you for that, when i run ./soap2day-dl.sh -p /tv_aTo2OTs.html -e 1.1 -d i get this

— Reply to this email directly, view it on GitHub https://github.com/KevCui/soap2day-dl/issues/22#issuecomment-1220891963 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AIPD3QILP6XIHBN7AJUSDNTVZ64JDANCNFSM55IGIBDA . You are receiving this because you are subscribed to this thread. https://github.com/notifications/beacon/AIPD3QNVMX3XKIHFIIHI7ILVZ64JDA5CNFSM55IGIBDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOJDCVKOY.gif Message ID: @. @.> >

azeem39 commented 2 years ago

that's the URL of the TV-series - i copied from the example section

cyber-hunt7 commented 2 years ago

Is there a way you can showcase a demo of a series. Any series

From: azeem39 @.> Sent: Friday, August 19, 2022 7:35 PM To: KevCui/soap2day-dl @.> Cc: Cybersecghost @.>; Comment @.> Subject: Re: [KevCui/soap2day-dl] Wait's forever for cookie (Issue #22)

that's the URL of the TV-series - i copied from the example section

— Reply to this email directly, view it on GitHub https://github.com/KevCui/soap2day-dl/issues/22#issuecomment-1220923753 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AIPD3QO5UBU2AEXA7YVLIH3VZ7AV7ANCNFSM55IGIBDA . You are receiving this because you commented. https://github.com/notifications/beacon/AIPD3QM2DABW5CGEXGEXVUTVZ7AV7A5CNFSM55IGIBDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOJDC5C2I.gif Message ID: @. @.> >

KevCui commented 2 years ago

Thank you for that, when i run ./soap2day-dl.sh -p /tv_aTo2OTs.html -e 1.1 -d i get this

  • getopts :hlsdnxp:e: opt
  • set_var ++ command -v curl
  • _CURL=/usr/bin/curl ++ command -v jq
  • _JQ=/usr/bin/jq ++ command -v pup
  • _PUP=/usr/local/bin/pup ++ command -v fzf
  • _FZF=/usr/bin/fzf ++ command -v chromium
  • _CHROME=/snap/bin/chromium
  • _HOST=https://soap2day.ac
  • _SEARCH_URL=https://soap2day.ac/search/keyword/ +++ realpath ./soap2day-dl.sh ++ dirname /home/soap2day-dl/soap2day-dl.sh
  • _SCRIPT_PATH=/home/soap2day-dl
  • _SEARCH_LIST_FILE=/home/soap2day-dl/search.list
  • _SOURCE_FILE=.source.html
  • _EPISODE_LINK_LIST=.episode.link
  • _EPISODE_TITLE_LIST=.episode.title
  • _MEDIA_HTML=.media.html
  • _SUBTITLE_LANG=English
  • _GET_RESPONSE_JS=/home/soap2day-dl/bin/getResponse.js
  • _FETCH_FILE_JS=/home/soap2day-dl/bin/fetchFile.js
  • _COOKIE_FILE=/home/soap2day-dl/cookie.json
  • _USER_AGENT_FILE=/home/soap2day-dl/user-agent
  • _GET_COOKIE_JS=/home/soap2day-dl/bin/getCookie.js
  • [[ -s /home/soap2day-dl/user-agent ]] ++ cat /home/soap2day-dl/user-agent
  • _USER_AGENT='Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, lik e Gecko) Chrome/92.0.4393.100 Safari/537.36' ++ get_cookie +++ is_file_expired /home/soap2day-dl/cookie.json 55 +++ local o +++ o=yes +++ [[ -f /home/soap2day-dl/cookie.json ]] +++ echo yes ++ [[ yes == \y\e\s ]] ++ local cookie

Hey @azeem39, unfortunately, the posted log appears to be incomplete. The log doesn't show the script bin/getCookie.js execution part. I cannot know what went wrong on your machine. If you have more logs to share, I can take another look for sure.

WolfgangDpunkt commented 2 years ago

Same here...I tried every solution advised. but always getting only endless lines of "[INFO] Wait a few seconds for fetching cookie..." (Debian/Raspberry PI OS Lite 64-bit)

when i run ./soap2day-dl.sh -p /tv_aTo2OTs.html -e 1.1 -d i get this:

[INFO] Wait a few seconds for fetching cookie... +++ /root/soap2day/soap2day-dl/bin/getCookie.js /usr/bin/chromium https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4406.100 Safari/537.36' ++ cookie= ++ [[ -z '' ]] ++ get_cookie +++ is_file_expired /root/soap2day/soap2day-dl/cookie.json 55 +++ local o +++ o=yes +++ [[ -f /root/soap2day/soap2day-dl/cookie.json ]] +++ echo yes ++ [[ yes == \y\e\s ]] ++ local cookie ++ print_info 'Wait a few seconds for fetching cookie...' ++ printf '%b\n' '\033[32m[INFO]\033[0m Wait a few seconds for fetching cookie...' [INFO] Wait a few seconds for fetching cookie... +++ /root/soap2day/soap2day-dl/bin/getCookie.js /usr/bin/chromium https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4406.100 Safari/537.36' ++ cookie= ++ [[ -z '' ]] ++ get_cookie +++ is_file_expired /root/soap2day/soap2day-dl/cookie.json 55 +++ local o +++ o=yes +++ [[ -f /root/soap2day/soap2day-dl/cookie.json ]] +++ echo yes ++ [[ yes == \y\e\s ]] ++ local cookie

KevCui commented 2 years ago

Hey @WolfgangDpunkt, could you please run /root/soap2day/soap2day-dl/bin/getCookie.js /usr/bin/chromium https://soap2day.ac 'Mozilla/5.0 (X11; CrOS aarch64 13816.64.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4406.100 Safari/537.36' and paste the logs? I need those logs for further investigations. Thanks!

WolfgangDpunkt commented 2 years ago

Thank you for your help! I have entered the command, as you told.

First realisation, apparently puppeteer wasn't installed properly after all (actually I use for other purposes already, but ok).

So after running "npm i puppeteer", I tried your command again. This time I got an error message that it was not possible to execute this command as root user.

Well, then I tried as a normal user, now I could run the cookie thing without any problems. one step achieved. but when I tried to continue using the script itself now (./soap2day-dl.sh -n 'wire') , new permission errors came up , I guess it now needs root rights again for further steps.

This looks like it needs a long trial and error, so sorry, but I give up and I'll just use a browser extension for downloading.

But thanks for trying to solve the error and for your work.

KevCui commented 2 years ago

Hey @WolfgangDpunkt, :ok_hand:, no problem of choosing the tool you prefer the most.

DutchCoder420 commented 2 years ago

same issue doesn't stop fetching cookie and all packages have been installed properly. used different user-agents but don't think that's the issue

MagnusStrom commented 2 years ago

Hello, I had this issue and installing puppeteer fixed it. However, Now it says media not found..

MagnusStrom commented 2 years ago

LOL, running .\soap2day-dl.sh doesn't work, but running soap2day-dl.sh does. Windows terimal is great. Thank you for continuing to maintain this project!

KevCui commented 2 years ago

Hey @EricMarcantonio, @valtov, @azeem39, @DutchCoder420, I just made some changes to show the JS errors in the debug logs. If the same issue still happens, could you please pull the latest commit and run the command with -d flag? Please share the logs here and I will investigate further. Thanks!

jones2126 commented 1 year ago

I had the same issue and with the various debug steps above I think I worked out the correct steps for me to get this work. Posting here in case it helps someone else. Happy to be corrected.

Make sure https://soap2day.ac/ is accessable in your browser

$ sudo apt-get install jq.
$ sudo snap install go --classic
$ export GOBIN=/usr/local/bin
$ alias goSudo='\sudo -E go'
$ go install github.com/ericchiang/pup@latest
$ which pup   # statement not required; returned /usr/local/bin/pup
$ sudo apt install fzf
$ fzf –version  # statement not required; returns 0.20.0
$ sudo apt update
$ sudo apt install chromium-browser
$ sudo apt install git
$ git clone https://github.com/KevCui/soap2day-dl.git
$ cd ~/soap2day-dl/bin
$ npm i puppeteer-core puppeteer-extra puppeteer-extra-plugin-stealth
$ cd ~/soap2day-dl
$ ./soap2day-dl.sh -n 'game of'

possible debug steps
$ ./soap2day-dl.sh -d 'game of'
delete ~/soap2day-dl/user-agent
$ sudo nano  ~/soap2day-dl/bin/getCookie.js  #line 12:  change headless: true to  headless: false.

@KevCui Thank you for sharing your work.

KevCui commented 1 year ago

Hi all, I didn't receive any further debug logs after 3 months. Unfortunately I can't investigate the reported issue without debug logs. Therefore, I'm closing this issue for now.

If you have any questions, please feel free to leave comments below or open new issues.

Thanks @jones2126 for sharing the well-written debug steps :+1: