Download interrupted without error

Greg-Lim commented 3 years ago

Download will freeze and the program will be exited with out any warnings or error. Episode 11 (1/3) [==......................] 1478340/bps 7% 102.6s C:\Users\auser\folder>

Could be due to loading too fast.

node version v14.15.1 I accidentally closed the issue.

vignedev commented 3 years ago

Just to update on the situation, currently I'm experimenting in the dev branch, which employs a different method to downloading episodes which is able to handle sudden interrupts and retry after a timeout.

It is a rushed-out solution so I'm not sure about pushing it to master, but from my short period of testing it seems to be working. I welcome anyone to test the dev branch.

Ragnyll commented 3 years ago

@vignedev i built the dev branch locally and tested it. Few things.

You need to add ansi-colors to the package.json . It might be some weird undeclared dependency issue. Running without this in the pakcage.json produces the following error:


❯ ./twist-dl --help                                                                                                                                                                                                            [00:41:30]
node:internal/modules/cjs/loader:927
throw err;
^

Error: Cannot find module 'ansi-colors' Require stack:

/twist-dl/index.js
//twist-dl/bin/twist-dl at Function.Module._resolveFilename (node:internal/modules/cjs/loader:924:15) at Function.Module._load (node:internal/modules/cjs/loader:769:27) at Module.require (node:internal/modules/cjs/loader:996:19) at require (node:internal/modules/cjs/helpers:92:18) at Object.
(/twist-dl/index.js:2:29) at Module._compile (node:internal/modules/cjs/loader:1092:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10) at Module.load (node:internal/modules/cjs/loader:972:32) at Function.Module._load (node:internal/modules/cjs/loader:813:14) at Module.require (node:internal/modules/cjs/loader:996:19) { code: 'MODULE_NOT_FOUND', requireStack: [ '/twist-dl/index.js', '/twist-dl/bin/twist-dl' ] }

Secondly: the retry works heres my output on success

❯ twist-dl -a 're-zero-kara-hajimeru-isekai-seikatsu-2nd-season-part-2' -e 1 -o .  

Re:Zero kara Hajimeru Isekai Seikatsu 2nd Season Part 2

Episode 1 (1/1)  [=.......................] 1175389/bps 4% 196.2s
error:  Failed to download episode 1 (retrying in 5s)
Error: read ECONNRESET
at TLSWrap.onStreamRead (node:internal/stream_base_commons:211:20) {
errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Episode 1 (1/1) (continuation) [=====...................] 4759669/bps 21% 37.8s
error:  Failed to download episode 1 (retrying in 5s)
Error: read ECONNRESET
at TLSWrap.onStreamRead (node:internal/stream_base_commons:211:20) {
errno: -104,
code: 'ECONNRESET',
syscall: 'read'
}
Episode 1 (1/1) (continuation) [========================] 9000821/bps 100% 0.0s
>>> elapsed time 59s

I'm gonna try this on a longer video (like movie length) and look at the code a bit. It's kinda crazy if this is the most graceful way of handling it.

So... fix works, but I dunno if you wanna push it out or not. I'd say push it just so people have something thats working, it can be polished after. In case it matters I'm running this on an Arch Linux host. I doubt this is a platform specific issue though

Ragnyll commented 3 years ago

linking @vignedev comment on this previously logged issue for reference : link

Their servers have started to interrupt the downloads after some time. The issue is that for whatever reason, node-fetch does not report the premature ending and shuts down the entire Node process. I'll investigate further on what the options are.

The current possible workaround, although not perfect in any way, is to run the program multiple times, since twist-dl is able to restore partial downloads.

which is what this current dev branch is effectively doing

vignedev commented 3 years ago

Hey there,

firstly, thank you for testing out the dev branch. Secondly, I feel very embarrassed about the fact that the ansi-colors package was not listed as a dependency despite the fact that it the project has been around for 2 years, so thank you again for reporting it.

Back to the issue at hand: from my testing the download would sometimes hang indefinitely for no reason, so I've abandoned the dev branch and shifted focus towards the fetch3 which uses the beta version of the node-fetch which actually reports premature ends and I've also implemented a download timeout. If it is not much of a bother, can you test this branch as well?

I'll be most likely merging fetch3 into master at the end of this week to push twist-dl back into a working state.

Ragnyll commented 3 years ago

Checked out the fetch3 branch. For some reason the twist-dl file that gets dropped in twist-dl/bin was not executable right off the bat. I just ran a chmod +x and it executed, but it could not fetch anything. I let it run for 5 minutes and the error just repeated

 ❯ bin/twist-dl                                                                                           [13:48:52]
✔ Pick your anime: · [object Object]
✔ Select episodes: (Re:Zero kara Hajimeru Isekai Seikatsu) · Episode 22

  twist-dl is currently under developement, if any problems occur, please submit an issue on the GitHub's repo.
  Remember:  If you have some money to spare, donate it to twist.moe so they can the servers up and running!

  They're 635$ short on money this month.
  To donate, please visit https://twist.moe/

  Re:Zero kara Hajimeru Isekai Seikatsu

error:  Failed to download episode 22, retrying in 5s
 FetchError: request to https://cdn.twist.moe/anime/rezero/[HorribleSubs]%20Re%20Zero%20kara%20Hajimeru%20Isekai%20Seikatsu%20-%2021%20[1080p].mp4 failed, reason: connect ECONNREFUSED 185.83.214.240:443
    at ClientRequest.<anonymous> (/twist-dl/node_modules/node-fetch/dist/index.cjs:1277:11)
    at ClientRequest.emit (node:events:378:20)
    at TLSSocket.socketErrorListener (node:_http_client:462:9)
    at TLSSocket.emit (node:events:378:20)
    at emitErrorNT (node:internal/streams/destroy:188:8)
    at emitErrorCloseNT (node:internal/streams/destroy:153:3)
    at processTicksAndRejections (node:internal/process/task_queues:81:21) {
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED',
  erroredSysCall: 'connect'
}

I tried this with 2 different shows (jujutsu kaisen ep 23 and from the logs rezero ep 22.

i'm now getting connection refused on the dev branch too.

 ❯ ./bin/twist-dl                                                                                         [13:52:37]
✔ Pick your anime: · [object Object]
✔ Select episodes: (Jujutsu Kaisen) · Episode 23

  twist-dl is currently under developement, if any problems occur, please submit an issue on the GitHub's repo.
  Remember:  If you have some money to spare, donate it to twist.moe so they can the servers up and running!

  They're 635$ short on money this month.
  To donate, please visit https://twist.moe/

  Jujutsu Kaisen

node:events:355
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 185.83.214.240:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1137:16)
Emitted 'error' event on NativeWrapper instance at:
    at ClientRequest.<anonymous> (/home/ragnyll/dev/twist-dl/lib/NativeWrapper.js:16:46)
    at ClientRequest.emit (node:events:378:20)
    at TLSSocket.socketErrorListener (node:_http_client:462:9)
    at TLSSocket.emit (node:events:378:20)
    at emitErrorNT (node:internal/streams/destroy:188:8)
    at emitErrorCloseNT (node:internal/streams/destroy:153:3)
    at processTicksAndRejections (node:internal/process/task_queues:81:21) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '185.83.214.240',
  port: 443
}

I think it may be worth contacting the twist server owners again. I dunno if they are blocking this.

Ragnyll commented 3 years ago

hmm i rebuilt it and ran it again just a minute ago and ran succesfully without the connection refused. I dont know what changed, but in whatever the current stat is, it seems to be working

vignedev / twist-dl

Download interrupted without error #21