tharyckgusmao / Skill-Share-Crawler---DL

Download Videos Skill Share per ID or per Class
125 stars 33 forks source link

Category download not working. Weird behavior. #2

Closed noobcoder2196 closed 7 years ago

noobcoder2196 commented 7 years ago

So at first I had some issues logging in with email and password because google kept asking me for reCAPTCHA so I tried logging in with facebook and it worked. Now, I tried downloading classes from the graphic design category and this happened:

skillshare-dl-bug2

skillshare-dl-bug1

It first repeatedly kept making a download list of one single course from that category (that too incomplete, with only 4 videos.) and then finally that same course with the right number of videos. The error log is included below for your ease:

> internal/streams/legacy.js:59
>       throw er; // Unhandled stream error in pipe.
>       ^
> 
> Error: Parse Error
>     at Socket.socketOnData (_http_client.js:440:20)
>     at emitOne (events.js:116:13)
>     at Socket.emit (events.js:211:7)
>     at addChunk (_stream_readable.js:263:12)
>     at readableAddChunk (_stream_readable.js:250:11)
>     at Socket.Readable.push (_stream_readable.js:208:10)
>     at TCP.onread (net.js:594:20)
> npm ERR! code ELIFECYCLE
> npm ERR! errno 1
> npm ERR! skillshare_dl@1.0.0 start: `node index.js`
> npm ERR! Exit status 1
> npm ERR!
> npm ERR! Failed at the skillshare_dl@1.0.0 start script.
> npm ERR! This is probably not a problem with npm. There is likely additional log
> ging output above.
> 
> npm ERR! A complete log of this run can be found in:
> (directory removed)

Any idea what's going on here and a possible fix? Also if it's possible to not terminate the login session each time the program returns an error, that'd be great. It's very tedious to keep logging in back again every time something goes wrong :(

tharyckgusmao commented 7 years ago

I have a problem with google captcha that still can not handle, the same does not appear to be resolved. It only works using facebook.

Repeated download streams are not bugs. To animate myself on the console I need to repeat a piece of information with new values.

The files of videos are listing in sequence, after resolved by the crawler in a moment all the videos appeared in order.

The animation on the console works something like this:

--------------------------
Video 1 - 0%
--------------------------
Video 1 - 1%
--------------------------

It may be happening resize from the console. So not following the flow of animation. But it's working.

Yes it would be easy to resolve keeping the session, and just save the cookie on file and use it again, updating.

Wait for the video to download and check the folder. And let me know if everything went well. I just tested linux so I do not know how it will work on windows.

By the video code you got something to download?

noobcoder2196 commented 7 years ago

_

> ```
> > ? What do you want to do ? 2 - Download all Videos per Classes
> > ? Select Classes Download Skill Share:    !! RED COLOR GLOBAL CATEGORY Illustra
> > tion, Music Production
> > Downloading  0 ... Illustration-Learning How to Draw: A Mindset, Method, and Exe
> > rcises
> > fs.js:885
> >   return binding.mkdir(pathModule._makeLong(path),
> >                  ^
> > 
> > Error: ENOENT: no such file or directory, mkdir 'C:\skillshare-dl\videos\Illustr
> > ation\Learning How to Draw: A Mindset, Method, and Exercises'
> >     at Object.fs.mkdirSync (fs.js:885:18)
> >     at module.exports (C:\skillshare-dl\app\utils\createFolder.js:20:8)
> >     at dlPerId (C:\skillshare-dl\app\components\DlPerId.js:22:7)
> >     at DlPerId (C:\skillshare-dl\app\components\DlPerId.js:160:3)
> >     at Request.request [as _callback] (C:\skillshare-dl\app\components\GetAllVid
> > eos.js:109:17)
> >     at Request.self.callback (C:\skillshare-dl\node_modules\request\request.js:1
> > 88:22)
> >     at emitTwo (events.js:126:13)
> >     at Request.emit (events.js:214:7)
> >     at Request.<anonymous> (C:\skillshare-dl\node_modules\request\request.js:117
> > 1:10)
> >     at emitOne (events.js:116:13)
> > npm ERR! code ELIFECYCLE
> > npm ERR! errno 1
> > npm ERR! skillshare_dl@1.0.0 start: `node index.js`
> > npm ERR! Exit status 1
> > npm ERR!
> > npm ERR! Failed at the skillshare_dl@1.0.0 start script.
> > npm ERR! This is probably not a problem with npm. There is likely additional log
> > ging output above.
> > 
> > npm ERR! A complete log of this run can be found in:
> > npm ERR!     C:\Users\izzy\AppData\Roaming\npm-cache\_logs\2017-11-05T19_07_52_8
> > 67Z-debug.log
> > 
> ```

_

I think you need to remove the semi-colon from the file or folder name. It messes up with window's naming conventions. My advice is to ditch every space and semicolon for a dash. That way, it'll keep windows and linux name conventions satisfied.

noobcoder2196 commented 7 years ago

Okay, so I tried running on linux lite (using vmware) and it ran perfectly...I chose the graphic design category and it was downloading smoothely but towards the end some videos were left incomplete and weren't being completed (no % was being added in the downloader) I wonder what caused this? Could it be the login becoming expired or something else?

tharyckgusmao commented 7 years ago

Hmm ... I use an arch linux environment, everything is very strange on windows for me.

I performed about 1TB of download and did not get any problems.

I think it's not a session problem, because you continue to use the means of the application. It may be that they updated the platform by preventing sequential access to the videos. I need to try downloading the videos in question to review. Please inform me by which category the problem occurred, you can also check if the json file was created for the video that gave problem.

videofolder -> tmp -> videoname .json

noobcoder2196 commented 7 years ago

Category was graphic design and yes, the json file was created. A total of 3-4 videos were left hanging without any progress.