Unable to parse course videos from JSON

[rafiulnakib]

Hi,

I was hoping you could help me with this. I followed your method and could download videos of one course. However, while trying to download videos of another course, I got the following error.

Can you please show me any work around?

Thanks.

[soraxas]

Can you post the run with the debug flag and post the log file somewhere? aka. run with

./run.sh URL --debug

Thanks!

[rafiulnakib]

I just did, same thing. :/

[soraxas]

Can you post the log file that generates in the same folder for debugging? It was the purpose for my last comment, thanks!

[rafiulnakib]

Sorry my bad, I didn't understand what you meant and I still don't know where to look for it. I could find one debug.log for pip, not sure if that's the one. I am not very proficient in these matters so if you could help a bit that would be awesome! Thanks.

[soraxas]

No worries! It should be named as echo360Downloader.log in the same folder as the run.sh script.

Like the following: You can upload to any sort of filehosting website or https://pastebin.com/

[rafiulnakib]

[DEBUG: 12-04 19:13] main binary_downloader link: ('https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-macosx.zip', 'phantomjs-2.1.1-macosx.zip'), bin path: /Users/raf/Downloads/echo360-master/bin/phantomjs-2.1.1-macosx/bin/phantomjs [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session {"capabilities": {"alwaysMatch": {"browserName": "phantomjs", "platformName": "any"}, "firstMatch": [{}]}, "desiredCapabilities": {"platform": "ANY", "browserName": "phantomjs", "version": "", "phantomjs.page.settings.userAgent": "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25", "javascriptEnabled": true}} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session HTTP/1.1" 200 699 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/url {"url": "https://view.streaming.sydney.edu.au:8443/ess/portal/section/2018_S2C_PHYS3069_ND", "sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/url HTTP/1.1" 200 74 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element {"using": "xpath", "sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25", "value": "//*[contains(@id,'username')]"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element HTTP/1.1" 500 14615 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source HTTP/1.1" 200 4918 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source HTTP/1.1" 200 4918 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] echo360.downloader No username found (no need to login?) [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source HTTP/1.1" 200 4918 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] echo360.downloader Dumping login page at https://view.streaming.sydney.edu.au:8443/ess/portal/section/2018_S2C_PHYS3069_ND: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">

   <meta http-equiv="Content-type" content="text/html;charset=UTF-8">
   <meta http-equiv="X-UA-Compatible" content="IE=edge">
   <!--[if lt IE 9]>
       <META HTTP-EQUIV="CACHE-CONTROL" CONTENT="NO-CACHE">
       <META HTTP-EQUIV="PRAGMA" CONTENT="NO-CACHE">
   <![endif]-->

  <style type="text/css">

     #verticallyCentered {
        color: black;
        background-color: transparent;
        text-align: center;
        position: absolute;
        top: 50%;
        left: 0px;
        width: 100%;
        height: 1px;
        overflow: visible;
        visibility: visible;
        display: block;
     }

     #horizontallyCentered {
        font-family: Verdana, Geneva, Arial, sans-serif;
        background-color: transparent;
        margin-left: -125px;
        position: absolute;
        top: -35px;
        left: 50%;
        width: 250px;
        height: 70px;
        visibility: visible
     }

        html, body, div, iframe { margin:0; padding:0; height:100%; }
        iframe { display:block; width:100%; border:none; }
   </style>
  <link rel="shortcut icon" href="/ess/image/favicon.ico">

</head>
<body>
<div>

</div>

[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/url {"url": "https://view.streaming.sydney.edu.au:8443/ess/client/api/sections/2018_S2C_PHYS3069_ND/section-data.json?pageSize=100", "sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/url HTTP/1.1" 200 74 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source HTTP/1.1" 200 238 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] echo360.course Dumping course page at https://view.streaming.sydney.edu.au:8443/ess/client/api/sections/2018_S2C_PHYS3069_ND/section-data.json?pageSize=100:

{"errorMessage":"Invalid user session.","error":401}

[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element {"using": "tag name", "sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25", "value": "pre"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element HTTP/1.1" 200 104 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element/:wdc:1543911215300/text {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25", "id": ":wdc:1543911215300"} [DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281 [DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element/:wdc:1543911215300/text HTTP/1.1" 200 132 [DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request

[soraxas]

Hi @rafiulnakib the updated script should works now, please delete the entire old folder, redownload it from Github, and try again :) please star this if it helps you too. Cheers

[rafiulnakib]

Worked perfectly! Thanks.