"Downloading from a file" option does not work

connecteev commented 1 year ago

When I select "yes" to the first 2 options ("Do you want all courses?" and "Do you want download from a file"), the script immediately ends with "Downloaded all videos for 'courses' api! ", but doesn't do anything.

Logs:

✔ Do you want all courses? … yes ✔ Do you want download from a file … yes found some files: 4 in folder: /opt/homebrew/lib/node_modules/ccdown/json ✔ Enter a file path eg: /opt/homebrew/lib/node_modules/ccdown/json/*.json › search-courses.json ✔ Enter email … valid_email@gmail.com ✔ Enter password … **** ✔ Enter a directory to save a file (eg: /Users/kp/Code/web/_course_downloaders) … /Users/kp/Code/web/_course_downloaders/videos ✔ Login... completed. ✔ Downloaded all videos for 'courses' api! (total: 17)

Attached is the nuxt.json file (containing all codecourse courses in the "nuxt.js" category: https://codecourse.com/subjects/nuxt-js/index . I placed the file (attached) at /opt/homebrew/lib/node_modules/ccdown/json/nuxt.json: (Note that I have renamed the file to .txt because github won't let me upload json files)

Attached: nuxt.json.txt

connecteev commented 1 year ago

Ideally the json file with work ONLY with the course links - the course names aren't really needed. Another file format option (instead of a json file) could be a plain text file, with each line representing a course URL.

muhamed-didovic commented 1 year ago

Basically option 'Download from a file' means I have the links and etc so don't hit API anymore, but it is misleading as I can see, I am thinking to remove it in order not to confuse user?

connecteev commented 1 year ago

Ah I see, yes - that's confusing! I actually think an option to download ONLY the courses that I want from a file that I feed ccdown would be a nice touch! Think of this use case:

Step 1: I ask the tool to get me a text list of all every course available (maybe this can be filtered by courses within a subject only, maybe this list can be sorted by newest / latest course)
Step 2: I manually create a new text file for ONLY the courses I want to download, and feed it to ccdown
Step 3: ccdown has an option to prompt for a file, and ccdown only downloads the course videos (and code, if available) for all courses on that list.

This use-case actually brings everything (#4, #5, #6) together and makes the experience all the better (huge time savings).

In the future I could run step 1 again to get a fresh list of courses, compare with my old list, create a new list with only the new courses I want to download (step 2), and then feed it to ccdown (step 3). It would make the workflow amazing.

Thanks again for your work on this!

muhamed-didovic commented 1 year ago

Hey @connecteev, regarding your points:

Step 1: I ask the tool to get me a text list of all every course available (maybe this can be filtered by courses within a subject only, maybe this list can be sorted by newest / latest course) This is already done with search-courses.json file, which is either updated by me or the user if he chooses so.

As I understand, the main point here is that I allow multiple/single course(es) download(s), maybe I can add some kind of 'list' for desired courses to be downloaded

muhamed-didovic commented 1 year ago

I pushed new version as well in order to reduce the noise of logging

connecteev commented 1 year ago

Thanks @muhamed-didovic, yes - a 'list' for desired courses to be downloaded would be great.

connecteev commented 1 year ago

Btw I pulled the latest version and it gives me this error now:

Also tried uninstalling and re-installing, and see the same problem.

muhamed-didovic commented 1 year ago

ah, I see, pushed new version, please verify

connecteev commented 1 year ago

I'm still seeing the error

connecteev commented 1 year ago

Actually it's a different one now after a fresh installation

muhamed-didovic commented 1 year ago

I pushed some fixes please pull new version and verify

connecteev commented 1 year ago

Thanks @muhamed-didovic Still seeing the same error...but there is a workaround. It looks like the error happens because it cannot find the json/search-courses.json file.

~/Code/web/_course_downloaders
$ ccdown
node:internal/modules/cjs/loader:1070
  const err = new Error(message);
              ^

Error: Cannot find module '/Users/kp/Code/web/_course_downloaders/json/search-courses.json'
Require stack:
- /opt/homebrew/lib/node_modules/ccdown/lib/helpers/search.cjs
    at Module._resolveFilename (node:internal/modules/cjs/loader:1070:15)
    at Module._load (node:internal/modules/cjs/loader:923:27)
    at Module.require (node:internal/modules/cjs/loader:1137:19)
    at require (node:internal/modules/helpers:121:18)
    at Object.<anonymous> (/opt/homebrew/lib/node_modules/ccdown/lib/helpers/search.cjs:6:16)
    at Module._compile (node:internal/modules/cjs/loader:1255:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1309:10)
    at Module.load (node:internal/modules/cjs/loader:1113:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:165:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:192:25)
    at async DefaultModuleLoader.import (node:internal/modules/esm/loader:246:24)
    at async loadESM (node:internal/process/esm_loader:40:7)
    at async handleMainPromise (node:internal/modules/run_main:66:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/opt/homebrew/lib/node_modules/ccdown/lib/helpers/search.cjs' ]
}

Node.js v20.3.0

If I create the json file manually from https://github.com/muhamed-didovic/ccdown/blob/main/json/search-courses.json and put it at json/search-courses.json it runs fine. It's not a big deal but just want to point out that previously the tool did not need the json file to be within the current directory.

connecteev commented 1 year ago

2 questions for you:

How can I ask the tool to re-create the json file with the latest list of courses on codecourse.com? What options in the ccdown tool do I need to select for that?
If I want to download just 5 select courses, is there a way to do that? Can I modify the json file to only download videos for 5 courses?

muhamed-didovic commented 1 year ago

Just to confirm which version are you running with command: ccdown -v I think I know what is the issue

connecteev commented 1 year ago

ccdown -v 0.19.4

muhamed-didovic commented 1 year ago

ok, thanks.

So to answer you questions:

To get new courses just follow this screen:
I am working on the 'list' thing for you so you can select which courses you want to download, probably tomorrow you should get it

connecteev commented 1 year ago

@muhamed-didovic thanks for clarifying, and sounds amazing, thank you!

On the list functionality, just one thought...I expect to have to download all courses under a topic, like laravel has 187. https://codecourse.com/subjects/laravel/index So, it would probably be best to allow a file to be ingested (versus a UI to select which courses to download..which would make it an unusable experience). Thanks for working on this feature and this great tool.

connecteev commented 1 year ago

For example, would be good to get this option to work with this attached file containing 2 courses: search-courses.json.txt

$ ccdown ✔ Do you want all courses? … yes ✔ Do you want download from a file … yes ✔ Enter a file path eg: /opt/homebrew/lib/node_modules/ccdown/json/*.json › search-courses.json ✔ Enter email … valid_email@gmail.com ✔ Enter password … ***** ✔ Enter a directory to save a file (eg: /Users/kp/Code/web/_course_downloaders) … /Users/kp/Code/web/_course_downloaders/videos ✔ Login... completed. MAIN errorr in cli.js: ReferenceError: require is not defined at all (file:///opt/homebrew/lib/node_modules/ccdown/lib/index.js:25:19)

Here is the attached search-courses.json file I used (containing 2 courses): search-courses.json.txt

muhamed-didovic commented 1 year ago

@connecteev pull new version, I think now it should be ok, also I am almost done with 'list' feature

muhamed-didovic commented 1 year ago

also pushed autocomplete-multi select for courses :)

connecteev commented 1 year ago

@muhamed-didovic Thanks for the quick turnaround!

I upgraded ccdown:

$ npm upgrade -g ccdown
changed 1 package in 3s
233 packages are looking for funding
run `npm fund` for details

$ ccdown -v
0.19.7

This option works okay: $ ccdown ✔ Do you want all courses? … yes ✔ Do you want download from a file … no

But this option errors out: $ ccdown ✔ Do you want all courses? … yes ✔ Do you want download from a file … yes ✔ Enter a file path eg: /opt/homebrew/lib/node_modules/ccdown/json/*.json › my_courses.json ✔ Enter email … valid_email@gmail.com ✔ Enter password … ***** ✔ Enter a directory to save a file (eg: /Users/kp/Code/web/_course_downloaders) … /Users/kp/Code/web/_course_downloaders/videos ✔ Login... completed. MAIN errorr in cli.js: ReferenceError: require is not defined at all (file:///opt/homebrew/lib/node_modules/ccdown/lib/index.js:25:19)

muhamed-didovic commented 1 year ago

Sorry, forgot to publish new version, which is: 0.20.0 If you want to see new multi-select, so when you are asked: 'Do you want all courses?' choose 'no'

connecteev commented 1 year ago

@muhamed-didovic Just tried ccdown version 0.20.0, and multi-select seems to work....

ccdown ✔ Do you want all courses? … no ✔ Choose "Y" if you want to search for a course otherwise choose "N" if you have a link for download … yes ✔ Do you want to search for a courses from a local file (which is faster) … no ? Search for a course › - Space to select. Return to submit Instructions: ↑/↓: Highlight option ←/→/[space]: Toggle selection [a,b,c]/delete: Filter choices enter/return: Complete answer

Filtered results for: Enter something to filter

◯ PHP Security ◯ Cloud Image Upload with Laravel ◯ Flexible flash notifications in Laravel ◯ Realtime chat with Pusher ◯ Database Models with PDO ◯ Social Network ◯ Realtime notifications with Pusher ◉ Send Email With PHPMailer ◉ The SOLID Design Principals ◯ The PHP Reflection API

connecteev commented 1 year ago

@muhamed-didovic

While the above UI works, my problem is this: I expect to have to download many (but probably not all) courses under a subject ("laravel" has 187: https://codecourse.com/subjects/laravel/index )

For example, if I have already downloaded, say 20 courses under "laravel", I may want a specific list of courses (say 50 under the subject "laravel", 5 under the subject "vue"). Multi-select makes it very tedious to find and select dozens of courses - it's not very practical.

Can we allow a file to be ingested (versus a UI to select which courses to download..which currently makes it an unusable experience)?

Ideally I could give ccdown a file with (ex: 127) courses, similar to the sample I have attached (easiest would be one line per course URL, though a Json format would also work fine) my_courses_preferred_format.txt

muhamed-didovic commented 1 year ago

hey @connecteev, made something for you regarding the download over subjects, here is the example: ✔ Do you want all courses? … no ✔ Choose "Y" if you want to search for a course otherwise choose "N" if you have a link for download … no ✔ Enter url for download. … https://codecourse.com/subjects/flutter

connecteev commented 1 year ago

@muhamed-didovic thanks! this works great and is going to save a ton of time. I closed out https://github.com/muhamed-didovic/ccdown/issues/4

connecteev commented 1 year ago

@muhamed-didovic Any chance you can take a look at this? Thanks for this useful package!

muhamed-didovic / ccdown

"Downloading from a file" option does not work #3