search

yoonwaiyan / stackskills-dl

Simple Ruby script to download all StackSkills tutorials.
https://yoonwaiyan.github.io/stackskills-dl/
MIT License
59 stars 19 forks source link

Error while trying to download single course #38

Closed linux95 closed 5 years ago

linux95 commented 5 years ago

I am having issues while trying to download single course. I receive the following error: ruby stackskills_dl.rb -u "username" -p "password" -c https://stackskills.com/courses/enrolled/51763

/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- mechanize (LoadError)
    from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:1:in `<top (required)>'
    from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from stackskills_dl.rb:4:in `<main>'

Also, I have built the project from scratch on a Fedora 27 box and receive this error: ruby stackskills_dl.rb -u "username" -p "password" -c https://stackskills.com/courses/enrolled/339711 Loaded login credentials from environment variables. Login Successfully. Finding https://stackskills.com/courses/enrolled/339711 from your list of courses Unable to find this course: https://stackskills.com/courses/enrolled/339711 from your list of courses. Number of courses found: 1

/home/user/stackskills-dl/lib/course_finder.rb:37:in `block in execute': undefined method `href' for nil:NilClass (NoMethodError)
    from /home/user/stackskills-dl/lib/course_finder.rb:36:in `each'
    from /home/user/stackskills-dl/lib/course_finder.rb:36:in `execute'
    from /home/user/stackskills-dl/lib/course_finder.rb:12:in `block in run'
    from /home/user/stackskills-dl/lib/utilities.rb:13:in `block in mkchdir'
    from /home/user/stackskills-dl/lib/utilities.rb:12:in `chdir'
    from /home/user/stackskills-dl/lib/utilities.rb:12:in `mkchdir'
    from /home/user/stackskills-dl/lib/course_finder.rb:11:in `run'
    from stackskills_dl.rb:11:in `<main>'

Can you provide any insight on why I can't download single courses? I have tried several variations using the -c and the -s switches. I was able to just start stackskills-dl and download the first page of my enrolled courses. Then it stopped and would not proceed to the second or third page of courses. Any help you can provide would be fantastic. Thank you.

yoonwaiyan commented 5 years ago

Hi, much thanks for reporting. Let me try to find out the cause for this two issues:

For the first issue, did you run bundle install?

As for the Fedora setup, may I have your favor to put a line p courses on line 35 of lib/course_finder.rb, execute it and paste the output of the execution on this line? I would like to observe the output for both situation where you're trying to download all courses including second page onwards, and trying to download one course using -c option to do further investigation.

linux95 commented 5 years ago

Yoonwaiyan, I edited the file for the Fedora box and below is the output:

Loaded login credentials from environment variables. Login Successfully. Finding https://stackskills.com/courses/enrolled/174879 from your list of courses Unable to find this course: https://stackskills.com/courses/enrolled/174879 from your list of courses. Number of courses found: 1

<Enumerator: [nil]:each>

I attempted to run the bundle install and received the following output:

root@dockerU:/tmp/stackskills-dl/stackskills-dl-master# bundle install Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all non-root users on this machine. Warning: the running version of Bundler is older than the version that created the lockfile. We suggest you upgrade to the latest version of Bundler by running gem install bundler. Fetching gem metadata from https://rubygems.org/.......... Fetching version metadata from https://rubygems.org/.. Using rake 12.3.0 Using ast 2.3.0 Using coderay 1.1.1 Installing unf_ext 0.0.7.2 with native extensions

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory: /var/lib/gems/2.3.0/gems/unf_ext-0.0.7.2/ext/unf_ext

/usr/bin/ruby2.3 -r ./siteconf20180819-12431-1t6vvp7.rb extconf.rb mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.3.0/gems/unf_ext-0.0.7.2 for inspection. Results logged to /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/unf_ext-0.0.7.2/gem_make.out Using highline 1.7.8 Using mime-types 2.99.1 Using net-http-digest_auth 1.4 Using net-http-persistent 2.9.4 Using mini_portile2 2.3.0 Using ntlm-http 0.1.1 Using webrobots 0.1.2 Using method_source 0.8.2 Using parallel 1.12.0 Using powerpack 0.1.1 Using slop 3.6.0 Using ruby-progressbar 1.9.0 Using unicode-display_width 1.3.0 Using bundler 1.11.2 Using rainbow 2.2.2 Using parser 2.4.0.2 An error occurred while installing unf_ext (0.0.7.2), and Bundler cannot continue. Make sure that gem install unf_ext -v '0.0.7.2' succeeds before bundling. root@dockerU:/tmp/stackskills-dl/stackskills-dl-master# gem install unf_ext -v '0.0.7.2' Building native extensions. This could take a while... ERROR: Error installing unf_ext: ERROR: Failed to build gem native extension.

current directory: /var/lib/gems/2.3.0/gems/unf_ext-0.0.7.2/ext/unf_ext

/usr/bin/ruby2.3 -r ./siteconf20180819-12442-1wg2sox.rb extconf.rb mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.3.0/gems/unf_ext-0.0.7.2 for inspection. Results logged to /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/unf_ext-0.0.7.2/gem_make.out root@dockerU:/tmp/stackskills-dl/stackskills-dl-master#

Please let me know if you need anything further and I will be happy to supply. Thus far, this has been a terrific program and I am hoping to help you successfully update so that everyone can enjoy. Thanks so much for your efforts.

yoonwaiyan commented 5 years ago

Is your Fedora box a Docker vm? If you're attempting to run a Docker maybe you can try follow my README for the commands though, just to avoid any missing dependencies. Otherwise you can google how to install missing dependencies by the error message, for example: https://stackoverflow.com/questions/20559255/error-while-installing-json-gem-mkmf-rb-cant-find-header-files-for-ruby

For this error undefined method 'href' for nil:NilClass: sorry can I have one more favor to observe your output after adding this line p all_courses after line 83 of the same file and run again? The function will look like this:

  def find_course(course_name)
    puts "Finding #{course_name} from your list of courses"
    course_href = course_name.split('/courses/').last
    p all_courses
    course = all_courses.select { |course| course.href =~ Regexp.new(course_href.to_s) }.first

    if course.nil?
      puts "Unable to find this course: #{course_name} from your list of courses."
    end

    course
  end

Apparently this course is missing from your list of courses by looking at your output. Let me know whether it's the case of the courses after first page are not listed, or you have this course listed inside your courses (the big chunk displayed after adding the line as mentioned) but the script couldn't download it. Btw I've fixed the case so that the download shouldn't begin without a course being found.

linux95 commented 5 years ago

yoowaiyan, The Fedora box is running the program in Docker. Below is the output from the addition on the code you suggested:

[user@dockerf stackskills-dl]$ ruby stackskills_dl.rb -u “username" -p “password" -c https://stackskills.com/courses/enrolled/174879 Loaded login credentials from environment variables. Login Successfully. Finding https://stackskills.com/courses/enrolled/174879 from your list of courses [#<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " How To Make 2D iPhone Games Using Swift 4 and iOS 11\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Create and Make Money With 2D iPhone Games. Designed for Beginners. Designed to Move Fast.\n" + " \n" + " \n" + " " "/courses/enrolled/159101"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " The Complete Python Course: Beginner to Advanced!\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Learn how to code in Python working on practical applications, and get started programming today!\n" + " \n" + " \n" + " " "/courses/enrolled/102831"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Master Video Game Making\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Beginners to Intermediates: Learn industry standard tools, technologies, terms and concepts. Then any tool and game engine is easily adapted to your skills.\n" + " \n" + " \n" + " " "/courses/enrolled/187722"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " The Complete Web Developer Course 2.0\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Web development from scratch in 6 weeks. Build 25 Websites using HTML 5, JS, CSS 3, PHP, MySQL, Bootstrap, Wordpress, APIs, Mobile Apps, Python all covered.\n" + " \n" + " \n" + " " "/courses/enrolled/71859"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Cisco New CCNA R/S (200-125): The Complete Course\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " This course has everything you need to prepare & pass Cisco's new CCNA R/S 200-125 certification exam\n" + " \n" + " \n" + " " "/courses/enrolled/170649"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Web application penetration testing professional - WAPTP v3.1\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Attack web application like never before with latest tools, tricks and attacks at professional level\n" + " \n" + " \n" + " " "/courses/enrolled/231518"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " NEW Cisco CCNA IPv4 Course!\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " This course is geared to enhance your knowledge & understanding of the IPv4 protocol which is still used in the workplace.\n" + " \n" + " \n" + " " "/courses/enrolled/170651"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " The Complete iOS 11 & Swift Developer Course - Build 20 Apps\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Use Xcode 9 & Swift 4 to make real apps like Uber and Instagram, with MLKit & ARKit. Includes AWS Credit and much more.\n" + " \n" + " \n" + " " "/courses/enrolled/197556"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " The Complete Cyber Security Course! Volume 2 : Network Security\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Network Security, WiFi Security, WiFi Hackers, Firewalls, Wireshark. Plus the Best Password Managers\n" + " \n" + " \n" + " " "/courses/enrolled/137514"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Cisco New CCNA CCENT / ICND1 (100-105): The Complete Course\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " The complete guide to prepare for Cisco's CCENT certification or otherwise known as the ICND1 for test # 100-105\n" + " \n" + " \n" + " " "/courses/enrolled/170652"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Learn how to make 6 games in the Unreal Engine\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Learn to make games from scratch in the popular engine Unreal 4!\n" + " \n" + " \n" + " " "/courses/enrolled/182238"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Hack People, Systems and Mobile Devices. Advanced Social Engineering\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " IT security is not just about firewalls or antiviruses, it is also about people. It's about mobile devices that we carry with us.\n" + " \n" + " \n" + " " "/courses/enrolled/230966"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Portrait Photography Masterclass\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Create Beautiful Portraits for Family, Friends, or Clients\n" + " \n" + " \n" + " " "/courses/enrolled/69803"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Game Development using Corona SDK in 2015 with ASO & Ads\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Everything you need to know about developing a game with Corona SDK.\n" + " \n" + " \n" + " " "/courses/enrolled/49333"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " JavaScript Programming: Learn by Making a Mobile Game\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Dive into JavaScript's Core Concepts By Building a Game From Scratch\n" + " \n" + " \n" + " " "/courses/enrolled/14464"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " The Ultimate Java 8 Tutorial - From Beginner to Professional\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Master All You Need to Know About Java with This Comprehensive 20-Hour Course\n" + " \n" + " \n" + " " "/courses/enrolled/55201"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Java Programming – the Master Course\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Get Premium, Step-By-Step Instruction on All Things Java\n" + " \n" + " \n" + " " "/courses/enrolled/53615"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Java Programming from Zero to Hero: Java Completed\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Jumpstart Your Java Journey in Under 4 Hours\n" + " \n" + " \n" + " " "/courses/enrolled/53326"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Node.js Security: Pentesting and Exploitation\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Learn Exploitation Development for Node.js Apps By Executing Code Reviews, Identifying Insecure Code & More\n" + " \n" + " \n" + " " "/courses/enrolled/33308"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Byte Size Chunks : Java Reflection, Annotations and Lambdas\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Master 3 Powerful Features to Take Your Java Expertise to the Next Level\n" + " \n" + " \n" + " " "/courses/enrolled/53431"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Byte Size Chunks: Java Model-View-Controller (MVC)\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Use This Design Pattern to Easily Build & Test Complex Apps, Facilitate Group Development, and More\n" + " \n" + " \n" + " " "/courses/enrolled/53432"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " From 0 to 1: Heavy Lifting with SQL & Databases\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Become a SQL Pro -- Conduct Exercises Using MySQL & SQLite\n" + " \n" + " \n" + " " "/courses/enrolled/51682"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " The White Hat Hacker Bundle\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Master Penetration Testing with 7 Courses on Hacking Node.js Apps, Windows OS, Wi-Fi Devices & More\n" + " \n" + " \n" + " " "/courses/enrolled/39908"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Intensive Ethical Hacking Series\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Turbocharge Your Ethical Hacking Skills: Master Crytography, Breach Firewalls, Attack Passwords & More\n" + " \n" + " \n" + " " "/courses/enrolled/33657"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Byte Size Chunks: Java Object-Oriented Programming & Design\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Conquer Java with Just 7 Hours of Premium Instruction\n" + " \n" + " \n" + " " "/courses/enrolled/52510"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " From 0 to 1: Data Structures & Algorithms in Java\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Master the Fundamentals of How Computer Programs Organize & Manipulate Data\n" + " \n" + " \n" + " " "/courses/enrolled/51691"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Learn Network Hacking/Penetration Testing From Scratch (WiFi & Wired)\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Uncover Vulnerabilities in Wired Networks By Mastering Practical Attacks Against Wireless Devices\n" + " \n" + " \n" + " " "/courses/enrolled/33211"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " From 0 to 1: Design Patterns - 24 That Matter - In Java\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Increase Your Coding Efficiency with 24 Tried-and-True Java Design Patterns\n" + " \n" + " \n" + " " "/courses/enrolled/51687"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " From 0 to 1: C Programming - Drill Deep\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " " "/courses/enrolled/51685"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " From 0 to 1: Learn Python Programming - Easy as Pie\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " " "/courses/enrolled/51693"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Game Developer Business & Legal Guide: Run an Indie Studio\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Navigate Business & Legal Issues without Shelling Out for a Lawyer\n" + " \n" + " \n" + " " "/courses/enrolled/49394"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Windows Exploit Development Megaprimer\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Execute Windows Software Exploitation Using Tried & True Techniques\n" + " \n" + " \n" + " " "/courses/enrolled/33311"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Learn to Code in Game Maker Language\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Quickly & Easily Develop Games Using This Popular Game Creation System\n" + " \n" + " \n" + " " "/courses/enrolled/49731"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " The Complete Android Marshmallow Development Course\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Learn By Building 5 Phone & Smartwatch Apps For the Latest Version of Android\n" + " \n" + " \n" + " " "/courses/enrolled/20929"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Master Unity By Building 6 Fully Featured Games From Scratch\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Become a Full-Fledged 2D Game Developer Using This Popular Game Engine\n" + " \n" + " \n" + " " "/courses/enrolled/48312"> , #<Mechanize::Page::Link "\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " Learn By Example: C++ Programming - 75 Solved Problems\n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " \n" + " " "/courses/enrolled/52516"> ] Unable to find this course: https://stackskills.com/courses/enrolled/174879 from your list of courses. Number of courses found: 1

<Enumerator: [nil]:each>

Also, one thing that I did notice is that the program is only capturing 39 courses (only the courses on the first page). I currently have 90 courses in My Course list. I don’t know why it is not capturing the rest of the courses. This could be why it stating that the course does not exist. However, I logged onto the site and did verify that I was enrolled in the course (I played the introduction video). Please let me know if I can provide more information. Thank you.

On Aug 20, 2018, at 10:25 AM, Yoon Wai Yan notifications@github.com<mailto:notifications@github.com> wrote:

Is your Fedora box a Docker vm? If you're attempting to run a Docker maybe you can try follow my README for the commands though, just to avoid any missing dependencies. Otherwise you can google how to install missing dependencies by the error message, for example: https://stackoverflow.com/questions/20559255/error-while-installing-json-gem-mkmf-rb-cant-find-header-files-for-ruby

For this error undefined methodhref' for nil:NilClass: sorry can I have one more favor to observe your output after adding this linep all_courses` after line 83 of the same file and run again? The function will look like this:

def find_course(course_name) puts "Finding #{course_name} from your list of courses" course_href = course_name.split('/courses/').last puts all_courses course = all_courses.select { |course| course.href =~ Regexp.new(course_href.to_s) }.first

if course.nil?
  puts "Unable to find this course: #{course_name} from your list of courses."
end

course

end

Apparently this course is missing from your list of courses by looking at your output. Let me know whether it's the case of the courses after first page are not listed, or you have this course listed inside your courses (the big chunk displayed after adding the line as mentioned) but the script couldn't download it. Btw I've fixed the case so that the download shouldn't begin without a course being found.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/yoonwaiyan/stackskills-dl/issues/38#issuecomment-414334453, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKYxWTt5XyRosjYV8GYWJWihQrjF6TiXks5uSsbPgaJpZM4V_ypz.

linux95 commented 5 years ago

yoowaiyan, So I have run a couple of tests from both an Ubuntu and Fedora installation and have discovered that the program will only read the first "page" of courses on StackSkills. I have attempted to download courses from the last "page" and received the following error: 

/tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:37:in `block in execute': undefined method `href' for nil:NilClass (NoMethodError)
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:36:in `each'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:36:in `execute'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:12:in `block in run'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/utilities.rb:13:in `block in mkchdir'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/utilities.rb:12:in `chdir'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/utilities.rb:12:in `mkchdir'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:11:in `run'
    from stackskills_dl.rb:11:in `<main>'

Is there something that I need to add to the course_finder.rb file to have it search beyond the first "page" of courses? Let me know and I will be happy to experiment. Thanks again.

yoonwaiyan commented 5 years ago

It's more to a bug where the program couldn't find the "Next" link while reading the web page, but it still works from my side. Can you please modify the navigate_pages to look like this and let me know the output:

  def navigate_pages(current_page, courses = [])
    links = current_page.links_with(href: %r{courses/enrolled/})
    courses << links

    next_page_link = current_page.link_with(text: /Next/)
    puts "~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
    p current_page
    p next_page_link
    puts "~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
    unless next_page_link.nil?
      next_page = next_page_link.click
      navigate_pages(next_page, courses)
    else
      return courses.flatten
    end
  end
linux95 commented 5 years ago

yoonwaiyan, Here is the output after the modification that you requested. Again, I tried to download the last course on the last page of the site. Please also note, that some of the courses may be part of a bundle that I purchased. I don't know if that makes a difference. Thank you.

root@dockerU:/tmp/stackskills-dl/stackskills-dl-master# ruby stackskills_dl.rb -c https://stackskills.com/courses/enrolled/345065
Login Email: email@email.com
Login password: *********
Login Successfully.
Finding https://stackskills.com/courses/enrolled/345065 from your list of courses
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#<Mechanize::Page
 {url #<URI::HTTPS https://stackskills.com/courses/enrolled>}
 {meta_refresh}
 {title "StackSkills"}
 {iframes}
 {frames}
 {links
  #<Mechanize::Page::Link
   "\n            StackSkills\n            \n          "
   "http://stackskills.com?sa=eyJ1c2VyX2lkIjo0MDkzMDA0LCJleHBpcmVzX2F0IjoxNTM0ODUzNjkxLjI2NTIyMDIsInNjaG9vbF9pZCI6MTQ1MywiYXV0aGVudGljYXRpb25fdG9rZW4iOiJCOUpnN25xTkZ6VHpQVVlEYm5La3M4dmV5YlVGVjVqYS13In0%3D--7e6c19e2bf9b092a5b3739ad78c2280cff4109ef">
  #<Mechanize::Page::Link "\n      All Courses\n    " "/courses">
  #<Mechanize::Page::Link "\n      My Courses\n    " "/courses/enrolled">
  #<Mechanize::Page::Link
   "\n      Course Bundles\n    "
   "https://stackskills.com/courses/category/Course%20Bundles?sa=eyJ1c2VyX2lkIjo0MDkzMDA0LCJleHBpcmVzX2F0IjoxNTM0ODUzNjkxLjI2ODA0OCwic2Nob29sX2lkIjoxNDUzLCJhdXRoZW50aWNhdGlvbl90b2tlbiI6IkI5Smc3bnFORnpUelBVWURibktrczh2ZXliVUZWNWphLXcifQ%3D%3D--f4b45324e9e3a8559570cd8dee20dda3e2362480">
  #<Mechanize::Page::Link "\n  \n  Name\n" nil>
  #<Mechanize::Page::Link "\n      Edit Profile\n    " "/current_user/profile">
  #<Mechanize::Page::Link
   "\n      Manage Subscriptions\n    "
   "/current_user/subscriptions">
  #<Mechanize::Page::Link
   "\n      Add / Change Credit Card\n    "
   "/current_user/credit_card">
  #<Mechanize::Page::Link "\n      Contact\n    " "/current_user/contact">
  #<Mechanize::Page::Link "\n      Log Out\n    " "/sign_out">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            How To Make 2D iPhone Games Using Swift 4 and iOS 11\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Create and Make Money With 2D iPhone Games. Designed for Beginners. Designed to Move Fast.\n          \n        \n      "
   "/courses/enrolled/159101">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            The Complete Python Course: Beginner to Advanced!\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Learn how to code in Python working on practical applications, and get started programming today!\n          \n        \n      "
   "/courses/enrolled/102831">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Master Video Game Making\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Beginners to Intermediates: Learn industry standard tools, technologies, terms and concepts. Then any tool and game engine is easily adapted to your skills.\n          \n        \n      "
   "/courses/enrolled/187722">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            The Complete Web Developer Course 2.0\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Web development from scratch in 6 weeks. Build 25 Websites using HTML 5, JS, CSS 3, PHP, MySQL, Bootstrap, Wordpress, APIs, Mobile Apps, Python all covered.\n          \n        \n      "
   "/courses/enrolled/71859">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Cisco New CCNA R/S (200-125): The Complete Course\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            This course has everything you need to prepare & pass Cisco's new CCNA R/S 200-125 certification exam\n          \n        \n      "
   "/courses/enrolled/170649">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Web application penetration testing professional - WAPTP v3.1\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Attack web application like never before with latest tools, tricks and attacks at professional level\n          \n        \n      "
   "/courses/enrolled/231518">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            NEW Cisco CCNA IPv4 Course!\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            This course is geared to enhance your knowledge & understanding of the IPv4 protocol which is still used in the workplace.\n          \n        \n      "
   "/courses/enrolled/170651">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            The Complete iOS 11 & Swift Developer Course - Build 20 Apps\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Use Xcode 9 & Swift 4 to make real apps like Uber and Instagram, with MLKit & ARKit. Includes AWS Credit and much more.\n          \n        \n      "
   "/courses/enrolled/197556">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            The Complete Cyber Security Course! Volume 2 : Network Security\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Network Security, WiFi Security, WiFi Hackers, Firewalls, Wireshark. Plus the Best Password Managers\n          \n        \n      "
   "/courses/enrolled/137514">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Cisco New CCNA CCENT / ICND1 (100-105): The Complete Course\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            The complete guide to prepare for Cisco's CCENT certification or otherwise known as the ICND1 for test # 100-105\n          \n        \n      "
   "/courses/enrolled/170652">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Learn how to make 6 games in the Unreal Engine\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Learn to make games from scratch in the popular engine Unreal 4!\n          \n        \n      "
   "/courses/enrolled/182238">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Hack People, Systems and Mobile Devices. Advanced Social Engineering\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            IT security is not just about firewalls or antiviruses, it is also about people. It's about mobile devices that we carry with us.\n          \n        \n      "
   "/courses/enrolled/230966">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Portrait Photography Masterclass\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Create Beautiful Portraits for Family, Friends, or Clients\n          \n        \n      "
   "/courses/enrolled/69803">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Game Development using Corona SDK in 2015 with ASO & Ads\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Everything you need to know about developing a game with Corona SDK.\n          \n        \n      "
   "/courses/enrolled/49333">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            JavaScript Programming: Learn by Making a Mobile Game\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Dive into JavaScript's Core Concepts By Building a Game From Scratch\n          \n        \n      "
   "/courses/enrolled/14464">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            The Ultimate Java 8 Tutorial - From Beginner to Professional\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master All You Need to Know About Java with This Comprehensive 20-Hour Course\n          \n        \n      "
   "/courses/enrolled/55201">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Java Programming – the Master Course\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Get Premium, Step-By-Step Instruction on All Things Java\n          \n        \n      "
   "/courses/enrolled/53615">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Java Programming from Zero to Hero: Java Completed\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Jumpstart Your Java Journey in Under 4 Hours\n          \n        \n      "
   "/courses/enrolled/53326">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Node.js Security: Pentesting and Exploitation\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Learn Exploitation Development for Node.js Apps By Executing Code Reviews, Identifying Insecure Code & More\n          \n        \n      "
   "/courses/enrolled/33308">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Byte Size Chunks : Java Reflection, Annotations and Lambdas\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master 3 Powerful Features to Take Your Java Expertise to the Next Level\n          \n        \n      "
   "/courses/enrolled/53431">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Byte Size Chunks: Java Model-View-Controller (MVC)\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Use This Design Pattern to Easily Build & Test Complex Apps, Facilitate Group Development, and More\n          \n        \n      "
   "/courses/enrolled/53432">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: Heavy Lifting with SQL & Databases\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Become a SQL Pro -- Conduct Exercises Using MySQL & SQLite\n          \n        \n      "
   "/courses/enrolled/51682">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            The White Hat Hacker Bundle\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master Penetration Testing with 7 Courses on Hacking Node.js Apps, Windows OS, Wi-Fi Devices & More\n          \n        \n      "
   "/courses/enrolled/39908">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Intensive Ethical Hacking Series\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Turbocharge Your Ethical Hacking Skills: Master Crytography, Breach Firewalls, Attack Passwords & More\n          \n        \n      "
   "/courses/enrolled/33657">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Byte Size Chunks: Java Object-Oriented Programming & Design\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Conquer Java with Just 7 Hours of Premium Instruction\n          \n        \n      "
   "/courses/enrolled/52510">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: Data Structures & Algorithms in Java\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master the Fundamentals of How Computer Programs Organize & Manipulate Data\n          \n        \n      "
   "/courses/enrolled/51691">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Learn Network Hacking/Penetration Testing From Scratch (WiFi & Wired)\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Uncover Vulnerabilities in Wired Networks By Mastering Practical Attacks Against Wireless Devices\n          \n        \n      "
   "/courses/enrolled/33211">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: Design Patterns - 24 That Matter - In Java\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Increase Your Coding Efficiency with 24 Tried-and-True Java Design Patterns\n          \n        \n      "
   "/courses/enrolled/51687">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: C Programming - Drill Deep\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            \n          \n        \n      "
   "/courses/enrolled/51685">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: Learn Python Programming  - Easy as Pie\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            \n          \n        \n      "
   "/courses/enrolled/51693">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Game Developer Business & Legal Guide: Run an Indie Studio\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Navigate Business & Legal Issues without Shelling Out for a Lawyer\n          \n        \n      "
   "/courses/enrolled/49394">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Windows Exploit Development Megaprimer\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Execute Windows Software Exploitation Using Tried & True Techniques\n          \n        \n      "
   "/courses/enrolled/33311">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Learn to Code in Game Maker Language\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Quickly & Easily Develop Games Using This Popular Game Creation System\n          \n        \n      "
   "/courses/enrolled/49731">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            The Complete Android Marshmallow Development Course\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Learn By Building 5 Phone & Smartwatch Apps For the Latest Version of Android\n          \n        \n      "
   "/courses/enrolled/20929">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Master Unity By Building 6 Fully Featured Games From Scratch\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Become a Full-Fledged 2D Game Developer Using This Popular Game Engine\n          \n        \n      "
   "/courses/enrolled/48312">
  #<Mechanize::Page::Link
   "\n        \n          \n          \n            \n          \n          \n            Learn By Example: C++ Programming - 75 Solved Problems\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            \n          \n        \n      "
   "/courses/enrolled/52516">
  #<Mechanize::Page::Link "2" "/courses/enrolled?page=2">
  #<Mechanize::Page::Link "3" "/courses/enrolled?page=3">
  #<Mechanize::Page::Link "Next ›" "/courses/enrolled?page=2">
  #<Mechanize::Page::Link "Last »" "/courses/enrolled?page=3">
  #<Mechanize::Page::Link "About Us" "pages/about-us">
  #<Mechanize::Page::Link
   "\n              Terms of Use\n            "
   "/pages/terms">
  #<Mechanize::Page::Link
   "\n              Privacy Policy\n            "
   "/pages/privacy">
  #<Mechanize::Page::Link
   "\n                For Instructors\n              "
   "/pages/instructors">
  #<Mechanize::Page::Link
   "\n              Contact Us\n            "
   "https://stackskills.zendesk.com/hc/en-us/requests/new">}
 {forms
  #<Mechanize::Form
   {name nil}
   {method "GET"}
   {action "/courses"}
   {fields [text:0x1e8dcfc type: text name: query value: ]}
   {radiobuttons}
   {checkboxes}
   {file_uploads}
   {buttons [button:0x1e8d20c type: submit name:  value: ]}>}>

#<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Byte Size Chunks : Java Reflection, Annotations and Lambdas\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master 3 Powerful Features to Take Your Java Expertise to the Next Level\n          \n        \n      "
 "/courses/enrolled/53431">

~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#<Mechanize::Page
 {url #<URI::HTTPS https://stackskills.com/courses/enrolled/53431>}
 {meta_refresh}
 {title "StackSkills"}
 {iframes}
 {frames}
 {links
  #<Mechanize::Page::Link "\n          \n        " "/courses/">
  #<Mechanize::Page::Link
   "\n        \n          \n        \n        Class Curriculum\n      "
   "/courses/53431">
  #<Mechanize::Page::Link
   "\n      \n        \n      \n      Your Instructor\n    "
   "/courses/53431/author_bio">
  #<Mechanize::Page::Link "\n  \n  Name\n" nil>
  #<Mechanize::Page::Link "\n      Edit Profile\n    " "/current_user/profile">
  #<Mechanize::Page::Link
   "\n      Manage Subscriptions\n    "
   "/current_user/subscriptions">
  #<Mechanize::Page::Link
   "\n      Add / Change Credit Card\n    "
   "/current_user/credit_card">
  #<Mechanize::Page::Link "\n      Contact\n    " "/current_user/contact">
  #<Mechanize::Page::Link "\n      Log Out\n    " "/sign_out">
  #<Mechanize::Page::Link
   "\n        \n          \n        \n        Class Curriculum\n      "
   "/courses/53431">
  #<Mechanize::Page::Link
   "\n      \n        \n      \n      Your Instructor\n    "
   "/courses/53431/author_bio">
  #<Mechanize::Page::Link "Start next lecture  ›" nil>
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              You, This course and Us\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784900">
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              Functional Programming: A State of Mind\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784901">
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              Lambda Functions: A Crossover Hit\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784902">
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              Lambda Functions: No more boiler-plate code\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784903">
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              Lambda functions for real - in code\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784904">
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              Recursion\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784906">
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              Reflection: Not a Dark Art\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784907">
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              Reflection: A real use case\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784908">
  #<Mechanize::Page::Link
   "\n          \n            \n               \n            \n          \n          \n            \n              Start\n            \n            \n              \n            \n            \n              Annotations: Notes to Self can be Super-Useful\n            \n          \n        "
   "/courses/bsc-reflection/lectures/784910">}
 {forms}>

nil
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[#<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            How To Make 2D iPhone Games Using Swift 4 and iOS 11\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Create and Make Money With 2D iPhone Games. Designed for Beginners. Designed to Move Fast.\n          \n        \n      "
 "/courses/enrolled/159101">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            The Complete Python Course: Beginner to Advanced!\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Learn how to code in Python working on practical applications, and get started programming today!\n          \n        \n      "
 "/courses/enrolled/102831">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Master Video Game Making\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Beginners to Intermediates: Learn industry standard tools, technologies, terms and concepts. Then any tool and game engine is easily adapted to your skills.\n          \n        \n      "
 "/courses/enrolled/187722">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            The Complete Web Developer Course 2.0\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Web development from scratch in 6 weeks. Build 25 Websites using HTML 5, JS, CSS 3, PHP, MySQL, Bootstrap, Wordpress, APIs, Mobile Apps, Python all covered.\n          \n        \n      "
 "/courses/enrolled/71859">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Cisco New CCNA R/S (200-125): The Complete Course\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            This course has everything you need to prepare & pass Cisco's new CCNA R/S 200-125 certification exam\n          \n        \n      "
 "/courses/enrolled/170649">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Web application penetration testing professional - WAPTP v3.1\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Attack web application like never before with latest tools, tricks and attacks at professional level\n          \n        \n      "
 "/courses/enrolled/231518">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            NEW Cisco CCNA IPv4 Course!\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            This course is geared to enhance your knowledge & understanding of the IPv4 protocol which is still used in the workplace.\n          \n        \n      "
 "/courses/enrolled/170651">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            The Complete iOS 11 & Swift Developer Course - Build 20 Apps\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Use Xcode 9 & Swift 4 to make real apps like Uber and Instagram, with MLKit & ARKit. Includes AWS Credit and much more.\n          \n        \n      "
 "/courses/enrolled/197556">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            The Complete Cyber Security Course! Volume 2 : Network Security\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Network Security, WiFi Security, WiFi Hackers, Firewalls, Wireshark. Plus the Best Password Managers\n          \n        \n      "
 "/courses/enrolled/137514">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Cisco New CCNA CCENT / ICND1 (100-105): The Complete Course\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            The complete guide to prepare for Cisco's CCENT certification or otherwise known as the ICND1 for test # 100-105\n          \n        \n      "
 "/courses/enrolled/170652">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Learn how to make 6 games in the Unreal Engine\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Learn to make games from scratch in the popular engine Unreal 4!\n          \n        \n      "
 "/courses/enrolled/182238">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Hack People, Systems and Mobile Devices. Advanced Social Engineering\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            IT security is not just about firewalls or antiviruses, it is also about people. It's about mobile devices that we carry with us.\n          \n        \n      "
 "/courses/enrolled/230966">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Portrait Photography Masterclass\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Create Beautiful Portraits for Family, Friends, or Clients\n          \n        \n      "
 "/courses/enrolled/69803">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Game Development using Corona SDK in 2015 with ASO & Ads\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Everything you need to know about developing a game with Corona SDK.\n          \n        \n      "
 "/courses/enrolled/49333">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            JavaScript Programming: Learn by Making a Mobile Game\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Dive into JavaScript's Core Concepts By Building a Game From Scratch\n          \n        \n      "
 "/courses/enrolled/14464">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            The Ultimate Java 8 Tutorial - From Beginner to Professional\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master All You Need to Know About Java with This Comprehensive 20-Hour Course\n          \n        \n      "
 "/courses/enrolled/55201">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Java Programming – the Master Course\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Get Premium, Step-By-Step Instruction on All Things Java\n          \n        \n      "
 "/courses/enrolled/53615">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Java Programming from Zero to Hero: Java Completed\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Jumpstart Your Java Journey in Under 4 Hours\n          \n        \n      "
 "/courses/enrolled/53326">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Node.js Security: Pentesting and Exploitation\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Learn Exploitation Development for Node.js Apps By Executing Code Reviews, Identifying Insecure Code & More\n          \n        \n      "
 "/courses/enrolled/33308">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Byte Size Chunks : Java Reflection, Annotations and Lambdas\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master 3 Powerful Features to Take Your Java Expertise to the Next Level\n          \n        \n      "
 "/courses/enrolled/53431">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Byte Size Chunks: Java Model-View-Controller (MVC)\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Use This Design Pattern to Easily Build & Test Complex Apps, Facilitate Group Development, and More\n          \n        \n      "
 "/courses/enrolled/53432">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: Heavy Lifting with SQL & Databases\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Become a SQL Pro -- Conduct Exercises Using MySQL & SQLite\n          \n        \n      "
 "/courses/enrolled/51682">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            The White Hat Hacker Bundle\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master Penetration Testing with 7 Courses on Hacking Node.js Apps, Windows OS, Wi-Fi Devices & More\n          \n        \n      "
 "/courses/enrolled/39908">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Intensive Ethical Hacking Series\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Turbocharge Your Ethical Hacking Skills: Master Crytography, Breach Firewalls, Attack Passwords & More\n          \n        \n      "
 "/courses/enrolled/33657">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Byte Size Chunks: Java Object-Oriented Programming & Design\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Conquer Java with Just 7 Hours of Premium Instruction\n          \n        \n      "
 "/courses/enrolled/52510">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: Data Structures & Algorithms in Java\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Master the Fundamentals of How Computer Programs Organize & Manipulate Data\n          \n        \n      "
 "/courses/enrolled/51691">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Learn Network Hacking/Penetration Testing From Scratch (WiFi & Wired)\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Uncover Vulnerabilities in Wired Networks By Mastering Practical Attacks Against Wireless Devices\n          \n        \n      "
 "/courses/enrolled/33211">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: Design Patterns - 24 That Matter - In Java\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Increase Your Coding Efficiency with 24 Tried-and-True Java Design Patterns\n          \n        \n      "
 "/courses/enrolled/51687">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: C Programming - Drill Deep\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            \n          \n        \n      "
 "/courses/enrolled/51685">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            From 0 to 1: Learn Python Programming  - Easy as Pie\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            \n          \n        \n      "
 "/courses/enrolled/51693">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Game Developer Business & Legal Guide: Run an Indie Studio\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Navigate Business & Legal Issues without Shelling Out for a Lawyer\n          \n        \n      "
 "/courses/enrolled/49394">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Windows Exploit Development Megaprimer\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Execute Windows Software Exploitation Using Tried & True Techniques\n          \n        \n      "
 "/courses/enrolled/33311">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Learn to Code in Game Maker Language\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Quickly & Easily Develop Games Using This Popular Game Creation System\n          \n        \n      "
 "/courses/enrolled/49731">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            The Complete Android Marshmallow Development Course\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Learn By Building 5 Phone & Smartwatch Apps For the Latest Version of Android\n          \n        \n      "
 "/courses/enrolled/20929">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Master Unity By Building 6 Fully Featured Games From Scratch\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            Become a Full-Fledged 2D Game Developer Using This Popular Game Engine\n          \n        \n      "
 "/courses/enrolled/48312">
, #<Mechanize::Page::Link
 "\n        \n          \n          \n            \n          \n          \n            Learn By Example: C++ Programming - 75 Solved Problems\n          \n          \n          \n            \n              \n            \n          \n          \n          \n            \n          \n        \n      "
 "/courses/enrolled/52516">
]
Unable to find this course: https://stackskills.com/courses/enrolled/345065 from your list of courses.
Number of courses found: 1
/tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:37:in `block in execute': undefined method `href' for nil:NilClass (NoMethodError)
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:36:in `each'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:36:in `execute'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:12:in `block in run'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/utilities.rb:13:in `block in mkchdir'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/utilities.rb:12:in `chdir'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/utilities.rb:12:in `mkchdir'
    from /tmp/stackskills-dl/stackskills-dl-master/lib/course_finder.rb:11:in `run'
    from stackskills_dl.rb:11:in `<main>'
linux95 commented 5 years ago

yoonwaiyan, Looking at the output from the file modification you requested, it appears that it read all of the courses on the first page then drop into one of the courses, Byte Size Chunks : Java Reflection, Annotations and Lambdas to be exact, and started to list them. Then it appears that it re-read the first page again. I hope this output helps you in determining the issue. Please don't hesitate to let me know if you need anything further to diagnose. Thanks.

yoonwaiyan commented 5 years ago

So much thanks for your input, you're correct, it's an edge case of the program which had just found the wrong link to follow up as the course contains the word "Next", so I've put up a fix ( https://github.com/yoonwaiyan/stackskills-dl/commit/c54e4cdc3ee05130caee3ba429621d36b9ba76b6 ) to make the pagination search to be more specific so that it don't mess up with the courses and the pagination links. Now you can discard all the changes above, update the codebase and try again. It should be working now 😄

linux95 commented 5 years ago

yoonwaiyan, I test the new fix yesterday and after downloading multiple courses on the first page, I receive the following error (screenshot attached). Let me know if you need more info. Thanks.

screen shot 2018-08-23 at 8 39 49 am
yoonwaiyan commented 5 years ago

May I have your the output for p lectures on line 113 of the same file for the particular course?

linux95 commented 5 years ago

yoonwaiyan, I have attached a file with the output requested. I couldn't paste as the output was greater than 65536 characters. Let me know if you need anything further. stackskills_output.txt

yoonwaiyan commented 5 years ago

Sorry for being late, I'm currently occupied by my full time job, would need some time to take a look at this issue.

Now I need the output after adding p href after the line of href = lecture.children[1] around line 120/121 to identify the part of the course which triggers the error.

linux95 commented 5 years ago

yoonwaiyan, I have attached a file with the output requested. Please let me know if you need more info. Thanks. 48312.debug.output.txt

yoonwaiyan commented 5 years ago

Hi @linux95 , both debug output seems to be the same and it's hard to trace it this way though, as both output doesn't seem to display the error message as the screenshot shows and there isn't any message indicating the first lecture being downloaded(there should be a STDOUT that look like "Downloading "). Is the error still appearing on your screen and did any of the lecture gets downloaded to your machine?

linux95 commented 5 years ago

yoonwaiyan, It appears that the updates you applied earlier are working correctly. For some strange reason, four (4) of the courses were producing errors while trying to download. However, after repeated attempts, they all downloaded without error. I believe this may be an issue on the StackSkills site and not the code. Thanks again for all of your help as I believe the bug has been fixed. Let me know if you need anything else from me as I am available to assist.

yoonwaiyan commented 5 years ago

I see, glad it worked for you :) I'm closing this issue for now as the issue is resolved, but feel free to open another issue if there's any error being thrown consistently.