The long functions forum(), render_forum(), wiki() and render_wiki() are exploded into several methods.
The file now contains two classes MoocForum and MoocWiki to hold the forum and wiki and the methods to extract and render them
MoocForum.add_categories() - Adds categories
MoocForum.prepare_forum_categories() - Get forum categories, calls add_categories()
MoocForum.populate_staff_users() - Populates the list of staff users
MoocForum.prepare_thread_list() - Prepare the list of threads
MoocForum.fetch_thread_data() - Fetches data for the threads
MoocForum.update_thread_children() - Updates the children of a thread with their data
MoocForum.annex_forum() - Main method to offline forum
MoocForum.render_forum() - Renders the forum
MoocWiki.get_first_page() - Gets the first page of the wiki
MoocWiki.add_to_wiki_data() - Adds a wiki URL to wiki_data
MoocWiki.update_wiki_page() - Updates a wiki page in wiki_data
MoocWiki.get_wiki_children() - Gets the children of the wiki
MoocWiki.annex_wiki() - Main method to offline a wiki
MoocWiki.render_wiki() - Renders the wiki
The booknav() and render_booknav() functions are moved to be methods in scraper.py. These were simple helper functions and this makes annex.py look much cleaner.
The method annex() in scraper.py is now broken into 3 methods as follows -
annex() - the main method to call others and put tab_paths
get_tab_path_and_name() - To get the tab name and path for our structure. It is this function which calls the forum and wiki annex methods
annex_extra_page() - To extract extra pages (with tab_paths not supported in get_tab_path_and_name) and book lists
The main logic behind the forum and wiki offlining is kept the same. However, offlining them might be unstable at times and we need to test that.
This one refactors a lot of long functions and hence addresses #48. Also #54 is fixed with this.
This one does quite a couple of changes -
annex.py is fully refactored -
The method annex() in scraper.py is now broken into 3 methods as follows -
The main logic behind the forum and wiki offlining is kept the same. However, offlining them might be unstable at times and we need to test that.
This one refactors a lot of long functions and hence addresses #48. Also #54 is fixed with this.