Open fishcakedev opened 4 years ago
Are you enrolled in any moodle courses? You could also send me a link to your moodle, then i can check it for you.
So you cleared out the log? Are the courses listed in the html in the log? So maybe they customized the dashboard. I can not create a guest account. But you could send me the html of this page moodle.uni-wuppertal.de/my/index.php?mynumber=-2
you can obfuscate course names and links... or just send me the whole html via mail to daniel@vogt.ml
But do not destroy the structure... so I can look why the bot struggle to find the courses.
You can check out the latest commit https://github.com/C0D3D3V/Moodle-Downloader/commit/85193dc5436bebafb4a3294730c8699548e69a9b, maybe it fixes the problem. I tried to clean up the prefix "Kursname" before every course name, if it did not work please give a feedback.
Ok, I checked out the latest code and tested it. Unfortunately, there is still a mistake. Enclosed the Error log and the course name of Moodle.
09:28:13 Finished course: 'Informatik-für-Telematiker-(WS19-20)(Janine-Breßler)' 09:28:13 Check course: 'Ortung-und-Navigation-für-Telematik-Dienste-ONTD-WS19-20' ID: 15066 Traceback (most recent call last): File "/Users/romankobosil/Documents/Telematik/Moodle-Downloader/src/moodleCrawler.py", line 1709, in
crawlMoodlePage(course[1], course[0], current_dir, mainpageURL + "my/") File "/Users/romankobosil/Documents/Telematik/Moodle-Downloader/src/moodleCrawler.py", line 1366, in crawlMoodlePage PageLinkContent, responsePageLink, pagelink) File "/Users/romankobosil/Documents/Telematik/Moodle-Downloader/src/moodleCrawler.py", line 440, in saveFile file_name, fileName, filetype, pathtoSearch) File "/Users/romankobosil/Documents/Telematik/Moodle-Downloader/src/moodleCrawler.py", line 707, in searchfordumpsSpecific re.escape(fileName) + '*' + re.escape(filetype)) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/fnmatch.py", line 56, in filter _cache[pat] = re_pat = re.compile(res) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 194, in compile return _compile(pattern, flags) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 251, in _compile raise error, v # invalid expression sre_constants.error: bad character range Ortung und Navigation für Telematik-Dienste ONTD [WS19/20] (Stefan Brunthaler)
I've got the same issue.
python2 moodleCrawler.py
20:51:24 Moodle Crawler started working.
20:51:24 Try to copy logintoken!
20:51:24 Download has started.
20:51:24 No Content-Length available.
20:51:24 Downloaded 29187 bytes
20:51:24 Download complete.
20:51:24 Warning: Found 3 forms!
20:51:24 Logintoken: AH5MssxHCDdw78GB7NPdSb96RvWACj7C
20:51:24 Try to login...
20:51:26 Download has started.
20:51:26 No Content-Length available.
20:51:26 Downloaded 81924 bytes
20:51:26 Downloaded 163848 bytes
20:51:26 Downloaded 182267 bytes
20:51:26 Download complete.
20:51:26 Logged in!
20:51:26 Searching Courses...
20:51:27 Download has started.
20:51:27 No Content-Length available.
20:51:27 Downloaded 81924 bytes
20:51:27 Downloaded 163848 bytes
20:51:27 Downloaded 182267 bytes
20:51:27 Download complete.
20:51:27 No link to this course was found!
20:51:27 Full page: <div class="card-body pr-1 course-info-container">
<div class="bg-pulse-grey w-100" style="height: 1rem"></div>
</div>
20:51:27 No link to this course was found!
20:51:27 Full page: <div class="card-body pr-1 course-info-container">
<div class="bg-pulse-grey w-100" style="height: 1rem"></div>
</div>
20:51:27 No link to this course was found!
20:51:27 Full page: <div class="card-body pr-1 course-info-container">
<div class="bg-pulse-grey w-100" style="height: 1rem"></div>
</div>
20:51:27 No link to this course was found!
20:51:27 Full page: <div class="card-body pr-1 course-info-container">
<div class="bg-pulse-grey w-100" style="height: 1rem"></div>
</div>
20:51:27 Unable to find courses
20:51:27 Full page: <section aria-label="Inhalt" class="region-main-content" id="region-main">
<span class="notifications" id="user-notifications"></span>
<div role="main"><span id="maincontent"></span><aside class="block-region" data-blockregion="content" data-droptarget="1" id="block-region-content"><a class="sr-only sr-only-focusable" href="#sb-9">Kursübersicht überspringen</a>
<section aria-labelledby="instance-117378-header" class="block_myoverview block card mb-3" data-block="myoverview" id="inst117378" role="complementary">
<div class="card-body p-3">
<h5 class="card-title d-inline" id="instance-117378-header">Kursübersicht</h5>
<div class="card-text content mt-3">
<div class="block-myoverview block-cards" data-region="myoverview" id="block-myoverview-5dd5993ee32285dd5993ec62c55" role="navigation">
<div aria-label="Steuerungen der Kursübersicht" class="d-flex align-items-center flex-wrap" data-region="filter">
<div class="dropdown mb-1 mr-auto">
<button aria-expanded="false" aria-haspopup="true" aria-label="Dropdown-Menü für die Gruppierung" class="btn btn-outline-secondary dropdown-toggle" data-toggle="dropdown" id="groupingdropdown" type="button">
<i aria-hidden="true" class="icon fa fa-filter fa-fw"></i>
<span class="d-sm-inline-block" data-active-item-text="">
Alle (außer verborgene)
</span>
</button>
<ul aria-labelledby="groupingdropdown" class="dropdown-menu" data-active-item-text="" data-show-active-item="">
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Alle Kurse außer den verborgenen Kursen" class="dropdown-item active" data-filter="grouping" data-pref="all" data-value="all" href="#">
Alle (außer verborgene)
</a>
</li>
<li class="dropdown-divider" role="presentation">
<span class="filler"> </span>
</li>
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Laufende Kurse anzeigen" class="dropdown-item" data-filter="grouping" data-pref="inprogress" data-value="inprogress" href="#">
Laufende
</a>
</li>
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Künftige Kurse anzeigen" class="dropdown-item" data-filter="grouping" data-pref="future" data-value="future" href="#">
Künftige
</a>
</li>
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Vergangene Kurse anzeigen" class="dropdown-item" data-filter="grouping" data-pref="past" data-value="past" href="#">
Vergangene
</a>
</li>
<li class="dropdown-divider" role="presentation">
<span class="filler"> </span>
</li>
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Favorisierte Kurse anzeigen" class="dropdown-item" data-filter="grouping" data-pref="favourites" data-value="favourites" href="#">
Als Favorit markiert
</a>
</li>
<li class="dropdown-divider" role="presentation">
<span class="filler"> </span>
</li>
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Verborgene Kurse anzeigen" class="dropdown-item" data-filter="grouping" data-pref="hidden" data-value="hidden" href="#">
Verborgene
</a>
</li>
</ul>
</div>
<div class="mb-1 mr-1 d-flex align-items-center">
<div class="dropdown">
<button aria-expanded="false" aria-haspopup="true" aria-label="Dropdown-Menü für die Sortierung" class="btn btn-outline-secondary dropdown-toggle" data-toggle="dropdown" id="sortingdropdown" type="button">
<i aria-hidden="true" class="icon fa fa-sort-amount-asc fa-fw"></i>
<span class="d-sm-inline-block" data-active-item-text="">
Kursname
</span>
</button>
<ul aria-labelledby="sortingdropdown" class="dropdown-menu" data-show-active-item="">
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Kurse nach Kursnamen sortieren" class="dropdown-item active" data-filter="sort" data-pref="title" data-value="fullname" href="#">
Kursname
</a>
</li>
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Kurse nach letztem Zugriffsdatum anzeigen" class="dropdown-item" data-filter="sort" data-pref="lastaccessed" data-value="ul.timeaccess desc" href="#">
Letzter Zugriff
</a>
</li>
</ul>
</div>
</div>
<div class="dropdown mb-1">
<button aria-expanded="false" aria-haspopup="true" aria-label="Dropdown-Menü für die Anzeige" class="btn btn-outline-secondary dropdown-toggle" data-toggle="dropdown" id="displaydropdown" type="button">
<i aria-hidden="true" class="icon fa fa-th fa-fw"></i>
<span class="d-sm-inline-block" data-active-item-text="">
Liste
</span>
</button>
<ul aria-labelledby="displaydropdown" class="dropdown-menu" data-show-active-item="">
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Kacheln anzeigen" class="dropdown-item" data-display-option="display" data-pref="cards" data-value="cards" href="#">
Kachel
</a>
</li>
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Zur Listenansicht wechseln" class="dropdown-item active" data-display-option="display" data-pref="list" data-value="list" href="#">
Liste
</a>
</li>
<li>
<a aria-controls="courses-view-5dd5993ee32285dd5993ec62c55" aria-label="Zur Beschreibungsansicht wechseln" class="dropdown-item" data-display-option="display" data-pref="summary" data-value="summary" href="#">
Beschreibung
</a>
</li>
</ul>
</div>
</div>
<div class="container-fluid p-0">
<div data-display="list" data-displaycategories="on" data-grouping="all" data-nocoursesimg="https://moodle.bulme.at/theme/image.php/classic/block_myoverview/1568231222/courses" data-paging="48" data-prev-display="list" data-region="courses-view" data-sort="fullname" id="courses-view-5dd5993ee32285dd5993ec62c55">
<div data-region="course-view-content">
<div aria-hidden="true" data-region="loading-placeholder-content">
<ul class="list-group">
<li class="list-group-item course-listitem">
<div class="col-md-6 p-0 d-flex align-items-center">
<div class="bg-pulse-grey rounded-circle mr-2" style="height: 1rem; width: 1rem;"></div>
<div class="bg-pulse-grey w-50" style="height: 1rem; margin: 0.5rem 0"></div>
</div>
</li>
<li class="list-group-item course-listitem">
<div class="col-md-6 p-0 d-flex align-items-center">
<div class="bg-pulse-grey rounded-circle mr-2" style="height: 1rem; width: 1rem;"></div>
<div class="bg-pulse-grey w-50" style="height: 1rem; margin: 0.5rem 0"></div>
</div>
</li>
<li class="list-group-item course-listitem">
<div class="col-md-6 p-0 d-flex align-items-center">
<div class="bg-pulse-grey rounded-circle mr-2" style="height: 1rem; width: 1rem;"></div>
<div class="bg-pulse-grey w-50" style="height: 1rem; margin: 0.5rem 0"></div>
</div>
</li>
<li class="list-group-item course-listitem">
<div class="col-md-6 p-0 d-flex align-items-center">
<div class="bg-pulse-grey rounded-circle mr-2" style="height: 1rem; width: 1rem;"></div>
<div class="bg-pulse-grey w-50" style="height: 1rem; margin: 0.5rem 0"></div>
</div>
</li>
</ul>
</div>
</div>
</div> </div>
</div>
<div class="footer"></div>
</div>
</div>
</section>
<span id="sb-9"></span><a class="sr-only sr-only-focusable" href="#sb-10">Zuletzt besuchte Kurse überspringen</a>
<section aria-labelledby="instance-117380-header" class="block_recentlyaccessedcourses block card mb-3" data-block="recentlyaccessedcourses" id="inst117380" role="complementary">
<div class="card-body p-3">
<h5 class="card-title d-inline" id="instance-117380-header">Zuletzt besuchte Kurse</h5>
<div class="card-text content mt-3">
<div class="block-recentlyaccessedcourses block-cards" data-region="recentlyaccessedcourses" data-userid="14312" id="block-recentlyaccessedcourses-5dd5993ee3e635dd5993ec62c56">
<div class="container-fluid p-0">
<div data-region="recentlyaccessedcourses-view" id="recentlyaccessedcourses-view-5dd5993ee3e635dd5993ec62c56">
<div data-region="loading-placeholder">
<div class="card-deck dashboard-card-deck one-row overflow-hidden" style="height: 13.05rem">
<div class="card dashboard-card border-0">
<div class="card-img-top bg-pulse-grey w-100" style="height: 7rem">
</div>
<div class="card-body pr-1 course-info-container">
<div class="bg-pulse-grey w-100" style="height: 1rem"></div>
</div>
</div>
<div class="card dashboard-card border-0">
<div class="card-img-top bg-pulse-grey w-100" style="height: 7rem">
</div>
<div class="card-body pr-1 course-info-container">
<div class="bg-pulse-grey w-100" style="height: 1rem"></div>
</div>
</div>
<div class="card dashboard-card border-0">
<div class="card-img-top bg-pulse-grey w-100" style="height: 7rem">
</div>
<div class="card-body pr-1 course-info-container">
<div class="bg-pulse-grey w-100" style="height: 1rem"></div>
</div>
</div>
<div class="card dashboard-card border-0">
<div class="card-img-top bg-pulse-grey w-100" style="height: 7rem">
</div>
<div class="card-body pr-1 course-info-container">
<div class="bg-pulse-grey w-100" style="height: 1rem"></div>
</div>
</div>
</div>
</div>
<div class="hidden" data-region="view-content">
<div class="d-flex paging-bar-container mb-3" data-region="paging-bar-container">
<div class="d-flex" data-region="paging-control-container">
<nav aria-label="Navigation mit Seitennummern" class="" data-active-page-number="1" data-aria-label-components-pagination-active-item="pagedcontentnavigationactiveitem, core" data-aria-label-components-pagination-item="pagedcontentnavigationitem, core" data-hide-control-on-single-page="" data-ignore-control-while-loading="" data-items-per-page="" data-region="paging-bar" id="paging-bar-5dd5993ee3e635dd5993ec62c56" role="navigation">
<ul class="pagination mb-0">
<li class="page-item" data-control="previous" data-region="page-item">
<a class="page-link" data-region="page-link" href="#">
<span aria-hidden="true" class="icon-no-margin dir-rtl-hide"><i aria-hidden="true" class="icon fa fa-chevron-left fa-fw"></i></span>
<span aria-hidden="true" class="icon-no-margin dir-ltr-hide"><i aria-hidden="true" class="icon fa fa-chevron-right fa-fw"></i></span>
</a>
</li>
<li class="page-item" data-control="next" data-region="page-item">
<a class="page-link" data-region="page-link" href="#">
<span aria-hidden="true" class="icon-no-margin dir-rtl-hide"><i aria-hidden="true" class="icon fa fa-chevron-right fa-fw"></i></span>
<span aria-hidden="true" class="icon-no-margin dir-ltr-hide"><i aria-hidden="true" class="icon fa fa-chevron-left fa-fw"></i></span>
</a>
</li>
</ul>
</nav>
</div>
</div>
<div class="card-deck dashboard-card-deck one-row fixed-width-cards justify-content-center overflow-hidden" data-region="card-deck" role="list">
</div>
</div>
<div class="hidden text-xs-center text-center mt-3" data-region="empty-message">
<img alt="Keine zuletzt besuchten Kurse" class="empty-placeholder-image-lg mt-1" role="presentation" src="https://moodle.bulme.at/theme/image.php/classic/block_recentlyaccessedcourses/1568231222/courses"/>
<p class="text-muted mt-3">Keine zuletzt besuchten Kurse</p>
</div>
</div>
</div>
</div>
<div class="footer"></div>
</div>
</div>
</section>
<span id="sb-10"></span></aside></div>
</section>
20:51:27 Scanning directory "/tmp/moodle"....
20:51:27 Finding potential dupes...
20:51:27 Found 0 sets of potential dupes...
20:51:27 Scanning for real dupes...
20:51:27 Update Complete
So you cleared out the log? Are the courses listed in the html in the log?
Are you talking about /tmp/moodle/.crawlhistory.log
?
It only contains LogFile:V1.0
So maybe they customized the dashboard.
The courses aren't visible instantly, first there is a loading animation, which shows four empty entries, it actually takes 1 or 2 seconds to show me the courses
I captured the traffic with mitmproxy
The html of https://moodle.bulme.at/my/index.php?mynumber=-2
just shows the four empty entries with the loading animation, courses aren't being loaded
Proposal: If you set onlycrawlcourses = "$id"
you could bypass this issue by accessing courses by their id, instead of crawling the dashboard.
I'll switch to German. :) If you want me to write in English, you can write that.
Also euer Problem ähnelt sich tatsächlich sehr. Hast du bereits probiert allcourses = true
zu setzen?
Ich schau mal, dass ich die Tage das um programmiere. Das Problem ist, dass die Moodle Version neu ist und dort das Dashboard neu designed wurde. Jetzt werden die Kurse wohl nachgeladen oder sind auserhalb des Bereiches in dem mein Skript sucht. Ich hab schon ein Plan wie ich das löse nur noch keine Zeit gehabt.
Hast du bereits probiert
allcourses = true
zu setzen?
Ja, aber es ändert nichts.
Eine einfache Alternative (oder auch Zusatzoption für dieses Skript) wäre vermutlich die Kurse einfach bei ihrer id aufzurufen und das Dashboard damit zu umgehen. (Schema: course/view.php?id=id
)
Ok.
Ja das habe ich mir auch überlegt. Das wäre recht schnell zu implementieren. Kann ich heut Abend noch machen. Werde aber dennoch an der besseren Lösung noch arbeiten (wird dann irgendwann in naher Zukunft veröffentlicht)
Check out the new option: extracrawlcourses
Ist jetzt nur eine Notlösung bis ich die automatische erkennung von Kursen korrigiert habe. Hoffe das funktioniert.
I created a new downloader, that should fix this issue... check it out here https://github.com/C0D3D3V/Moodle_Downloader_2
terminal output with loglevel 5:
`22:11:05 Moodle Crawler started working. 22:11:05 Try to copy logintoken! 22:11:05 Download has started. 22:11:05 No Content-Length available. 22:11:05 Downloaded 33324 bytes 22:11:05 Download complete. 22:11:05 Warning: Found 2 forms! 22:11:05 Logintoken: deletedtokenforthispost 22:11:05 Try to login... 22:11:06 Download has started. 22:11:06 No Content-Length available. 22:11:06 Downloaded 81924 bytes 22:11:06 Downloaded 147104 bytes 22:11:06 Download complete. 22:11:06 Logged in! 22:11:06 Searching Courses... 22:11:07 Download has started. 22:11:07 No Content-Length available. 22:11:07 Downloaded 81924 bytes 22:11:07 Downloaded 163848 bytes 22:11:07 Downloaded 192558 bytes 22:11:07 Download complete. 22:11:07 No link to this course was found! 22:11:07 Full page: