Open SanZamoyski opened 1 month ago
Forum-dl probably fails to detect which extractor to use because of the forum's custom theme. There's currently no way to tweak the detection code from the command line, so this can only be fixed manually by modifying the detector code.
There's no way to log in as I haven't implemented cookies yet. Sadly, I haven't had much time to continue this project as I'm currently very busy with another one.
I understand. Can You point a bit place where to tweak?
The PhpBB detection code starts here. In fact, you can see what is the problem from the first line straight away... But this is probably just the tip of the iceberg. Feel free to open a PR if you can get it working.
But! I have another project, Skrob, which I intend as a future replacement for Forum-dl. Though I haven't finished or wrote documentation for it yet, it may be of use to you.
If you install it (git clone https://github.com/mikwielgus/skrob
, then pip install -e skrob
), you should be able to crawl this forum and scrape the text of its posts with the following command:
skrob -n 1 "title; .forumlink::attr(href) -> {title; .topictitle::attr(href) -> {title; .postbody; td.gensmall[width^='100'][align='right'][nowrap='nowrap'] a::attr(href) ->}}" "https://www.t4-forum.pl/index.php"
Unfortunately, this outputs raw line-by-line-streamed XHTML, which requires further processing. But it may be a good start!
Hi!
I want to download whole forum. Also, is there way to log in before downloading or sid is enought?
Thanks and best regards!