search

dipu-bd / lightnovel-crawler

Generate and download e-books from online sources.
https://pypi.org/project/lightnovel-crawler/
GNU General Public License v3.0
1.42k stars 279 forks source link

Help cant search Fix this bug #2331

Closed Nephalemzz closed 3 months ago

Nephalemzz commented 5 months ago

Describe the bug

Searching: 53%|██████████████████████████████████▌ | 145/273 [01:22<03:05, 1.45s/source] ! Error: decoding to str: need a bytes-like object, NoneType found <class 'TypeError'> File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\app.py", line 103, in search_novel search_novels(self) File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\novel_search.py", line 125, in search_novels app.search_results = _combine_results(results) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\novel_search.py", line 46, in _combine_results key = slugify(item.title) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\slugify\slugify.py", line 104, in slugify text = _unicode(text, 'utf-8', 'ignore') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

! Error: Cancelled by user

Exception ignored in: <module 'threading' from 'C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64qbz5n2kfra8p0\Lib\threading.py'> Traceback (most recent call last): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64qbz5n2kfra8p0\Lib\threading.py", line 1592, in _shutdown atexit_call() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64qbz5n2kfra8p0\Lib\concurrent\futures\thread.py", line 31, in _python_exit t.join() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64qbz5n2kfra8p0\Lib\threading.py", line 1147, in join self._wait_for_tstate_lock() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.12_3.12.752.0_x64qbz5n2kfra8p0\Lib\threading.py", line 1167, in _wait_for_tstate_lock if lock.acquire(block, timeout): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\init__.py", line 25, in destroy sys.exit(1) SystemExit: 1

Let us know

Does not matter which novel i try and search it gives this error. did not have it on 3.4.2.

App source: EXE | pip install clone App version: 3.5.1 Your OS: Windows

Nephalemzz commented 5 months ago

Tried to downgrade to python 3.11 and error still occurs

                      [#] Lightnovel Crawler v3.5.1
              https://github.com/dipu-bd/lightnovel-crawler

-> Press Ctrl + C to exit

? Enter novel page url or query novel: super god gene Searching: 34%|██████████████████████▋ | 93/270 [00:11<00:31, 5.60source/s]Failed to get any results, likely auth failure Searching: 73%|███████████████████████████████████████████████▍ | 197/270 [01:10<00:28, 2.58source/s] ! Error: decoding to str: need a bytes-like object, NoneType found <class 'TypeError'> File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\app.py", line 103, in search_novel search_novels(self) File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\novel_search.py", line 125, in search_novels app.search_results = _combine_results(results) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\novel_search.py", line 46, in _combine_results key = slugify(item.title) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\slugify\slugify.py", line 115, in slugify text = str(text, 'utf-8', 'ignore') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Nephalemzz commented 5 months ago

lightnovel-crawl> python lncrawl -s https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/

                      [#] Lightnovel Crawler v3.5.1
              https://github.com/dipu-bd/lightnovel-crawler

-> Press Ctrl + C to exit

Retrieving novel info... Exception in thread Thread-1 (read_novel_info): Traceback (most recent call last): File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\templates\madara.py", line 66, in select_chapter_tags raise Exception("No chapters on first URL") Exception: No chapters on first URL

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64qbz5n2kfra8p0\Lib\threading.py", line 1045, in _bootstrap_inner self.run() File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64qbz5n2kfra8p0\Lib\threading.py", line 982, in run self._target(*self._args, self._kwargs) File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\templates\soup\general.py", line 34, in read_novel_info for item in self.parse_chapter_list(soup): File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\templates\soup\chapter_only.py", line 13, in parse_chapter_list for tag in self.select_chapter_tags(soup): File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\templates\madara.py", line 70, in select_chapter_tags response = self.submit_form( ^^^^^^^^^^^^^^^^^ File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\scraper.py", line 230, in submit_form return self.post_response(url, data=data, headers=headers, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\scraper.py", line 211, in post_response return self.process_request( ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\scraper.py", line 130, in __process_request raise e File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\scraper.py", line 123, in process_request response.raise_for_status() File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://boxnovel.com/wp-admin/admin-ajax.php

! Error: No chapters found <class 'Exception'> File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\bots\console\integration.py", line 107, in start raise e File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\bots\console\integration.py", line 101, in start _download_novel() File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\bots\console\integration.py", line 85, in _download_novel self.app.get_novel_info() File "C:\Users\daved\OneDrive\Desktop\lightnovel-crawl\lncrawl\core\app.py", line 137, in get_novel_info raise Exception("No chapters found")

zGadli commented 5 months ago

Error: decoding to str: need a bytes-like object, NoneType found This error is fixed. But the error which occurs when you try to download from URL : novel_url. Yeah, I am not sure how to fix this.

Nephalemzz commented 4 months ago

Thanks zGadli, Well one down is better then nothing. What is the fix to this or the article with the fix steps.

Nephalemzz commented 4 months ago

Just tried to do a serch and still showing the error; decoding to str -> Press Ctrl + C to exit

? Enter novel page url or query novel: super god gene Searching: 36%|███████████████████████▍ | 102/283 [00:07<00:17, 10.13source/s]Failed to get any results, likely auth failure Searching: 73%|███████████████████████████████████████████████▎ | 206/283 [01:09<00:29, 2.60source/s] ! Error: decoding to str: need a bytes-like object, NoneType found <class 'TypeError'> File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\lncrawl\core\app.py", line 103, in search_novel search_novels(self) File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\lncrawl\core\novel_search.py", line 125, in search_novels app.search_results = _combine_results(results) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\lncrawl\core\novel_search.py", line 46, in _combine_results key = slugify(item.title) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\daved\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\slugify\slugify.py", line 115, in slugify text = str(text, 'utf-8', 'ignore') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Searching: 100%|█████████████████████████████████████████████████████████████████| 283/283 [01:20<00:00, 2.60source/s]

zGadli commented 4 months ago

@Nephalemzz If you want the fix you will have to clone the dev branch of this repo to your local machine and run it.