search

JimmXinu / FanFicFare

FanFicFare is a tool for making eBooks from stories on fanfiction and other web sites.
Other
757 stars 163 forks source link

Error when downloading from deviant art with login credentials #1052

Closed Aces-and-Jacks closed 6 months ago

Aces-and-Jacks commented 6 months ago

So when I first tested it with a star wars link from #947 it worked ok. Then I started downloading the series that I wanted. But when it got to certain chapters there was an error.

I changed a setting in the config before I started downloading but that was just for automatic login entry from what I remember. 

calibre, version 6.29.0
ERROR: Error Updating Metadata: <p>An error has occurred while FanFicFare was updating calibre's metadata for <a href='https://www.deviantart.com/travolore/art/Star-Wars-Bare-Squadron-Prologue-Death-729664865'>Unknown</a>.</p>

Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.dialogs", line 715, in do_loop
    self.foreach_function(book)
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 1356, in prep_download_loop
    story = self.get_story_metadata_only(adapter)
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 1236, in get_story_metadata_only
    adapter.getStoryMetadataOnly(get_cover=False)
  File "/Users/****/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/base_adapter.py", line 325, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/Users/****/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/base_adapter.py", line 429, in doExtractChapterUrlsAndMetadata
    return self.extractChapterUrlsAndMetadata()
  File "/Users/****/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/adapter_deviantartcom.py", line 172, in extractChapterUrlsAndMetadata
    self.performLogin(self.url)
  File "/Users/****/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/adapter_deviantartcom.py", line 111, in performLogin
    'lu_token2': soup.find('input', {'name': 'lu_token2'})['value'],
TypeError: 'NoneType' object is not subscriptable

Maybe there's a clear answer in the error message but I don't really know what to check for

JimmXinu commented 6 months ago

I'm not getting any errors. You didn't include your FFF version and there was a fix for deviantart in cf28bc26f07ceec727cbd91f3de0c33dbfd8822a.

Have you tried the latest test version?

Aces-and-Jacks commented 6 months ago

I updated to the latest FFF version, however I'm still on an older calibre version due to software limitations

JimmXinu commented 6 months ago

The latest released version, or the latest test version?

Aces-and-Jacks commented 6 months ago

The one suggested by calibre. 4.32.3

Aces-and-Jacks commented 6 months ago

I didn't get any errors the first time which was after I updated. But after messing with the personal config and I did a bulk download I started getting errors for login related stuff

Aces-and-Jacks commented 6 months ago

Is there anything I can do on my end that may help with solving this?

JimmXinu commented 6 months ago

Trying the test version as suggested would be nice.

Are you sure you have the correct username/password?

Aces-and-Jacks commented 6 months ago

I copied my password directly from where I have it, and I haven't altered it. So I'm sure about that.

I'll try the test version

Aces-and-Jacks commented 6 months ago

I get this now Status, Title, Author, Comment, URL Error, Unknown, Unknown, 'NoneType' object is not subscriptable, https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989

Aces-and-Jacks commented 6 months ago

On v4.32.10

JimmXinu commented 6 months ago

I can only replicate your error by deliberately using a bad username. I've uploaded a change to better detect and report that login failure.

If you are still seeing a login failure with a known good username/password, and were doing a number of downloads, you've probably tripped a download or login limit on the site.

Aces-and-Jacks commented 6 months ago

I'll check the username again in case. Maybe email was a bad choice for username

Aces-and-Jacks commented 6 months ago

you've probably tripped a download or login limit on the site

That could be it

Aces-and-Jacks commented 6 months ago

So you were right in it being the wrong username. I have no idea how I got it the first time then. However now I get this

Status, Title, Author, Comment, URL Error, Unknown, Unknown, 'NoneType' object has no attribute 'get_text', https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989

But it could just be a download limit as you said. Tho actually probably not since I can download other stories without logging in

JimmXinu commented 6 months ago

v4.32.11 should give you a 'login failed' at least. And FFF handles a login failed by prompting for a (better) user/pass.

If you've been trying to login with a bad username over and over I wouldn't be surprised if they are now blocking your IP or network. No idea how long that might last; hours, days, weeks?

Aces-and-Jacks commented 6 months ago

Well I just created the account. So I've only tried a little bit for restricted stories that need an account. As I said before (since I edited the comment) I'm able to download stories even without the account login. It's just that for certain ones that need the login it's not working. I'm able to login to the site but maybe somehow it's detecting the app I'm using? Or something like that.

I'll try again a different time and if I still encounter the same issue with the latest test version I'll update this

Aces-and-Jacks commented 6 months ago

But it isn't a login issue anymore since you were right in the first one being wrong. I corrected it and I'm no longer getting the login failed prompt. Tho including that helped a bunch.

Sorry if I'm being too repetitive.

JimmXinu commented 6 months ago

That's new information. In that case, I would assume your account isn't fully validated yet.

Regardless, I don't see anything further wrong with FFF.

Aces-and-Jacks commented 6 months ago

Would this help? 

Failed to initialize plugin: '/Users/****/Library/Preferences/calibre/plugins/Goodreads More Tags.zip'
Traceback (most recent call last):
  File "calibre_plugins.goodreads_more_tags.config", line 12, in <module>
    from PyQt5 import Qt as QtGui, QtCore, QtWidgets
ImportError: cannot import name 'Qt' from 'qt' (/Applications/calibre.app/Contents/Frameworks/plugins/python-lib.bypy.frozen/qt/__init__.pyc)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "calibre/customize/ui.py", line 777, in initialize_plugins
  File "calibre/customize/ui.py", line 64, in load_plugin
  File "calibre/customize/zipplugin.py", line 307, in load
  File "importlib/__init__.py", line 126, in import_module
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "calibre/customize/zipplugin.py", line 206, in exec_module
  File "calibre_plugins.goodreads_more_tags.__init__", line 12, in <module>
    from .config import plugin_prefs, KEY_INTEGRATION_ENABLED, KEY_INTEGRATION_TIMEOUT
  File "calibre/customize/zipplugin.py", line 206, in exec_module
  File "calibre_plugins.goodreads_more_tags.config", line 15, in <module>
    from PyQt4 import QtGui, QtCore, QtWidgets
ModuleNotFoundError: No module named 'PyQt4'
Using calibre Qt style: True
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
ApplicationPaletteChange event received
FFF: DEBUG: 2024-03-24 21:18:02,768: calibre_plugins.fanficfare_plugin.fff_plugin(215): Plugin FanFicFare macmenuhack file_path:/Users/****/Library/Preferences/calibre/plugins/fanficfare_macmenuhack.txt
calibre Debug log
calibre 6.29  embedded-python: True
macOS-12.7.3-arm64-arm-64bit Darwin ('64bit', '')
('Darwin', '21.6.0', 'Darwin Kernel Version 21.6.0: Sun Dec 17 22:57:33 PST 2023; root:xnu-8020.240.18.706.2~1/RELEASE_ARM64_T6000')
Python 3.10.1
OSX: ('12.7.3', ('', '', ''), 'arm64')
Interface language: None
Successfully initialized third party plugins: DeDRM (10, 0, 3) && F_rating (3, 2, 0) && FanFicFare (4, 32, 11) && Generate Cover (2, 3, 4) && Goodreads (1, 7, 9) && Goodreads Sync (1, 16, 3) && Goodreads_Rating (4, 0, 0) && Import List (1, 9, 3) && Kobo Books (1, 9, 2) && Kobo Utilities (2, 16, 13) && SmartEject (2, 5, 0) && View Manager (1, 10, 2)
calibre 6.29  embedded-python: True
macOS-12.7.3-arm64-arm-64bit Darwin ('64bit', '')
('Darwin', '21.6.0', 'Darwin Kernel Version 21.6.0: Sun Dec 17 22:57:33 PST 2023; root:xnu-8020.240.18.706.2~1/RELEASE_ARM64_T6000')
Python 3.10.1
OSX: ('12.7.3', ('', '', ''), 'arm64')
Interface language: None
Successfully initialized third party plugins: DeDRM (10, 0, 3) && F_rating (3, 2, 0) && FanFicFare (4, 32, 11) && Generate Cover (2, 3, 4) && Goodreads (1, 7, 9) && Goodreads Sync (1, 16, 3) && Goodreads_Rating (4, 0, 0) && Import List (1, 9, 3) && Kobo Books (1, 9, 2) && Kobo Utilities (2, 16, 13) && SmartEject (2, 5, 0) && View Manager (1, 10, 2)
QPA platform: cocoa
devicePixelRatio: 2.0
logicalDpi: 72.0 x 72.0
physicalDpi: 127.50000191485789 x 127.50000191485789
[0.00] Starting up...
[0.02] Showing splash screen...
[0.18] splash screen shown
[0.18] Initializing db...
[0.20] db initialized
[0.20] Constructing main UI...
DEBUG:    0.0 Kobo Utilities Configuration - CreateNewCustomColumn is supported
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 KoboUtilites::dialogs.py - loading translations
DEBUG:    0.0 KoboUtilites::action.py - loading translations
Loaded cached store plugin for: Amazon DE Kindle at version: 16
DEBUG:    0.3 No Kobo Touch, Glo or Mini appears to be connected
DEBUG:    0.3 rebuild_menus - self.supports_ratings=None, self.supports_tiles=None
DEBUG:    0.3 KoboUtilities:set_toolbar_button_tooltip - start: text='None'
DEBUG:    0.3 device_driver_name - could not load extended driver. Exception= No module named 'calibre_plugins.kobotouch_extended'
DEBUG:    0.3 KoboUtilities:set_toolbar_button_tooltip - setting to text='Utilities to use with Kobo ereaders

Driver: KoboTouch'
DEBUG:    0.3 device_driver_name - could not load extended driver. Exception= No module named 'calibre_plugins.kobotouch_extended'
[1.01] main UI initialized...
[1.01] Hiding splash screen
Starting QuickView
FFF: DEBUG: 2024-03-24 21:18:30,167: calibre_plugins.fanficfare_plugin.dialogs(493): self.extraoptions['anthology_url']:NOT FOUND
FFF: DEBUG: 2024-03-24 21:18:30,172: calibre_plugins.fanficfare_plugin.fff_plugin(1150): FanFicFare v4.32.11
FFF: DEBUG: 2024-03-24 21:18:30,289: configurable.py(1080): use_browser_cache:
FFF: DEBUG: 2024-03-24 21:18:30,289: configurable.py(1100): use_basic_cache:true
FFF: DEBUG: 2024-03-24 21:18:30,289: adapter_deviantartcom.py(154): URL: https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989
FFF: DEBUG: 2024-03-24 21:18:30,290: cache_basic.py(116): 
========== MISS (GET) BasicCache
https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989
FFF: DEBUG: 2024-03-24 21:18:30,290: fetcher_requests.py(114): 
---------- REQ (GET) RequestsFetcher
https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989
FFF: DEBUG: 2024-03-24 21:18:30,995: fetcher_requests.py(127): response code:200
FFF: DEBUG: 2024-03-24 21:18:30,996: decorators.py(112): fromcache:False
FFF: DEBUG: 2024-03-24 21:18:30,996: requestable.py(55): Encoding:utf8
FFF: DEBUG: 2024-03-24 21:18:31,117: cache_basic.py(116): 
========== MISS (GET) BasicCache
https://www.deviantart.com/users/login
FFF: DEBUG: 2024-03-24 21:18:31,117: fetcher_requests.py(114): 
---------- REQ (GET) RequestsFetcher
https://www.deviantart.com/users/login
FFF: DEBUG: 2024-03-24 21:18:31,283: fetcher_requests.py(127): response code:200
FFF: DEBUG: 2024-03-24 21:18:31,283: decorators.py(112): fromcache:False
FFF: DEBUG: 2024-03-24 21:18:31,283: requestable.py(55): Encoding:utf8
FFF: DEBUG: 2024-03-24 21:18:31,331: adapter_deviantartcom.py(101): Will now login to deviantARt as (Pioneer9672)
FFF: DEBUG: 2024-03-24 21:18:31,331: cache_basic.py(116): 
========== MISS (POST) BasicCache
https://www.deviantart.com/_sisu/do/step2
FFF: DEBUG: 2024-03-24 21:18:31,331: fetcher_requests.py(114): 
---------- REQ (POST) RequestsFetcher
https://www.deviantart.com/_sisu/do/step2
FFF: DEBUG: 2024-03-24 21:18:31,618: fetcher_requests.py(127): response code:200
FFF: DEBUG: 2024-03-24 21:18:31,618: decorators.py(112): fromcache:False
FFF: DEBUG: 2024-03-24 21:18:31,618: requestable.py(55): Encoding:utf8
FFF: DEBUG: 2024-03-24 21:18:31,671: adapter_deviantartcom.py(127): Will now send password to deviantARt
FFF: DEBUG: 2024-03-24 21:18:31,671: cache_basic.py(116): 
========== MISS (POST) BasicCache
https://www.deviantart.com/_sisu/do/signin
FFF: DEBUG: 2024-03-24 21:18:31,671: fetcher_requests.py(114): 
---------- REQ (POST) RequestsFetcher
https://www.deviantart.com/_sisu/do/signin
FFF: DEBUG: 2024-03-24 21:18:33,655: fetcher_requests.py(127): response code:200
FFF: DEBUG: 2024-03-24 21:18:33,655: decorators.py(112): fromcache:False
FFF: DEBUG: 2024-03-24 21:18:33,656: requestable.py(55): Encoding:utf8
FFF: DEBUG: 2024-03-24 21:18:33,658: cache_basic.py(116): 
========== MISS (GET) BasicCache
https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989
FFF: DEBUG: 2024-03-24 21:18:33,658: fetcher_requests.py(114): 
---------- REQ (GET) RequestsFetcher
https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989
FFF: DEBUG: 2024-03-24 21:18:33,998: fetcher_requests.py(127): response code:200
FFF: DEBUG: 2024-03-24 21:18:33,999: decorators.py(112): fromcache:False
FFF: DEBUG: 2024-03-24 21:18:34,000: requestable.py(55): Encoding:utf8
FFF: ERROR: 2024-03-24 21:18:34,155: calibre_plugins.fanficfare_plugin.dialogs(726): Exception: {'title': 'Unknown', 'author_sort': ['Unknown'], 'author': ['Unknown'], 'comments': '', 'good': False, 'status': 'Error', 'showerror': True, 'calibre_id': None, 'begin': None, 'end': None, 'comment': "'NoneType' object has no attribute 'get_text'", 'url': 'https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989', 'site': 'www.deviantart.com', 'series': '', 'added': False, 'pubdate': None, 'publisher': None, 'uniqueurl': 'https://www.deviantart.com/littlest-lily/art/Out-of-their-Element-54-990310989[None-None]', 'listorder': 0, 'collision': 'Update EPUB if New Chapters'}:'NoneType' object has no attribute 'get_text'
Traceback (most recent call last):
  File "calibre_plugins.fanficfare_plugin.dialogs", line 713, in do_loop
    self.foreach_function(book)
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 1356, in prep_download_loop
    story = self.get_story_metadata_only(adapter)
  File "calibre_plugins.fanficfare_plugin.fff_plugin", line 1236, in get_story_metadata_only
    adapter.getStoryMetadataOnly(get_cover=False)
  File "/Users/****/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/base_adapter.py", line 325, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/Users/****/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/base_adapter.py", line 429, in doExtractChapterUrlsAndMetadata
    return self.extractChapterUrlsAndMetadata()
  File "/Users/****/Library/Preferences/calibre/plugins/FanFicFare.zip/fanficfare/adapters/adapter_deviantartcom.py", line 200, in extractChapterUrlsAndMetadata
    title = soup.select_one('h1').get_text()
AttributeError: 'NoneType' object has no attribute 'get_text'
[20.40] splash screen hidden
[20.40] Started up in 20.40 seconds with 169 books

I didn't know there was a debug option in calibre. If it's just the website limiting, I'll wait for a few days and try again to see if it does the same thing.

JimmXinu commented 6 months ago

That's a different error than you had before. They've added another different message indicating 'mature content'. I've uploaded a change to address it.

Aces-and-Jacks commented 6 months ago

Thank you