Upgrading to version 400 breaks Twitter and e621 subs

bbappserver commented 4 years ago

Both the Twitter(nitter) and e621 parsers break for subscriptions(but not new galleries) after upgrading to 400, they worked fine in 399. Something about the upgrade seems to have deleted parsers and URL classe about specific urls cached but not longer usable because of the missing parsers and classes.

Environment

Hydrus version: 400 Platform: Macos 10.13 OS specifics:

Steps to reproduce

Have a sub in twitter or e621 prior to upgrade to 400, make sure it has run to colect some weird urls
Upgrade to 400
Let subs start or nudge them.

Error message / Log file

URLClassException
Could not find a parser for https://e621.net/post/index/1/.<SomeUsername>!
Traceback (most recent call last):
  File "./git/hydrusqt/hydrus/client/networking/ClientNetworkingDomain.py", line 668, in _GetURLToFetchAndParser
    ( parser_url_class, parser_url ) = self._GetNormalisedAPIURLClassAndURL( url )
  File "./git/hydrusqt/hydrus/client/networking/ClientNetworkingDomain.py", line 568, in _GetNormalisedAPIURLClassAndURL
    raise HydrusExceptions.URLClassException( 'Could not find a URL Class for ' + url + '!' )
hydrus.core.HydrusExceptions.URLClassException: Could not find a URL Class for https://e621.net/post/index/1/.<SomeUsername>!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./git/hydrusqt/hydrus/client/importing/ClientImportSubscriptions.py", line 1347, in Sync
    self._SyncQueries( job_key )
  File "./git/hydrusqt/hydrus/client/importing/ClientImportSubscriptions.py", line 605, in _SyncQueries
    self._SyncQuery( job_key, gug, query_header, query_log_container, status_prefix )
  File "./git/hydrusqt/hydrus/client/importing/ClientImportSubscriptions.py", line 687, in _SyncQuery
    ( login_ok, login_reason ) = query_header.GalleryLoginOK( HG.client_controller.network_engine, self._name )
  File "./git/hydrusqt/hydrus/client/importing/ClientImportSubscriptionQuery.py", line 372, in GalleryLoginOK
    nj = self._example_gallery_seed.GetExampleNetworkJob( self._GenerateNetworkJobFactory( subscription_name ) )
  File "./git/hydrusqt/hydrus/client/importing/ClientImportGallerySeeds.py", line 195, in GetExampleNetworkJob
    ( url_to_check, parser ) = HG.client_controller.network_engine.domain_manager.GetURLToFetchAndParser( self.url )
  File "./git/hydrusqt/hydrus/client/networking/ClientNetworkingDomain.py", line 1600, in GetURLToFetchAndParser
    result = self._GetURLToFetchAndParser( url )
  File "./git/hydrusqt/hydrus/client/networking/ClientNetworkingDomain.py", line 672, in _GetURLToFetchAndParser
    raise HydrusExceptions.URLClassException( 'Could not find a parser for ' + url + '!' + os.linesep * 2 + str( e ) )
hydrus.core.HydrusExceptions.URLClassException: Could not find a parser for https://e621.net/post/index/1/.<SomeUsername>!

Could not find a URL Class for https://e621.net/post/index/1/.<SomeUsername>!

Could not find a URL Class for https://e621.net/post/index/1/.<SomeUsername>!

Could not find a URL Class for https://twitter.com/i/profiles/show/<redactedusername>/media_timeline

Could not find a URL Class for https://twitter.com/i/profiles/show/<redactedusername>/timeline/tweets

Error corrected for twitter by reinstalling API parserand URL classes from an earlier versions static defaults folders.

The reason for Twitter's issue seems logical given that nitter recently replaced it. The reason Why e621 stopped working for the same reason is unclear. I have yet to resolve and correct e621.

Zweibach commented 4 years ago

According to an unusual error with pre-run domain checking, exposed by the new subscription code and e621 subs, where the gallery url has also recently changed, is now fixed this should now be changed as of v401.

ShadowJonathan commented 4 years ago

@bbappserver please confirm if this isn't the case

hydrusnetwork / hydrus