Flexget / Flexget

The official FlexGet repository
http://www.flexget.com
MIT License
1.74k stars 472 forks source link

Crash due to UnicodeEncodeError: surrogates not allowed #1813

Closed programatix closed 5 years ago

programatix commented 7 years ago

Expected behaviour:

No crash?

Actual behaviour:

Crashed with a crash report marked as "BUG". I think the crash is not caused by flexget, but perhaps flexget can apply some recovery steps? Also, I have no idea which file or directory name which caused this error. If I can identify it, perhaps I can just rename the file/directory.

Steps to reproduce:

Execute the task

Config:

  FetchSubsFromPath:
    priority: 20
    filesystem:
      path: 
        - /mnt/storage1/Videos/Movies
        - /mnt/storage1/Videos/TV Series
      regexp: '.*\.(avi|mkv|mp4)$'  # only include filenames with these extensions
      recursive: yes
    accept_all: yes
    seen: local
    template: subliminal

Log:

2017-05-06 01:03 CRITICAL task          FetchSubsFromPath BUG: Unhandled error in plugin retry_failed: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed
2017-05-06 01:03 CRITICAL manager       FetchSubsFromPath An unexpected crash has occurred. Writing crash report to /opt/etc/flexget/crash_report.2017.05.06.010303941450.log. Please verify you are running the latest version of flexget by using "flexget -V" from CLI or by using version_checker plugin at http://flexget.com/wiki/Plugins/version_checker. You are currently using version 2.10.39
2017-05-06 01:03 WARNING  task          FetchSubsFromPath Aborting task (plugin: retry_failed)
2017-05-06 01:03 CRITICAL task          FetchSubsFromPath BUG: Unhandled error in plugin backlog: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed
Traceback (most recent call last):
  File "/opt/lib/python3.6/site-packages/flexget/task.py", line 483, in __run_plugin
    return method(*args, **kwargs)
  File "/opt/lib/python3.6/site-packages/flexget/event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "/opt/lib/python3.6/site-packages/flexget/plugins/filter/retry_failed.py", line 182, in on_task_filter
    filter(FailedEntry.url == entry['original_url']).first()
  File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2755, in first
    ret = list(self[0:1])
  File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2547, in __getitem__
    return list(res)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
    return self._execute_and_instances(context)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lib/python3.6/site-packages/flexget/task.py", line 612, in _execute
    self.__run_task_phase(phase)
  File "/opt/lib/python3.6/site-packages/flexget/task.py", line 453, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
  File "/opt/lib/python3.6/site-packages/flexget/task.py", line 516, in __run_plugin
    self.abort(msg, traceback=traceback)
  File "/opt/lib/python3.6/site-packages/flexget/task.py", line 370, in abort
    raise TaskAbort(reason, silent=silent)
flexget.task.TaskAbort: BUG: Unhandled error in plugin retry_failed: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/lib/python3.6/site-packages/flexget/task.py", line 483, in __run_plugin
    return method(*args, **kwargs)
  File "/opt/lib/python3.6/site-packages/flexget/event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "/opt/lib/python3.6/site-packages/flexget/plugins/input/backlog.py", line 129, in on_task_abort
    self.learn_backlog(task)
  File "/opt/lib/python3.6/site-packages/flexget/plugins/input/backlog.py", line 163, in learn_backlog
    self.add_backlog(task, entry, amount, session=session)
  File "/opt/lib/python3.6/site-packages/flexget/utils/database.py", line 31, in wrapper
    return func(*args, **kwargs)
  File "/opt/lib/python3.6/site-packages/flexget/plugins/input/backlog.py", line 144, in add_backlog
    filter(BacklogEntry.task == task.name).first()
  File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2755, in first
    ret = list(self[0:1])
  File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2547, in __getitem__
    return list(res)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
    return self._execute_and_instances(context)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed

Additional information:

liiight commented 7 years ago

Flexget does not play nicely with python 3.6 yet... Especially with db stuff

cvium commented 7 years ago

Python 3.6 is not supported.

On Sat, 6 May 2017, 20:28 programatix, notifications@github.com wrote:

Expected behaviour:

No crash? Actual behaviour:

Crashed with a crash report marked as "BUG". I think the crash is not caused by flexget, but perhaps flexget can apply some recovery steps? Also, I have no idea which file or directory name which caused this error. If I can identify it, perhaps I can just rename the file/directory. Steps to reproduce:

Execute the task Config:

FetchSubsFromPath: priority: 20 filesystem: path:

  • /mnt/storage1/Videos/Movies
  • /mnt/storage1/Videos/TV Series regexp: '.*.(avi|mkv|mp4)$' # only include filenames with these extensions recursive: yes accept_all: yes seen: local template: subliminal

Log:

2017-05-06 01:03 CRITICAL task FetchSubsFromPath BUG: Unhandled error in plugin retry_failed: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed 2017-05-06 01:03 CRITICAL manager FetchSubsFromPath An unexpected crash has occurred. Writing crash report to /opt/etc/flexget/crash_report.2017.05.06.010303941450.log. Please verify you are running the latest version of flexget by using "flexget -V" from CLI or by using version_checker plugin at http://flexget.com/wiki/Plugins/version_checker. You are currently using version 2.10.39 2017-05-06 01:03 WARNING task FetchSubsFromPath Aborting task (plugin: retry_failed) 2017-05-06 01:03 CRITICAL task FetchSubsFromPath BUG: Unhandled error in plugin backlog: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed Traceback (most recent call last): File "/opt/lib/python3.6/site-packages/flexget/task.py", line 483, in __run_plugin return method(*args, *kwargs) File "/opt/lib/python3.6/site-packages/flexget/event.py", line 23, in call return self.func(args, *kwargs) File "/opt/lib/python3.6/site-packages/flexget/plugins/filter/retry_failed.py", line 182, in on_task_filter filter(FailedEntry.url == entry['original_url']).first() File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2755, in first ret = list(self[0:1]) File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2547, in getitem return list(res) File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2855, in iter return self._execute_and_instances(context) File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute return meth(self, multiparams, params) File "/opt/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement compiled_sql, distilled_params File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context context) File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception util.reraise(exc_info) File "/opt/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise raise value File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context context) File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute cursor.execute(statement, parameters) UnicodeEncodeError: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/lib/python3.6/site-packages/flexget/task.py", line 612, in _execute self.run_task_phase(phase) File "/opt/lib/python3.6/site-packages/flexget/task.py", line 453, in run_task_phase response = self.__run_plugin(plugin, phase, args) File "/opt/lib/python3.6/site-packages/flexget/task.py", line 516, in __run_plugin self.abort(msg, traceback=traceback) File "/opt/lib/python3.6/site-packages/flexget/task.py", line 370, in abort raise TaskAbort(reason, silent=silent) flexget.task.TaskAbort: BUG: Unhandled error in plugin retry_failed: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/lib/python3.6/site-packages/flexget/task.py", line 483, in __run_plugin return method(*args, kwargs) File "/opt/lib/python3.6/site-packages/flexget/event.py", line 23, in call return self.func(*args, *kwargs) File "/opt/lib/python3.6/site-packages/flexget/plugins/input/backlog.py", line 129, in on_task_abort self.learn_backlog(task) File "/opt/lib/python3.6/site-packages/flexget/plugins/input/backlog.py", line 163, in learn_backlog self.add_backlog(task, entry, amount, session=session) File "/opt/lib/python3.6/site-packages/flexget/utils/database.py", line 31, in wrapper return func(args, kwargs) File "/opt/lib/python3.6/site-packages/flexget/plugins/input/backlog.py", line 144, in add_backlog filter(BacklogEntry.task == task.name).first() File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2755, in first ret = list(self[0:1]) File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2547, in getitem return list(res) File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2855, in iter return self._execute_and_instances(context) File "/opt/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 945, in execute return meth(self, multiparams, params) File "/opt/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement compiled_sql, distilled_params File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context context) File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception util.reraise(*exc_info) File "/opt/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise raise value File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context context) File "/opt/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute cursor.execute(statement, parameters) UnicodeEncodeError: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed

Additional information:

  • Flexget Version: 2.10.39
  • Python Version: 3.6.1
  • Installation method: pip
  • OS and version: Asuswrt-Merlin

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Flexget/Flexget/issues/1813, or mute the thread https://github.com/notifications/unsubscribe-auth/AApNLpLQYIy1PSt8p8EDGNF59zyEcawuks5r3LgQgaJpZM4NS2wG .

programatix commented 7 years ago

Hmm, this is very confusing. When I was using python 2, I'm having lots of issue due to the inclusion of dependency of lib2to3. This library, to my understanding is a library that helps python2 scripts to run on python3. Some python2 distribution does not include this because it isn't needed. This is especially true for my router. Due to this, every time I upgrade flexget, I have to edit flexget to remove reference to this library. I previously requested that a check should be done in flexget to see which version of python it is running on to determine whether to import lib2to3 or not but was declined.

Now since python3 is made available for my router, I began to use it and all the problem with lib2to3 has been resolved.

As of that, I really do not expect to see reply that state python 3.6 is not supported. Or does it mean python 3 is actually supported (pre 3.6) but not 3.6?

tubedogg commented 7 years ago

Per the install page, Python 3.3, 3.4, and 3.5 are supported.

programatix commented 7 years ago

Ah, thanks for the info. Rechecked my router installation packages. It's either python 2.7.11 or python 3.6.1. Sigh...

liiight commented 6 years ago

is this still relevant?

gazpachoking commented 5 years ago

I feel like this might not be an issue anymore. Open again or re-create if it is.