Incorrect permissions on temp directories

jmuchovej commented 4 years ago

I believe this is related to the container, if not – would appreciate some guidance.

Flexget seems to source from 4 different tmp directories:

/tmp
/var/tmp
/usr/tmp (doesn't exist)
/run/s6/services/flexget

The existing tmp directories don't have 777 as permissions, but rather 755 – seemingly causing Flexget to error out "believing" it can't use one of the tmp directories.

The fix seems to be just setting the permissions to 777.

cpoppema commented 4 years ago

Hello @ionlights,

I don't remember ever seeing issues regarding tmp directories before, so I'm curious:

are there any errors being reported in the logfiles or that you can see in docker logs flexget ?
what version of flexget are you running where you see these problems ?
do you know of any configuration you have that might rely on tmp directories ?

jmuchovej commented 4 years ago

`docker logs flexget`

2020-04-19 12:00:30 CRITICAL plugin                        Exception while loading plugin flexget.plugins.output.download
Traceback (most recent call last):
  File "/usr/bin/flexget", line 8, in <module>
    sys.exit(main())
    |   |    -> <function main at 0x7fa0bc58a5f0>
    |   -> <built-in function exit>
    -> <module 'sys' (built-in)>
  File "/usr/lib/python3.7/site-packages/flexget/__init__.py", line 41, in main
    manager.start()
    |       -> <function Manager.start at 0x7fa0ba4893b0>
    -> <flexget.manager.Manager object at 0x7fa0bbcbe190>
  File "/usr/lib/python3.7/site-packages/flexget/manager.py", line 373, in start
    self.initialize()
    |    -> <function Manager.initialize at 0x7fa0ba4157a0>
    -> <flexget.manager.Manager object at 0x7fa0bbcbe190>
  File "/usr/lib/python3.7/site-packages/flexget/manager.py", line 215, in initialize
    extra_components=[os.path.join(self.config_base, 'components')],
                      |  |    |    |    -> '/root/.flexget'
                      |  |    |    -> <flexget.manager.Manager object at 0x7fa0bbcbe190>
                      |  |    -> <function join at 0x7fa0bc655170>
                      |  -> <module 'posixpath' from '/usr/lib/python3.7/posixpath.py'>
                      -> <module 'os' from '/usr/lib/python3.7/os.py'>
  File "/usr/lib/python3.7/site-packages/flexget/plugin.py", line 555, in load_plugins
    _load_plugins_from_dirs(extra_plugins)
    |                       -> ['/root/.flexget/plugins', '/usr/lib/python3.7/site-packages/flexget/plugins']
    -> <function _load_plugins_from_dirs at 0x7fa0baa00320>
  File "/usr/lib/python3.7/site-packages/flexget/plugin.py", line 473, in _load_plugins_from_dirs
    _import_plugin(module_name, plugin_path)
    |              |            -> PosixPath('/usr/lib/python3.7/site-packages/flexget/plugins/output/download.py')
    |              -> 'flexget.plugins.output.download'
    -> <function _import_plugin at 0x7fa0ba9f7710>
> File "/usr/lib/python3.7/site-packages/flexget/plugin.py", line 424, in _import_plugin
    import_module(module_name)
    |             -> 'flexget.plugins.output.download'
    -> <function import_module at 0x7fa0bc60dcb0>
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           |          |           |    |        |        -> 0
           |          |           |    |        -> None
           |          |           |    -> 0
           |          |           -> 'flexget.plugins.output.download'
           |          -> <function _gcd_import at 0x7fa0bc727b00>
           -> <module 'importlib._bootstrap' (frozen)>
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3.7/site-packages/flexget/plugins/output/download.py", line 25, in <module>
    class PluginDownload:
  File "/usr/lib/python3.7/site-packages/flexget/plugins/output/download.py", line 103, in PluginDownload
    tmp_path=tempfile.gettempdir(),
             |        -> <function gettempdir at 0x7fa0bb546f80>
             -> <module 'tempfile' from '/usr/lib/python3.7/tempfile.py'>
  File "/usr/lib/python3.7/tempfile.py", line 294, in gettempdir
    tempdir = _get_default_tempdir()
              -> <function _get_default_tempdir at 0x7fa0bb546cb0>
  File "/usr/lib/python3.7/tempfile.py", line 229, in _get_default_tempdir
    dirlist)
    -> ['/tmp', '/var/tmp', '/usr/tmp', '/run/s6/services/flexget']

FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/run/s6/services/flexget']
Could not start manager: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/run/s6/services/flexget']

Flexget Version

root@flexget:/# flexget -V
3.1.51
You are on the latest release.

Config

I'm not sure. This happens right from boot in the container. Based on the traceback, it seems like it happens when loading flexget.plugins.output.download (I didn't see this earlier, but I didn't scroll back on the traceback).

cpoppema commented 4 years ago

I confirmed I tested with the same version, 3.1.51, and configured a basic download-option as well (plugins.output.download), but starting the container, I did not see this error.

Following the stacktrace, the issue starts here:

  File "/usr/lib/python3.7/site-packages/flexget/plugins/output/download.py", line 103, in PluginDownload
    tmp_path=tempfile.gettempdir(),

To test if I would have any issues with just this command, I've ran it separate from flexget:

$ docker exec flexget python3 -c 'import tempfile; print(tempfile.gettempdir())'
/tmp

This seems to work for me. Does this separate command also fail for you ?

As I've understand there can be three main reasons you would get this error:

1) permissions 2) disk space 3) inode limit

So let's start with the first, in my running container the permissions are 777, which is already different than from what you mentioned:

 $ docker exec flexget ls -l / | grep tmp
drwxrwxrwt   1 root root   4096 Apr 20 09:18 tmp

Do you per chance have a volume mount in your docker configuration to /tmp ?

jmuchovej commented 4 years ago

To test if I would have any issues with just this command, I've ran it separate from flexget:
$ docker exec flexget python3 -c 'import tempfile; print(tempfile.gettempdir())'
/tmp

We get the same results here.

As I've understand there can be three main reasons you would get this error:

permissions

disk space

inode limit

So, permissions appear different. I'm skeptical disk space is a problem, as this happens on boot, and I have a few hundred GB available on the partition docker works under. I'm not sure about the inode limit, do you have some resources on discerning this?

$ docker exec flexget ls -l / | grep tmp
drwxrwxrwt   1 root root   4096 Apr 20 09:18 tmp

I have a different result:

$ docker exec flexget ls -l / | grep tmp
drwxr-xr-x   1 root root   4096 Apr 20 12:03 tmp

Do you per chance have a volume mount in your docker configuration to /tmp ?

No. I don't believe so, I've also attached a snippet of my docker-compose as well:

A snippet of my `docker-compose` that launches Flexget

```yaml ... flexget: # https://hub.docker/.com/r/linuxserver/flexget image: cpoppema/docker-flexget restart: unless-stopped container_name: flexget hostname: flexget # ports: [ "5050:5050", ] volumes: - ./flexget:/config:rw - ./letsencrypt:/certs:ro - ../media/Downloads:/downloads:rw env_file: ["env.global", "env.flexget", ] ... ```

Something curious I've noticed is that I only have these permissions in containers not from linuxserver. It seems like the container hasn't been updated for 3mo or so, so I'm hesitant to say it's an issue with the container.

jmuchovej commented 4 years ago

Weird. I deleted the image and pulled it back down. Now seems as if everything's all good. Permissions on /tmp match what's expected (drwxrwxrwt).

Guess this might have been some weird artifact of some changes I made. Sorry for the hassle. 😅

cpoppema / docker-flexget