cpoppema / docker-flexget

An auto updating FlexGet container.
82 stars 39 forks source link

Python feature releases break auto-update #55

Closed 13k closed 4 years ago

13k commented 4 years ago

The auto-update scripts are installing alpine packages that depend on python. If the python version in the image (currently 3.7) is different than the actual current python version in alpine (python@edge = 3.8.0-r0), it will break everything that was installed in the image with the older python version, including pip.

Is there a reason for packages like boost-python3@edge to be in the auto-update workflow? This potentially changes the python minor version (from 3.7 to 3.8, in this case) and breaks everything. I see it is already installed in the image.

I guess updating the image with the latest python and pushing to the registry should fix it, but whenever a new minor version is released, this will always happen.

Container logs:

[s6-init] making user provided files available at /var/run/s6/etc...
[s6-init] ensuring user provided files have correct perms...
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 10-adduser: executing...
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/

Brought to you by linuxserver.io
We gratefully accept donations at:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    911
-------------------------------------

[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 20-update-flexget: executing...
Collecting pip
  Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1
Requirement already up-to-date: urllib3[socks] in /usr/lib/python3.7/site-packages (1.25.6)
Collecting PySocks!=1.5.7,<2.0,>=1.5.6; extra == "socks"
  Downloading https://files.pythonhosted.org/packages/8d/59/b4572118e098ac8e46e399a1dd0f2d85403ce8bbaad9ec79373ed6badaf9/PySocks-1.7.1-py3-none-any.whl
Installing collected packages: PySocks
Successfully installed PySocks-1.7.1
Requirement already up-to-date: chardet in /usr/lib/python3.7/site-packages (3.0.4)
Collecting cloudscraper
  Downloading https://files.pythonhosted.org/packages/e7/82/7219c4c61b02ab8376a4261bb9a3f0ea0a8ce0aa2c81e17128ea86c4543f/cloudscraper-1.2.7-py2.py3-none-any.whl (103kB)
Requirement already satisfied, skipping upgrade: pyopenssl>=17.0 in /usr/lib/python3.7/site-packages (from cloudscraper) (19.0.0)
Collecting brotli>=1.0.7
  Downloading https://files.pythonhosted.org/packages/cd/9c/7955895f5672ecc85270244582c6b53ff95bb4c24bf77bd9271d42351635/Brotli-1.0.7.zip (496kB)
Requirement already satisfied, skipping upgrade: requests>=2.9.2 in /usr/lib/python3.7/site-packages (from cloudscraper) (2.22.0)
Collecting js2py>=0.60
  Downloading https://files.pythonhosted.org/packages/25/2f/b184437870d15cdcc1cc52b73f1f7d6bb9ab2062d93c6a0b205ce8f323d3/Js2Py-0.66.tar.gz (973kB)
Collecting requests-toolbelt>=0.9.1
  Downloading https://files.pythonhosted.org/packages/60/ef/7681134338fc097acef8d9b2f8abe0458e4d87559c689a8c306d0957ece5/requests_toolbelt-0.9.1-py2.py3-none-any.whl (54kB)
Requirement already satisfied, skipping upgrade: six>=1.5.2 in /usr/lib/python3.7/site-packages (from pyopenssl>=17.0->cloudscraper) (1.12.0)
Requirement already satisfied, skipping upgrade: cryptography>=2.3 in /usr/lib/python3.7/site-packages (from pyopenssl>=17.0->cloudscraper) (2.7)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3.7/site-packages (from requests>=2.9.2->cloudscraper) (1.25.6)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/lib/python3.7/site-packages (from requests>=2.9.2->cloudscraper) (2019.9.11)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /usr/lib/python3.7/site-packages (from requests>=2.9.2->cloudscraper) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/lib/python3.7/site-packages (from requests>=2.9.2->cloudscraper) (2.8)
Collecting tzlocal>=1.2
  Downloading https://files.pythonhosted.org/packages/ef/99/53bd1ac9349262f59c1c421d8fcc2559ae8a5eeffed9202684756b648d33/tzlocal-2.0.0-py2.py3-none-any.whl
Collecting pyjsparser>=2.5.1
  Downloading https://files.pythonhosted.org/packages/48/ef/c72abcfa2c6accd03e7c89c400790fc3d908c5804d50a7c4e9ceabd74d23/pyjsparser-2.7.1.tar.gz
Requirement already satisfied, skipping upgrade: asn1crypto>=0.21.0 in /usr/lib/python3.7/site-packages (from cryptography>=2.3->pyopenssl>=17.0->cloudscraper) (1.0.1)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /usr/lib/python3.7/site-packages (from cryptography>=2.3->pyopenssl>=17.0->cloudscraper) (1.12.3)
Collecting pytz
  Downloading https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl (509kB)
Requirement already satisfied, skipping upgrade: pycparser in /usr/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyopenssl>=17.0->cloudscraper) (2.19)
Installing collected packages: brotli, pytz, tzlocal, pyjsparser, js2py, requests-toolbelt, cloudscraper
    Running setup.py install for brotli: started
    Running setup.py install for brotli: finished with status 'done'
    Running setup.py install for pyjsparser: started
    Running setup.py install for pyjsparser: finished with status 'done'
    Running setup.py install for js2py: started
    Running setup.py install for js2py: finished with status 'done'
Successfully installed brotli-1.0.7 cloudscraper-1.2.7 js2py-0.66 pyjsparser-2.7.1 pytz-2019.3 requests-toolbelt-0.9.1 tzlocal-2.0.0
Collecting irc_bot
  Downloading https://files.pythonhosted.org/packages/af/ea/efd8c29f5f91c5bac44104beffdcb0f96d2bb1d1251aea93156ea2858e32/irc_bot-1.0.34-py2.py3-none-any.whl
Collecting future
  Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB)
Requirement already satisfied, skipping upgrade: six in /usr/lib/python3.7/site-packages (from irc_bot) (1.12.0)
Installing collected packages: future, irc-bot
    Running setup.py install for future: started
    Running setup.py install for future: finished with status 'done'
Successfully installed future-0.18.2 irc-bot-1.0.34
Collecting python-telegram-bot
  Downloading https://files.pythonhosted.org/packages/26/aa/82dfedea526128a55429527ca5e43c023846e00cc3df30c8dbfebe94bbb2/python_telegram_bot-12.2.0-py2.py3-none-any.whl (349kB)
Requirement already satisfied, skipping upgrade: certifi in /usr/lib/python3.7/site-packages (from python-telegram-bot) (2019.9.11)
Collecting tornado>=5.1
  Downloading https://files.pythonhosted.org/packages/30/78/2d2823598496127b21423baffaa186b668f73cd91887fcef78b6eade136b/tornado-6.0.3.tar.gz (482kB)
Requirement already satisfied, skipping upgrade: cryptography in /usr/lib/python3.7/site-packages (from python-telegram-bot) (2.7)
Requirement already satisfied, skipping upgrade: future>=0.16.0 in /usr/lib/python3.7/site-packages (from python-telegram-bot) (0.18.2)
Requirement already satisfied, skipping upgrade: asn1crypto>=0.21.0 in /usr/lib/python3.7/site-packages (from cryptography->python-telegram-bot) (1.0.1)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in /usr/lib/python3.7/site-packages (from cryptography->python-telegram-bot) (1.12.0)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /usr/lib/python3.7/site-packages (from cryptography->python-telegram-bot) (1.12.3)
Requirement already satisfied, skipping upgrade: pycparser in /usr/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography->python-telegram-bot) (2.19)
Installing collected packages: tornado, python-telegram-bot
    Running setup.py install for tornado: started
    Running setup.py install for tornado: finished with status 'done'
Successfully installed python-telegram-bot-12.2.0 tornado-6.0.3
Collecting rarfile
  Downloading https://files.pythonhosted.org/packages/88/0b/107dde3f330d04668e126932a09002ac47348841453aa0391634381fa087/rarfile-3.1.tar.gz (121kB)
Installing collected packages: rarfile
    Running setup.py install for rarfile: started
    Running setup.py install for rarfile: finished with status 'done'
Successfully installed rarfile-3.1
Collecting subliminal
  Downloading https://files.pythonhosted.org/packages/ca/0e/acdebf8ecf30557206f2e3852c96564f7585d2f27f33f4fd5a608fcfdc2c/subliminal-2.0.5-py3-none-any.whl (67kB)
Collecting appdirs>=1.3
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting enzyme>=0.4.1
  Downloading https://files.pythonhosted.org/packages/dd/99/e4eee822d9390ebf1f63a7a67e8351c09fb7cd75262e5bb1a5256898def9/enzyme-0.4.1.tar.gz (47kB)
Requirement already satisfied, skipping upgrade: pytz>=2012c in /usr/lib/python3.7/site-packages (from subliminal) (2019.3)
Requirement already satisfied, skipping upgrade: six>=1.9.0 in /usr/lib/python3.7/site-packages (from subliminal) (1.12.0)
Requirement already satisfied, skipping upgrade: requests>=2.0 in /usr/lib/python3.7/site-packages (from subliminal) (2.22.0)
Collecting beautifulsoup4>=4.4.0
  Downloading https://files.pythonhosted.org/packages/3b/c8/a55eb6ea11cd7e5ac4bacdf92bac4693b90d3ba79268be16527555e186f0/beautifulsoup4-4.8.1-py3-none-any.whl (101kB)
Collecting click>=4.0
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
Collecting stevedore>=1.0.0
  Downloading https://files.pythonhosted.org/packages/b1/e1/f5ddbd83f60b03f522f173c03e406c1bff8343f0232a292ac96aa633b47a/stevedore-1.31.0-py2.py3-none-any.whl (43kB)
Collecting babelfish>=0.5.2
  Downloading https://files.pythonhosted.org/packages/34/b7/b36c651a9136990060ab4d6c9a32de81752123105b940b2f3b958e5c6cd0/babelfish-0.5.5.tar.gz (90kB)
Collecting dogpile.cache>=0.6.0
  Downloading https://files.pythonhosted.org/packages/ac/6a/9ac405686a94b7f009a20a50070a5786b0e1aedc707b88d40d0c4b51a82e/dogpile.cache-0.9.0.tar.gz (325kB)
Requirement already satisfied, skipping upgrade: rarfile>=2.7 in /usr/lib/python3.7/site-packages (from subliminal) (3.1)
Collecting pysrt>=1.0.1
  Downloading https://files.pythonhosted.org/packages/f6/33/16ad65a8973cb8bcb494af09ee1b9ab5ffdd6ff300bce5d3ac7d3cb1f2cc/pysrt-1.1.1.tar.gz (104kB)
Requirement already satisfied, skipping upgrade: chardet>=2.3.0 in /usr/lib/python3.7/site-packages (from subliminal) (3.0.4)
Collecting guessit>=2.0.1
  Downloading https://files.pythonhosted.org/packages/3e/4d/0f3c55ff4fbef55e6e1086e9f273cef4f70d8aa1b975f2a4d9821e0fdc34/guessit-3.1.0.tar.gz (144kB)
Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /usr/lib/python3.7/site-packages (from requests>=2.0->subliminal) (2.8)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3.7/site-packages (from requests>=2.0->subliminal) (1.25.6)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/lib/python3.7/site-packages (from requests>=2.0->subliminal) (2019.9.11)
Collecting soupsieve>=1.2
  Downloading https://files.pythonhosted.org/packages/81/94/03c0f04471fc245d08d0a99f7946ac228ca98da4fa75796c507f61e688c2/soupsieve-1.9.5-py2.py3-none-any.whl
Collecting pbr!=2.1.0,>=2.0.0
  Downloading https://files.pythonhosted.org/packages/46/a4/d5c83831a3452713e4b4f126149bc4fbda170f7cb16a86a00ce57ce0e9ad/pbr-5.4.3-py2.py3-none-any.whl (110kB)
Collecting decorator>=4.0.0
  Downloading https://files.pythonhosted.org/packages/8f/b7/f329cfdc75f3d28d12c65980e4469e2fa373f1953f5df6e370e84ea2e875/decorator-4.4.1-py2.py3-none-any.whl
Collecting rebulk==2.*
  Downloading https://files.pythonhosted.org/packages/ad/f6/3b27f7399ac8486d86e239e0a44acacfd0e0a3e5903071420c0b0cf8b465/rebulk-2.0.0.tar.gz (257kB)
Collecting python-dateutil
  Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
Installing collected packages: appdirs, enzyme, soupsieve, beautifulsoup4, click, pbr, stevedore, babelfish, decorator, dogpile.cache, pysrt, rebulk, python-dateutil, guessit, subliminal
    Running setup.py install for enzyme: started
    Running setup.py install for enzyme: finished with status 'done'
    Running setup.py install for babelfish: started
    Running setup.py install for babelfish: finished with status 'done'
    Running setup.py install for dogpile.cache: started
    Running setup.py install for dogpile.cache: finished with status 'done'
    Running setup.py install for pysrt: started
    Running setup.py install for pysrt: finished with status 'done'
    Running setup.py install for rebulk: started
    Running setup.py install for rebulk: finished with status 'done'
    Running setup.py install for guessit: started
    Running setup.py install for guessit: finished with status 'done'
Successfully installed appdirs-1.4.3 babelfish-0.5.5 beautifulsoup4-4.8.1 click-7.0 decorator-4.4.1 dogpile.cache-0.9.0 enzyme-0.4.1 guessit-3.1.0 pbr-5.4.3 pysrt-1.1.1 python-dateutil-2.8.1 rebulk-2.0.0 soupsieve-1.9.5 stevedore-1.31.0 subliminal-2.0.5
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
OK: 397 MiB in 97 packages
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/5) Installing boost-system@edge (1.71.0-r1)
(2/5) Installing libressl3.0-libcrypto@edge (3.0.2-r0)
(3/5) Installing libressl3.0-libssl@edge (3.0.2-r0)
(4/5) Installing libressl3.0-libtls@edge (3.0.2-r0)
(5/5) Installing libressl-dev@edge (3.0.2-r0)
OK: 416 MiB in 102 packages
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/testing/x86_64/APKINDEX.tar.gz
fetch http://nl.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/6) Upgrading python3@edge (3.7.4-r0 -> 3.8.0-r0)
(2/6) Upgrading py3-lxml@edge (4.4.1-r2 -> 4.4.1-r3)
(3/6) Upgrading boost-python3@edge (1.71.0-r0 -> 1.71.0-r1)
(4/6) Installing libtorrent-rasterbar@testing (1.2.2-r2)
(5/6) Installing py3-libtorrent-rasterbar@testing (1.2.2-r2)
(6/6) Upgrading python3-dev@edge (3.7.4-r0 -> 3.8.0-r0)
Executing busybox-1.30.1-r2.trigger
OK: 424 MiB in 104 packages
/var/run/s6/etc/cont-init.d/20-update-flexget: /usr/bin/pip: /usr/bin/python3.7: bad interpreter: No such file or directory
[cont-init.d] 20-update-flexget: exited 126.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
cpoppema commented 4 years ago

Thanks for reporting in such detail @13k. I will confirm @aauren's PR as soon as I can and push new images.

libtorrent-rasterbar wasn't always installed by default, it used to be installed only for TORRENT_PLUGIN="deluge". Later I learned some other FlexGet plugins also rely on libtorrent and I moved some installation instructions around. Dependency management for deluge/libtorrent-rasterbar has always been poor for alpine. I was very happy to see a python-only client for deluge, but we were still stuck with libtorrent-rasterbar. Since this is only available in @edge, whenever there's an update it almost immediately complains about not having the right python bindings for it when a different python version was installed (other than from @edge). That is why I am looking forward to aauren's solution since it builds our own python bindings instead of relying on what is available in @edge :smile:.

Is there a reason for packages like boost-python3@edge to be in the auto-update workflow? This potentially changes the python minor version (from 3.7 to 3.8, in this case) and breaks everything.

boost-python3 is a dependency for libtorrent-rasterbar (a dependency for some plugins used by FlexGet). Since we're auto-updating FlexGet (and it's plugin) I figured it to only make sense to update each other dependencies as well, thinking every latest version is compatible with each other - or if not: dependency management will sort that out. About the part where it potentially breaks for every update, yes that is totally the case right now, though it doesn't happen too often and is usually a pretty easy fix. Nonetheless, I'm looking forward to merge a long term solution like building libtorrent-rasterbar in the image and removing the auto-update (see #56).

cpoppema commented 4 years ago

New images are online and you should be able to pull them right about now :smile:

13k commented 4 years ago

thanks for taking the time to review this!

Oshayr commented 4 years ago

Thank you for the fast fix....I'm dependent on docker & flexget