Feature request: Docker

[blasphemite]

Wondering if a Docker implementation has crossed your mind.

It looks like an image exists, but it uses a modified older version of ATR and eventually has errors and no longer monitors. I fumbled around trying to create a working image with the new version, but I haven't been able to get it working (hard to do when you don't know Python!).

[Instinctlol]

Hey, that's a great idea. I'll look into this when the streamlink issues are fixed.

[evan-sm]

It's hard to dockerize since attr_cmd.py was implemented. Docker isn't friendly with interactive apps. I am myself use legacy check.py with my own Dockerfile for my own private needs :P Lot of work needed, great tool though, would PR it :P too bad I switched to golang :P

[CipherdevNL]

Hey, nice to see that my docker image was noticed! :). And sorry for the bad maintenance of the Docker image itself. Wasn't expecting it to be noticed by other people than myself.

I did some modification to the source code to realize the docker image. If I can help with realizing a generic docker image feel free to ask / contact me.

[danyal]

+1 would love a docker image. Maybe with config options to setup all the users etc and remove the interactive requirement.

[CipherdevNL]

Will try to make a start on the docker implementation in the week of December 14. If there are any wishes please tell me, so I can take that in mind when starting on the implementation.

[danyal]

Great! I only have a bit of experience with docker so not sure what the best options are for configuration of the daemon but maybe the docker config file could have comma separated list of users to follow + quality? And also allow you to specify the directory of where to save the files.

[CipherdevNL]

As promised I've created a PR for adding docker support. The PR requires manual building of the docker image and publishing to a registry.

Now it's waiting until it gets approved :)

[danyal]

@CipherdevNL Awesome. Will you also post some info on how to use?

[danyal]

@CipherdevNL Got this working from your 26_Dockerized-image branch. Is there any reason this wouldn't work as an image on dockerhub? Also, I'm assuming that in order to make changes to streamers etc you'd stop the container, make changes to the docker-compose and then rebuild the image?

[CipherdevNL]

@danyal It will just work as an image on docker hub. Only you have to push it to your own docker hub repo. There is no official repo (yet?) from instinctlol. I've my personal docker hub repo with the image you can use (https://hub.docker.com/repository/docker/cipherdevnl/automatic-twitch-recorder).

I did encounter a bug when streamers go offline. There is a fix inside the image tagged with dev. Still need to test it and create a PR for it.

Indeed, you have to stop the container to change the streamers to watch. But it's not needed to rebuild the image. Because the code will fetch the streamers from the environment variables provided through docker.

[danyal]

Ah that's helpful thanks! What is the bug you've encountered when streamers go offline?

[danyal]

A nice feature I thought of is an ENV variable that lets you specify your timezone so that file date/time could be outputted in your local timezone. Seems like this is a change needed in the docker image.

[CipherdevNL]

Sorry for the late reply, I'm lately busy with work and a school project beside it (deadline is approaching 😅).

When streamers go offline the automatic stream watcher will hang indefinitely until the container gets restarted. I've traced it down to a wrong implementation of streamlink inside this project. (Could be that this issue occurred after a specific version upgrade).

When a streamer goes offline streamlink will give 3-4 warnings about a missing HSL stream. When it's missing the streams from Twitch it will not provide any data through the file descriptors. After a while Python will time out and throw a read timeout exception inside the watcher thread.

Due to the exception it will not safely close the streamlink descriptor, so it will keep trying to fetch the latest HLS stream from twitch and will keep throwing warnings. This will result in a massive container log (unusable on Synology devices) and a broken watcher.

With the fix that's inside the dev container this issue got solved. There is one new issue I've encountered due to a bad internet connection. Which will also hang the recorder. When I have time I will take a look at that also.

New unhandled exception:

Exception in thread Thread-33004:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 172, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f19547d8128>: Failed to establish a new connection: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 446, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.twitch.tv', port=443): Max retries exceeded with url: /helix/streams?first=100&user_id=108947379&user_id=25771813 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f19547d8128>: Failed to establish a new connection: [Errno -3] Try again',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.6/threading.py", line 1182, in run
    self.function(*self.args, **self.kwargs)
  File "/opt/automatic-twitch-recorder/automatic_twitch_recorder/daemon.py", line 98, in _check_streams
    streams_info = automatic_twitch_recorder.twitch.get_stream_info(*user_ids)
  File "/opt/automatic-twitch-recorder/automatic_twitch_recorder/twitch.py", line 63, in get_stream_info
    r = requests.get(get_user_id_url, headers=get_twitch_auth())
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.twitch.tv', port=443): Max retries exceeded with url: /helix/streams?first=100&user_id=108947379&user_id=25771813 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f19547d8128>: Failed to establish a new connection: [Errno -3] Try again',))

Regarding the timezone support. Don't know how you can do it outside the docker container. Could you give an example on how to do it? If we have to implement in through code please file a new feature request :).

[CipherdevNL]

By looking through the open issues I noted that some else has reported (#36) my (fixed) issue.

[danyal]

Thanks for the info and all your work on this. I've forked it as well and am tinkering around with timezone. The way I've seen it done is in the docker compose you specify timezone as a variable and then that get's used in the code. I think watcher.py line 30 needs to be modified.

[danyal]

@CipherdevNL I was wrong, no code change is needed. You just need to install tzdata in the docker image and then you can pass the TZ env variable in docker compose. See here: https://medium.com/rahasak/configure-docker-timezone-in-linux-71893fe51499

[danyal]

Tried the latest image and it worked for a while but then got this error:

Exception in thread Thread-1276: Traceback (most recent call last): File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/local/lib/python3.6/threading.py", line 1182, in run self.function(*self.args, **self.kwargs) File "/opt/automatic-twitch-recorder/automatic_twitch_recorder/daemon.py", line 103, in _check_streams self.streamers[streamer_name].update({'stream_info': stream_info})  KeyError: ''