Add docker support for standalone database

[sepo83]

With this pull request one can build a docker-container that runs a standalone database for plugin.video.mediathekview. The container then includes a MariaDB and this repo (esp. the mvupdate3 and dependent data) as well as a cron-job for regular updates. All can be configured via environment variables when running the container. I also updated the documentation for more details.

The dockerimage is based on linuxserver/mariadb (see https://docs.linuxserver.io/images/docker-mariadb) in order to make cron work.

Remark: The usual docker approach would run the MariaDB and plugin.video.mediathekview in a separate containers. I decided to put it in one container in order to keep the "footprint" small :smile: However, I think running a separate DB-container could be added easily as an option, if needed.

[codingPF]

Sounds nice...let me check this and come back to you

[codingPF]

There is only one Cronjob at 4am setup. I would have expected something like every second hour...any reasons for one update per day or maybe i did miss something?

[sepo83]

hmmm.. the plan was to load the cronjob with the timespec you set in the docker variable CRON_TIMESPEC. Let me check this.

EDIT: On my docker container I set CRON_TIMESPEC to 13 2,14 * * * and the log tells me that it was running around 2 am and 2 pm. However, the default value is set to 4:15 am. Could you please check if setting CRON_TIMESPEC to a valid cron timespec works for you (run the container with e.g. -e CRON_TIMESPEC="13 2,14 * * * ")? I could also set the default value to something more suitable...

[codingPF]

Yes, the question was more on the default value... I would expect something like hourly / every second hour as a default Why did you choose this default?

question: why do you include "-f" on the cmd? python3 /plugin.video.mediathekview/mvupdate3 mysql -vv -d $MYSQL_DATABASE -u $MYSQL_USER -p $MYSQL_PASSWORD -f I would believe the default (without parameter f) should work fine?

question: I remember from some playing around with VMs that mysql keeps growing due to binlog. Did you have a look at this for your image? I can see these growing constantly...

-rw-rw---- 1 abc abc 30731 Jan 24 20:34 mariadb-bin.000001 -rw-rw---- 1 abc abc 1537 Jan 24 20:34 mariadb-bin.000002 -rw-rw---- 1 abc abc 110055742 Jan 24 20:40 mariadb-bin.000003 -rw-rw---- 1 abc abc 111598446 Jan 24 20:41 mariadb-bin.000004 -rw-rw---- 1 abc abc 110021367 Jan 24 20:42 mariadb-bin.000005 -rw-rw---- 1 abc abc 106236380 Jan 24 21:07 mariadb-bin.000006 -rw-rw---- 1 abc abc 105313801 Jan 24 21:17 mariadb-bin.000007 -rw-rw---- 1 abc abc 333 Jan 24 21:32 mariadb-bin.index -rw-rw---- 1 abc abc 105570222 Jan 24 21:32 mariadb-bin.000008 -rw-rw---- 1 abc abc 26090392 Jan 24 21:33 mariadb-bin.000009

Question: Can we go to INFO level only? (just one -v)

[sepo83]

Using 4:15am as default has some "historic" reasons. When I started using mvupdate some time ago it was running on a raspberry pi and it took some time (and quite some performance). So I run it at night when probably no one was using the server.

-f was chosen to give more flexibility (from my understanding it forces the update if the preset interval has not passed yet). So it might be needed if you want to updated in shorter intervals, e.g. when trying something. I think if someone uses mvupdate via commandline resp. in docker container, he is an "advanced" user and knows what to do. I would suggest to create another docker environment variable MVUPDATE_FLAGS which is empty on default and where one can define all optional flags to be used (except -d, -u, -d which are filled via the other docker variables). This would also also answer the question regarding -v :smile:

Regarding the growing memory... No, actually did not check this yet... However, the maria-db image which I am using as a base has a config file under /config/custom.cnf where you can customize a lot of things. It contains e.g.:

expire_logs_days        = 10
max_binlog_size         = 100M

From my understanding this should limit the size of binlog and it seems to work on my container:

drwxrwxrwx 2 abc abc 4.0K Jan 25 02:18 .
drwxr-xr-x 3 abc abc 4.0K Jan  3 20:49 ..
-rw-rw---- 1 abc abc 102M Jan 16 02:14 mariadb-bin.000135
-rw-rw---- 1 abc abc 103M Jan 16 02:14 mariadb-bin.000136
...
-rw-rw---- 1 abc abc 101M Jan 25 02:18 mariadb-bin.000231
-rw-rw---- 1 abc abc 101M Jan 25 02:18 mariadb-bin.000232
-rw-rw---- 1 abc abc  16M Jan 25 14:13 mariadb-bin.000233
-rw-rw---- 1 abc abc 3.6K Jan 25 02:18 mariadb-bin.index

Perhaps you can have a look at this config file. I am open for suggestions here.

[codingPF]

If you dont not mind i would drop the -f to keep the update logic inside of the application code. When going to update every hour this should do incremental updates which are rather quick (maybe k rows per run) and should not harm your server. I did switch off binlog completly on my test server. Actually 10 * 100MB for nothing seems quite a lot me. I do not have any docker experience or clou how to change the config file but i would suggest switch it of or limit even further. What do you think?

[sepo83]

Ok, I will change the call of mvupdate to:

python3 /plugin.video.mediathekview/mvupdate3 mysql -v -d $MYSQL_DATABASE -u $MYSQL_USER -p $MYSQL_PASSWORD 

I will also change the default of CRON_TIMESPEC to "0 /1 "

Regarding binlog... technically seen I could try to edit (regex) /config/custom.cnf on every startup of the container with the same script I use to load cronjob. However, I think that config file is meant to be changed manually. The user will only have to do this once since it is stored in a persistent docker-volume like the mysql db itself. Another thing would be: If the guys from linuxserver.io change the maria-db base image in a way that they change the content/syntax of custom.cnf then editing the config file automatically may break the container. Actually, I would prefer to add this change as a recommendation in the documentation. Something like: In order to safe memory change the following paramters in /config/custom.cnf:

expire_logs_days        = 0
max_binlog_size         = 0

What do you think?

EDIT: After RTFM :facepalm: : ... Setting expire_log_days = 0 will actually disable limiting the memory consumption. Replace log_bin = /config/log/mysql/mariadb-bin with skip-log-bin will disable binlog.

[sepo83]

I added all agreed changes to the pull request. Could you please also test my changes. Are there any open points left?

[codingPF]

Thank you...I will have a look at it...I am currently preparing the new release - please excuse my response time.... Do you mind to change crontab to 4-22/1 because the MV Backend is not running between 22-4 I did some research on the binlog configuration of the image but could not find a simple solution. So maybe it is best to go with the explaination on the readme and leave it to the user.

[sepo83]

No problem... I did change the default cron timespec.

[codingPF]

Testng from my side worked fine....fixed typos on the readme and added vi support to edit configuration files

[major-mayer]

Thanks for the great Dockerfile. I've just tested it and it works very well. Previously I've used my own MariaDB Docker container and run the script there inside, but with this addition the process is much cleaner.

One improvement that i could think of is pushing the Docker image to hub.docker.com. This would simplify the process even more and you can build the image automatically when the Dockerfile is changed/ a new commit was made.

[codingPF]

Thanks for the compliment...that goes all to @sepo83 for his work! I am not sure how the (version) update process works when having an image at hub.docker.com but I will check by time