Closed bossanova808 closed 3 months ago
Pretty sure there is a Kodi setting to wait for network sql on boot
On Sun, 14 Jul 2024, 11:31 Jeremy Daalder, @.***> wrote:
I am having issues with system/container restarts - it seems the Kodi dockers I run (two) sometimes start before the DB is actually ready to connect.
In my compose I have a healthcheck on the db container (MariaDB)
db: image: mariadb:latest container_name: db environment: - PUID=${UID} - PGID=${GUID} - TZ=${TZ} - "MARIADB_ROOT_PASSWORD_HASH=${MARIADB_ROOT_PASSWORD_HASH}" - MARIADB_AUTO_UPGRADE=1 volumes: - ${MARIADB_OFFICIAL_CONF}:/etc/mysql/conf.d - ${MARIADB_OFFICIAL_DATA}:/var/lib/mysql ports: - 3306:3306 restart: unless-stopped # Healthcheck from https://mariadb.org/mariadb-server-docker-official-images-healthcheck-without-mysqladmin/ healthcheck: test: [ "CMD", "healthcheck.sh", "--connect", "--innodb_initialized" ] start_period: 1m start_interval: 10s interval: 1m timeout: 5s retries: 3
And the Kodi containers depend on this:
kodi-adults: container_name: kodi-adults image: matthuisman/kodi-headless:Omega depends_on: db: condition: service_healthy environment: - PUID=${UID} - PGID=${GUID} - TZ=${TZ} volumes: - ${KODI_ADULTS_CONFIG}:/config/.kodi:rw # All offset by one to avoid LMS... ports: - 8081:8080 # WebUI - 9091:9090 # Websockets - 9778:9777 # esall interface port restart: unless-stopped
...but that is not enough, it seems.
When this happens, I see this in the Kodi logs:
2024-07-12 17:37:06.059 T:182 info
: Running database version Addons33 2024-07-12 17:37:06.062 T:182 info : Running database version ViewModes6 2024-07-12 17:37:06.063 T:182 info : Running database version Textures13 2024-07-12 17:37:06.065 T:182 debug : connect replacing configured host db with resolved host 172.18.0.9 2024-07-12 17:37:06.067 T:182 error : Unable to open database: kodi_music_adults83 [2003](Can't connect to MySQL server on '172.18.0.9:3306' (111)) 2024-07-12 17:37:06.067 T:182 debug : connect replacing configured host db with resolved host 172.18.0.9 2024-07-12 17:37:06.067 T:182 error : Unable to open database: kodi_music_adults82 [2003](Can't connect to MySQL server on '172.18.0.9:3306' (111)) .... 2024-07-12 17:37:06.073 T:182 error : Unable to create new database If I manually restart the containers a few minutes after e.g. a system reboot, then they work as expected and connect to the DB normally (and have been running these for some time now successfully, bar this issue that has appeared in recent months)
I'm struggling to work out a solution here, as docker isn't so great at this sort of thing as far as I can tell, I can't e.g. delay the startup of these, and the MariaDB container must be reporting healthy before it's really ready, I suppose, which allows the Kodi containers to then start.
Ideally there would be a healthcheck on the Kodi containers that actually tested this and restarted themselves if the DB is not actually ready to accept a connection...but I don't think there are tools within that container to check this (e.g. some manual way of checking that db:3306 is connectable)
Any ideas?
— Reply to this email directly, view it on GitHub https://github.com/matthuisman/docker-kodi-headless/issues/57, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPQAKK2SAIEPSSBVZKAD63ZMG2DXAVCNFSM6AAAAABK2U2WQKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQYDOMJZGYYTEOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Hmm... looks like there are two possibilities:
https://kodi.wiki/view/Wake_on_lan
And a GUI setting:
<setting id="powermanagement.waitfornetwork" default="true">120</setting>
I will give those a try & report back. But I am not sure that will be enough as it depends what sort of test it is actually running...wait for network is not per se enough, it really needs to wait for a successful DB connection.
(Most of the Kodi systems (e.g. the elecs) - implement their own wait on network, at the system level, before Kodi start, I believe...)
Wake on Lan may work https://discourse.osmc.tv/t/wait-to-start-kodi-until-it-has-mysql-alive-signal/8466/4
On Sun, 14 Jul 2024, 12:01 Jeremy Daalder, @.***> wrote:
Hmm... looks like there are two possibilities:
https://kodi.wiki/view/Wake_on_lan
And a GUI setting:
120 I will give those a try & report back. But I am not sure that will be enough as it depends what sort of test it is actually running...wait for network is not per se enough, it really needs to wait for a successful DB connection.
(Most of the Kodi systems (e.g. the elecs) - implement their own wait on network, at the system level, before Kodi start, I believe...)
— Reply to this email directly, view it on GitHub https://github.com/matthuisman/docker-kodi-headless/issues/57#issuecomment-2227141864, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPQAKOW75NYZQZTN3FSURTZMG5WHAVCNFSM6AAAAABK2U2WQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRXGE2DCOBWGQ . You are receiving this because you commented.Message ID: @.***>
Unfortunately neither of these work following a reboot:
wakeonlan.xml
<onaccesswakeup>
<netinittimeout>120</netinittimeout>
<netsettletime>500</netsettletime>
<wakeup>
<host>db</host>
<!-- <mac></mac> -->
<pingport>3306</pingport>
<pingmode>1</pingmode>
<timeout>600</timeout>
<waitonline>38</waitonline>
<waitonline2>40</waitonline2>
<waitservices>10</waitservices>
</wakeup>
</onaccesswakeup>
In the log:
2024-07-14 13:21:52.135 T:171 info <general>: WakeOnAccess - Load settings :
2024-07-14 13:21:52.135 T:171 info <general>: WakeOnAccess - Enabled:FALSE
2024-07-14 13:21:52.135 T:171 info <general>: -Network init timeout : [120] sec
2024-07-14 13:21:52.135 T:171 info <general>: -Network settle time : [500] ms
2024-07-14 13:21:52.135 T:171 error <general>: Missing <mac> tag or it's empty
...seems one needs a MAC address. Not sure if those are determinate in docker compose scenarios....
Just put in any Mac address? The wol failing doesn't matter. But it should wait after that for the MySQL. See that thread I linked
On Sun, 14 Jul 2024, 15:24 Jeremy Daalder, @.***> wrote:
Unfortunately neither of these work following a reboot:
wakeonlan.xml
120 500 db 3306 1 600 38 40 10 In the log:
2024-07-14 13:21:52.135 T:171 info
: WakeOnAccess - Load settings : 2024-07-14 13:21:52.135 T:171 info : WakeOnAccess - Enabled:FALSE 2024-07-14 13:21:52.135 T:171 info : -Network init timeout : [120] sec 2024-07-14 13:21:52.135 T:171 info : -Network settle time : [500] ms 2024-07-14 13:21:52.135 T:171 error : Missing tag or it's empty ...seems one needs a MAC address. Not sure if those are determinate in docker compose scenarios....
— Reply to this email directly, view it on GitHub https://github.com/matthuisman/docker-kodi-headless/issues/57#issuecomment-2227181452, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPQAKO6FD5AXV24S6P7XJLZMHVOBAVCNFSM6AAAAABK2U2WQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRXGE4DCNBVGI . You are receiving this because you commented.Message ID: @.***>
Yep, I read the thread, but in this case it definitely does not work...
I added a Mac address to wakeonlan.xml
:
<onaccesswakeup>
<netinittimeout>120</netinittimeout>
<netsettletime>500</netsettletime>
<wakeup>
<host>db</host>
<mac>26:5A:DC:DD:EE:D3</mac>
<pingport>3306</pingport>
<pingmode>1</pingmode>
<timeout>600</timeout>
<waitonline>38</waitonline>
<waitonline2>40</waitonline2>
<waitservices>10</waitservices>
</wakeup>
</onaccesswakeup>
I also needed this in advancedsettings.xml
to force it to enabled:
<powermanagement>
<wakeonaccess>true</wakeonaccess>
</powermanagement>
To be sure, I added the MAC address to the container:
db:
image: mariadb:latest
container_name: db
mac_address: 26:5A:DC:DD:EE:D3
(etc as above)
In the kodi log I then see this, in theory, working:
2024-07-14 13:57:10.912 T:165 info <general>: WakeOnAccess - Load settings :
2024-07-14 13:57:10.912 T:165 info <general>: WakeOnAccess - Enabled:TRUE
2024-07-14 13:57:10.912 T:165 info <general>: -Network init timeout : [120] sec
2024-07-14 13:57:10.912 T:165 info <general>: -Network settle time : [500] ms
2024-07-14 13:57:10.912 T:165 info <general>: Registering wakeup entry:
2024-07-14 13:57:10.912 T:165 info <general>: HostName : db
2024-07-14 13:57:10.912 T:165 info <general>: MacAddress : 26:5A:DC:DD:EE:D3
2024-07-14 13:57:10.912 T:165 info <general>: PingPort : 3306
2024-07-14 13:57:10.912 T:165 info <general>: PingMode : 1
2024-07-14 13:57:10.912 T:165 info <general>: Timeout : 600 (sec)
2024-07-14 13:57:10.912 T:165 info <general>: WaitForOnline : 38 (sec)
2024-07-14 13:57:10.912 T:165 info <general>: WaitForOnlineEx : 40 (sec)
2024-07-14 13:57:10.912 T:165 info <general>: WaitForServices : 10 (sec)
But then soon after:
2024-07-14 13:57:11.529 T:181 debug <general>: connect replacing configured host db with resolved host 172.18.0.9
2024-07-14 13:57:11.538 T:181 error <general>: Unable to open database: kodi_music_adults83 [2003](Can't connect to MySQL server on '172.18.0.9:3306' (111))
2024-07-14 13:57:11.538 T:181 debug <general>: connect replacing configured host db with resolved host 172.18.0.9
2024-07-14 13:57:11.538 T:181 error <general>: Unable to open database: kodi_music_adults82 [2003](Can't connect to MySQL server on '172.18.0.9:3306' (111))
So the db
container is, seemingly, up and responding to the ping on 3306
... but not actually ready for connections (?!)
If I manually restart the Kodi containers after the system is up a minute or two, they then work fine.
Why is the db to slow to come up?
On Sun, 14 Jul 2024, 16:09 Jeremy Daalder, @.***> wrote:
Yep, I read the thread, but in this case it definitely does not work...
I added a Mac address to wakeonlan.xml:
120 500 db 26:5A:DC:DD:EE:D3 3306 1 600 38 40 10 I also needed this in advancedsettings.xml to force it to enabled:
<powermanagement> <wakeonaccess>true</wakeonaccess> </powermanagement>
To be sure, I added the MAC address to the container:
db: image: mariadb:latest container_name: db mac_address: 26:5A:DC:DD:EE:D3
(etc as above)
In the kodi log I then see this, in theory, working:
2024-07-14 13:57:10.912 T:165 info
: WakeOnAccess - Load settings : 2024-07-14 13:57:10.912 T:165 info : WakeOnAccess - Enabled:TRUE 2024-07-14 13:57:10.912 T:165 info : -Network init timeout : [120] sec 2024-07-14 13:57:10.912 T:165 info : -Network settle time : [500] ms 2024-07-14 13:57:10.912 T:165 info : Registering wakeup entry: 2024-07-14 13:57:10.912 T:165 info : HostName : db 2024-07-14 13:57:10.912 T:165 info : MacAddress : 26:5A:DC:DD:EE:D3 2024-07-14 13:57:10.912 T:165 info : PingPort : 3306 2024-07-14 13:57:10.912 T:165 info : PingMode : 1 2024-07-14 13:57:10.912 T:165 info : Timeout : 600 (sec) 2024-07-14 13:57:10.912 T:165 info : WaitForOnline : 38 (sec) 2024-07-14 13:57:10.912 T:165 info : WaitForOnlineEx : 40 (sec) 2024-07-14 13:57:10.912 T:165 info : WaitForServices : 10 (sec) But then soon after:
2024-07-14 13:57:11.529 T:181 debug
: connect replacing configured host db with resolved host 172.18.0.9 2024-07-14 13:57:11.538 T:181 error : Unable to open database: kodi_music_adults83 [2003](Can't connect to MySQL server on '172.18.0.9:3306' (111)) 2024-07-14 13:57:11.538 T:181 debug : connect replacing configured host db with resolved host 172.18.0.9 2024-07-14 13:57:11.538 T:181 error : Unable to open database: kodi_music_adults82 [2003](Can't connect to MySQL server on '172.18.0.9:3306' (111)) So the db container is, seemingly, up and responding to the ping on 3306 ... but not actually ready for connections (?!)
If I manually restart the Kodi containers after the system is up a minute or two, they then work fine.
— Reply to this email directly, view it on GitHub https://github.com/matthuisman/docker-kodi-headless/issues/57#issuecomment-2227188487, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPQAKM24QLQ26RPAH67AALZMH2XDAVCNFSM6AAAAABK2U2WQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRXGE4DQNBYG4 . You are receiving this because you commented.Message ID: @.***>
It's not, as far as I am aware, really, but apparently just long enough - here's the docker logs
2024-07-14 13:57:08+10:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.4.2+maria~ubu2404 started.
2024-07-14 13:57:09+10:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2024-07-14 13:57:09+10:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-07-14 13:57:09+10:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.4.2+maria~ubu2404 started.
2024-07-14 13:57:10+10:00 [Note] [Entrypoint]: MariaDB upgrade not required
2024-07-14 13:57:11 0 [Note] Starting MariaDB 11.4.2-MariaDB-ubu2404 source revision 3fca5ed772fb75e3e57c507edef2985f8eba5b12 as process 1
2024-07-14 13:57:11 0 [Note] InnoDB: Compressed tables use zlib 1.3
2024-07-14 13:57:11 0 [Note] InnoDB: Number of transaction pools: 1
2024-07-14 13:57:11 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-07-14 13:57:11 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2024-07-14 13:57:11 0 [Note] InnoDB: Initializing buffer pool, total size = 2.000GiB, chunk size = 1.000GiB
2024-07-14 13:57:11 0 [Note] InnoDB: Completed initialization of buffer pool
2024-07-14 13:57:11 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2024-07-14 13:57:11 0 [Note] InnoDB: Resetting space id's in the doublewrite buffer
2024-07-14 13:57:11 0 [Note] InnoDB: End of log at LSN=9901259632
2024-07-14 13:57:11 0 [Note] InnoDB: Opened 3 undo tablespaces
2024-07-14 13:57:11 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2024-07-14 13:57:11 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2024-07-14 13:57:11 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2024-07-14 13:57:11 0 [Note] InnoDB: log sequence number 9901259632; transaction id 5730132
2024-07-14 13:57:11 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-07-14 13:57:11 0 [Note] Plugin 'wsrep-provider' is disabled.
2024-07-14 13:57:11 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-07-14 13:57:13 0 [Note] InnoDB: Buffer pool(s) load completed at 240714 13:57:13
2024-07-14 13:57:19 0 [Note] Server socket created on IP: '0.0.0.0'.
2024-07-14 13:57:19 0 [Note] Server socket created on IP: '::'.
2024-07-14 13:57:19 0 [Warning] 'proxies_priv' entry '@% root@7380f97f3a17' ignored in --skip-name-resolve mode.
2024-07-14 13:57:19 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
2024-07-14 13:57:19 0 [Note] mariadbd: ready for connections.
Version: '11.4.2-MariaDB-ubu2404' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
I've only noticed this fairly recently (last couple months maybe?) - so perhaps it's a change to the mariadb startup that is the root cause?
Is a gross hack but all I can think to do here is a delayed restart of the containers after reboot:
# A couple minutes after reboot, restart the Kodi containers due to mariadb startup slowness?
@reboot sleep 120 && docker compose -f ~/docker/docker-compose.yml restart kodi-adults kodi-kids
Unfortunately, when I do container updates etc, I can run in to the same problem, so it's not really a solve as such but a band-aid for reboots at least..
You could created own docker containr extending from mine and just add a sleep before it starts Kodi?
On Sun, 14 Jul 2024, 16:33 Jeremy Daalder, @.***> wrote:
Is a gross hack but all I can think to do here is a delayed restart of the container after reboot:
A couple minutes after reboot, restart the Kodi containers due to mariadb startup slowness?
@reboot sleep 120 && docker compose -f ~/docker/docker-compose.yml restart kodi-adults kodi-kids
Unfortunately, when I do container updates etc, I can run in to the same problem, so it's not really a solve as such but a bandaid for reboots at least..
— Reply to this email directly, view it on GitHub https://github.com/matthuisman/docker-kodi-headless/issues/57#issuecomment-2227192264, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPQAKOWKSD6XDO4ZDEIAA3ZMH5QTAVCNFSM6AAAAABK2U2WQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRXGE4TEMRWGQ . You are receiving this because you commented.Message ID: @.***>
Unfortunately I'm (as yet?) ignorant as to how to actually do that - I've done a bit of reading just now, but it's not super obvious how one does it with a thing that is running as a service, as seems to be the case here - e.g. in other similar scenarios I see, one can overrule an entrypoint in a compose file (simpler than extending the image) - but there doesn't seem to be one here.
I've never extended a container and don't really know the implications - how does it work for updates etc? I guess if I need to do this I will learn :)
You'd think, though, given docker is a way of packaging services, it would be more helpful about this sort of thing, with some sort of mechanism in compose for a delay...and the whole thing is weird as the mariadb healthcheck explicitly says:
--connect
This is active when a external user can connect to the TCP port of MariaDB Server. This strictly tests just the TCP connection and not if any authentication works.
--innodb_initialized
This test is true when InnoDB has completed initializing. This includes any rollback or crash recovery that may be occurring in the background as MariaDB is starting.
The connecting user must have [USAGE](https://mariadb.com/kb/en/grant/#the-usage-privilege) privileges to perform this test.
So you'd really think that this healthcheck would cover it...
Ok I think I am making some progress here. Am I right in thinking I would add the sleep to:
I.e. essentially change that to something like:
#!/usr/bin/with-contenv bash
sleep 20 && s6-setuidgid abc /usr/bin/kodi
In practise this seems to just be changing that file (with the same path) and creating a Dockerfile
with just:
FROM matthuisman/kodi-headless:Omega
COPY root/ /
...that appears to build and run like yours.
Am I doing this right??
looks about right. i guess i could expose a sleep like as this an environment variable? then you could just use my stock one?
Well that would be awesome if you would - it seems to work ok in a quick test (i.e. kodi starts after 20 seconds rather than straight away).
If you're willing to add it, it's one less bit of extra complexity at my end, and I suspect my docker setup is not so strange/uncommon that others might ultimately run into a similar issue (I have, in one compose, nginx, mariadb, Kodi, a music server and a PiHole - pretty basic by modern standards!).
(I just donated but to be clear, that's just a general thank you for this & for support etc., not meant to imply this is conditional on that! Add if it suits you, if not I'll work things out at my end of course!).
Done :) latest image pushed with KODI_DELAY env variable that can be set
Done, and indeed this solves the issue - Kodi headless comes back up after a reboot connects to the db perfectly. Awesome.
Thank you!
I am having issues with system/container restarts - it seems the Kodi dockers I run (two) sometimes start before the DB is actually ready to connect.
In my compose I have a healthcheck on the db container (MariaDB)
And the Kodi containers depend on this:
...but that is not enough, it seems.
When this happens, I see this in the Kodi logs:
If I manually restart the containers a few minutes after e.g. a system reboot, then they work as expected and connect to the DB normally (and have been running these for some time now successfully, bar this issue that has appeared in recent months)
I'm struggling to work out a solution here, as docker isn't so great at this sort of thing as far as I can tell, I can't e.g. delay the startup of these, and the MariaDB container must be reporting healthy before it's really ready, I suppose, which allows the Kodi containers to then start.
Ideally there would be a healthcheck on the Kodi containers that actually tested this and restarted themselves if the DB is not actually ready to accept a connection...but I don't think there are tools within that container to check this (e.g. some manual way of checking that
db:3306
is connectable)Any ideas?