search

iptv-org / epg

Utilities for downloading the EPG (Electronic Program Guide) for thousands of TV channels from hundreds of sources.
https://iptv-org.github.io/
The Unlicense
1.75k stars 189 forks source link

EPG not updating using cron command #2190

Closed fdotcico closed 10 months ago

fdotcico commented 10 months ago

Describe your issue

Hi guys, I came across this project and want to start using it to implement epg in my Plex + xteve iptv.

My goal is to run this project in a container on my NAS through Portainer, so that I can serve the epg xml in xteve.

I was able to get it working, but to enable the update I am forced to restart the container each time. This is because even if I use the string --cron"@hourly" the container tries to update the epg list for only the two-day window in which I started the container.

I would like this update to be rolling and then download the new guide every day.

I am currently using this:

Dockerfile

FROM node:current-alpine3.17

RUN apk update && \
    apk add --no-cache bash git && \
    adduser -D -s /bin/bash epg_grabber

RUN mkdir /opt/epg_grabber && \
    chown epg_grabber /opt/epg_grabber

USER epg_grabber
WORKDIR /opt/epg_grabber/

RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber/epg && \
    cd /opt/epg_grabber/epg && \
    npm install

EXPOSE 3000

start.sh

#!/bin/bash

npm --prefix /opt/epg_grabber/epg run grab -- --channels=custom_channels.xml --cron="5 6,9,12,15,18,21,0 * * *" &
npm --prefix /opt/epg_grabber/epg run serve

docker-compose

version: '3'
services:
  epg_grabber:
    container_name: EPG-Grabber
    image: epg_grabber:latest
    environment:
      - TZ=Europe/Rome
    volumes:
      - /volume1/..../epg/start.sh:/opt/epg_grabber/start.sh
      - /volume1/..../epg/custom_channels.xml:/opt/epg_grabber/epg/custom_channels.xml
    command: bash -c "/opt/epg_grabber/start.sh"
    ports:
      - "3000:3000"

Thanks!

freearhey commented 10 months ago

I think the problem here:

#!/bin/bash

npm --prefix /opt/epg_grabber/epg run grab -- --channels=custom_channels.xml --cron="5 6,9,12,15,18,21,0 * * *" &
npm --prefix /opt/epg_grabber/epg run serve

These commands must be run in parallel. In your case, the guide is most likely generated, but the server is not running.

fdotcico commented 10 months ago

Looking at the container log, the server continues to be active but as you can see it queries the different services "tvguide.myjcom.jp" and "guidatv.sky.co.uk" for the dates October 21 and 22, while we are on October 23.

Log abstract

run #13:
  [32/104] tvguide.myjcom.jp (ja) - JOEXDTV.jp - Oct 21, 2023 (42 programs)
  [33/104] tvguide.myjcom.jp (ja) - JOCXDTV.jp - Oct 22, 2023 (50 programs)
  [34/104] tvguide.myjcom.jp (ja) - JOCXDTV.jp - Oct 21, 2023 (41 programs)
  [35/104] tvguide.myjcom.jp (ja) - JOAXDTV.jp - Oct 22, 2023 (46 programs)
  [36/104] tvguide.myjcom.jp (ja) - JOAXDTV.jp - Oct 21, 2023 (37 programs)
  [37/104] tvguide.myjcom.jp (ja) - TBSNews.jp - Oct 22, 2023 (7 programs)
  [38/104] tvguide.myjcom.jp (ja) - TBSNews.jp - Oct 21, 2023 (7 programs)
  [39/104] tvguide.myjcom.jp (ja) - NHKGeneralTV.jp - Oct 22, 2023 (69 programs)
  [40/104] tvguide.myjcom.jp (ja) - NHKGeneralTV.jp - Oct 21, 2023 (75 programs)
  [41/104] tvguide.myjcom.jp (ja) - NHKEducationalTV.jp - Oct 22, 2023 (45 programs)
  [42/104] tvguide.myjcom.jp (ja) - NHKEducationalTV.jp - Oct 21, 2023 (62 programs)
  [43/104] tvguide.myjcom.jp (ja) - NHKBSPremium.jp - Oct 22, 2023 (39 programs)
  [44/104] tvguide.myjcom.jp (ja) - NHKBSPremium.jp - Oct 21, 2023 (46 programs)
  [45/104] tvguide.myjcom.jp (ja) - NHKBS1.jp - Oct 22, 2023 (44 programs)
  [46/104] tvguide.myjcom.jp (ja) - NHKBS1.jp - Oct 21, 2023 (33 programs)
  [47/104] tvguide.myjcom.jp (ja) - BSTBS.jp - Oct 22, 2023 (37 programs)
  [48/104] tvguide.myjcom.jp (ja) - BSTBS.jp - Oct 21, 2023 (38 programs)
  [49/104] tvguide.myjcom.jp (ja) - BSTVTokyo.jp - Oct 22, 2023 (45 programs)
  [50/104] tvguide.myjcom.jp (ja) - BSTVTokyo.jp - Oct 21, 2023 (47 programs)
  [51/104] tvguide.myjcom.jp (ja) - BSNipponTV.jp - Oct 22, 2023 (48 programs)
  [59/104] guidatv.sky.it (it) - MediasetExtra.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [60/104] guidatv.sky.it (it) - MediasetExtra.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [61/104] guidatv.sky.it (it) - SkyTG24.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [62/104] guidatv.sky.it (it) - SkyTG24.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [63/104] guidatv.sky.it (it) - Rete4.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [64/104] guidatv.sky.it (it) - Rete4.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [65/104] guidatv.sky.it (it) - RaiStoria.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [66/104] guidatv.sky.it (it) - RaiStoria.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [67/104] guidatv.sky.it (it) - RaiSport.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [68/104] guidatv.sky.it (it) - RaiSport.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [69/104] guidatv.sky.it (it) - RaiPremium.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [70/104] guidatv.sky.it (it) - RaiPremium.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [71/104] guidatv.sky.it (it) - RaiNews24.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [72/104] guidatv.sky.it (it) - RaiNews24.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [73/104] guidatv.sky.it (it) - RaiMovie.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [74/104] guidatv.sky.it (it) - RaiMovie.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [75/104] guidatv.sky.it (it) - RaiGulp.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [76/104] guidatv.sky.it (it) - RaiGulp.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [92/104] guidatv.sky.it (it) - La7.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [93/104] guidatv.sky.it (it) - La5.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [94/104] guidatv.sky.it (it) - La5.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [95/104] guidatv.sky.it (it) - Italia2.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [96/104] guidatv.sky.it (it) - Italia2.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [97/104] guidatv.sky.it (it) - Italia1.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [98/104] guidatv.sky.it (it) - Italia1.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [99/104] guidatv.sky.it (it) - Iris.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [100/104] guidatv.sky.it (it) - Iris.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [101/104] guidatv.sky.it (it) - CieloTV.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [102/104] guidatv.sky.it (it) - CieloTV.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  [103/104] guidatv.sky.it (it) - Canale5.it - Oct 22, 2023 (0 programs)
    ERR: Request failed with status code 422
  [104/104] guidatv.sky.it (it) - Canale5.it - Oct 21, 2023 (0 programs)
    ERR: Request failed with status code 422
  saving to "guide.xml"...
  done in 00h 01m 26s
 HTTP  10/23/2023 3:10:49 PM 192.168.1.150 GET /guide.xml **#XTEVE Scheduled XML update**
 HTTP  10/23/2023 3:10:49 PM 192.168.1.150 Returned 200 in 5 ms **#XTEVE Scheduled XML update**
freearhey commented 10 months ago

I think I figured out what the problem might be. I'll try to fix it as soon as I can.

freearhey commented 10 months ago

Fixed in https://github.com/iptv-org/epg/commit/7237ba0f1ee4ee0eeb276e40504d177ff4a014a0

MatthK commented 10 months ago

@fdotcico

I was looking for a Docker image and tried to build one based on your example given. However, when I try to build it, I get the following error message.

Dockerfile

FROM node:current-alpine3.17

RUN apk update && \
    apk add --no-cache bash git && \
    adduser -D -s /bin/bash epg_grabber

RUN mkdir /opt/epg_grabber && \
    chown epg_grabber /opt/epg_grabber

USER epg_grabber
WORKDIR /opt/epg_grabber/

RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber
RUN cd /opt/epg_grabber
RUN npm install

EXPOSE 3000

RUN echo "50 6,9,12,15,18,21,0 * * * epg_grabber npm --prefix /opt/epg_grabber/epg run grab -- --channels=custom_channels.xml" >> /etc/crontab
ENTRYPOINT /bin/bash -c "cron && npm --prefix /opt/epg_grabber/epg run serve"

Error message after using docker build .

matth@vs03:~/epg$ docker build .
[+] Building 6.4s (10/11)                                                                                                                                             docker:default
 => [internal] load build definition from Dockerfile                                                                                                                            0.0s
 => => transferring dockerfile: 645B                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/node:current-alpine3.17                                                                                                      1.5s
 => [1/8] FROM docker.io/library/node:current-alpine3.17@sha256:c8e4f0ad53631bbf60449e394a33c5b8b091343a8702bd03615c7c13ae4c445d                                                0.0s
 => CACHED [2/8] RUN apk update &&     apk add --no-cache bash git &&     adduser -D -s /bin/bash epg_grabber                                                                   0.0s
 => CACHED [3/8] RUN mkdir /opt/epg_grabber &&     chown epg_grabber /opt/epg_grabber                                                                                           0.0s
 => CACHED [4/8] WORKDIR /opt/epg_grabber/                                                                                                                                      0.0s
 => CACHED [5/8] RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber                                                                         0.0s
 => CACHED [6/8] RUN cd /opt/epg_grabber                                                                                                                                        0.0s
 => ERROR [7/8] RUN npm install                                                                                                                                                 4.8s
------
 > [7/8] RUN npm install:
4.563 npm notice
4.563 npm notice New patch version of npm available! 10.2.0 -> 10.2.1
4.563 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.2.1>
4.563 npm notice Run `npm install -g npm@10.2.1` to update!
4.563 npm notice
4.564 npm ERR! code 126
4.564 npm ERR! path /opt/epg_grabber/node_modules/libxmljs2
4.565 npm ERR! command failed
4.565 npm ERR! command sh -c node-pre-gyp install --fallback-to-build --loglevel http
4.566 npm ERR! sh: node-pre-gyp: Text file busy
4.567
4.567 npm ERR! A complete log of this run can be found in: /home/epg_grabber/.npm/_logs/2023-10-27T09_19_31_227Z-debug-0.log
------
Dockerfile:15
--------------------
  13 |     RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber
  14 |     RUN cd /opt/epg_grabber
  15 | >>> RUN npm install
  16 |
  17 |     EXPOSE 3000
--------------------
ERROR: failed to solve: process "/bin/sh -c npm install" did not complete successfully: exit code: 126

Did you not get this error? Any idea on how to get over that?

fdotcico commented 10 months ago

Major difference is here 

RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber/epg && \
    cd /opt/epg_grabber/epg && \
    npm install

Try this 

FROM node:current-alpine3.17

RUN apk update && \
    apk add --no-cache bash git && \
    adduser -D -s /bin/bash epg_grabber

RUN mkdir /opt/epg_grabber && \
    chown epg_grabber /opt/epg_grabber

USER epg_grabber
WORKDIR /opt/epg_grabber/

RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber/epg && \
    cd /opt/epg_grabber/epg && \
    npm install

EXPOSE 3000

RUN echo "50 6,9,12,15,18,21,0 * * * epg_grabber npm --prefix /opt/epg_grabber/epg run grab -- --channels=custom_channels.xml" >> /etc/crontab
ENTRYPOINT /bin/bash -c "cron && npm --prefix /opt/epg_grabber/epg run serve"
MatthK commented 10 months ago

Tried with exactly the Dockerfile you stated and got the same error:

matth@vs03:~/epg$ docker build .
[+] Building 9.3s (8/9)                                                                                                                                               docker:default
 => [internal] load build definition from Dockerfile                                                                                                                            0.0s
 => => transferring dockerfile: 663B                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/node:current-alpine3.17                                                                                                      0.9s
 => [1/6] FROM docker.io/library/node:current-alpine3.17@sha256:c8e4f0ad53631bbf60449e394a33c5b8b091343a8702bd03615c7c13ae4c445d                                                0.0s
 => CACHED [2/6] RUN apk update &&     apk add --no-cache bash git &&     adduser -D -s /bin/bash epg_grabber                                                                   0.0s
 => CACHED [3/6] RUN mkdir /opt/epg_grabber &&     chown epg_grabber /opt/epg_grabber                                                                                           0.0s
 => CACHED [4/6] WORKDIR /opt/epg_grabber/                                                                                                                                      0.0s
 => ERROR [5/6] RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber/epg &&     cd /opt/epg_grabber/epg &&     npm install                    8.3s
------
 > [5/6] RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber/epg &&     cd /opt/epg_grabber/epg &&     npm install:
0.309 Cloning into '/opt/epg_grabber/epg'...
7.989 npm notice
7.989 npm notice New patch version of npm available! 10.2.0 -> 10.2.1
7.989 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.2.1>
7.989 npm notice Run `npm install -g npm@10.2.1` to update!
7.989 npm notice
7.990 npm ERR! code 126
7.990 npm ERR! path /opt/epg_grabber/epg/node_modules/libxmljs2
7.991 npm ERR! command failed
7.991 npm ERR! command sh -c node-pre-gyp install --fallback-to-build --loglevel http
7.991 npm ERR! sh: node-pre-gyp: Text file busy
7.992
7.992 npm ERR! A complete log of this run can be found in: /home/epg_grabber/.npm/_logs/2023-10-27T09_53_42_865Z-debug-0.log
------
Dockerfile:13
--------------------
  12 |
  13 | >>> RUN git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber/epg && \
  14 | >>>     cd /opt/epg_grabber/epg && \
  15 | >>>     npm install
  16 |
--------------------
ERROR: failed to solve: process "/bin/sh -c git clone --depth 1 -b master https://github.com/iptv-org/epg.git /opt/epg_grabber/epg &&     cd /opt/epg_grabber/epg &&     npm install" did not complete successfully: exit code: 126
seang96 commented 9 months ago

@MatthK try using

FROM node:20-alpine

What is happening is the libxmljs2 release for nodejs 21.1 does not exist, so it trys to build it from the source code, but your DockerFile does not include the required packages for node-pre-gyp to run and I believe it also doesn't run on 21.1.