Request for new catchup placeholder and new catchup type

[Mr-Groch]

Hi, there is popular placeholder ${timestamp} used by many IPTV providers (this is standart placeholder supported in Perfect Player, OTT Navigator, etc). It means the same as our {lutc}.

Example m3u list fragment:

catchup=“append” catchup-source="?utc=${start}&lutc=${timestamp}"

Second request is to support catchup=“shift” type without need for catchup-source. It is simple to implement. “Shift” type means the same as append with ?utc={utc}&lutc={lutc} or &utc={utc}&lutc={lutc} (if channel URL contains ‘?’ sign, then add suffix &utc={utc}&lutc={lutc}, if not - then add ?utc={utc}&lutc={lutc})

Those 2 requests was implemented in Archive Client in CoreELEC, so it has been tested in production :)

[phunkyfish]

Timestamp is easy.

For shift you will need to explain further. In this case where does the query string to append come from? Or does it default to the values you have given above?

[Mr-Groch]

This shift type comes from SIPTV and because SIPTV is still very popular, many IPTV providers have adopted theirs backends to this standard. Query string to append for shift is ?utc={utc}&lutc={lutc} or &utc={utc}&lutc={lutc} (if channel URL contains ‘?’ sign, then add suffix &utc={utc}&lutc={lutc}, if not - then add ?utc={utc}&lutc={lutc})

Moreover that append string should be used also when there is no catchup tag in m3u list, but there is only timeshift tag (original historical tag introduced by SIPTV - https://siptv.eu/howto/ - at bottom)

Example:

#EXTINF:0 timeshift="5" tvg-id="13Ulica.pl" tvg-logo="http://logo.tv/logo/13%20Ulica%20HD.png" country="pl" audio-track="pl" group-title="Rozrywka" ,13 Ulica HD
http://list.tv:8888/325/mono.m3u8?token=secret

Catchup query string should be: http://list.tv:8888/325/mono.m3u8?token=secret&utc={utc}&lutc={lutc}

Another example:

#EXTINF:0 id="3_468" group-title="Popularne" tvg-id="TVP1.pl" tvg-name="TVP 1 HD" tvg-logo="http://logo.tv/assets/logo/4019.png" catchup="shift" catchup-days="3",TVP 1 HD
https://list.tv/play/hls-custom-c468-secret.m3u8

Catchup query string should be: https://list.tv/play/hls-custom-c468-secret.m3u8?utc={utc}&lutc={lutc}

[phunkyfish]

Ok, if there is a timeshift=“x” or catchup=“shift” then use that format.

[Mr-Groch]

Exactly :) But IMHO if there is timeshift=“x” and catchup=“something” (not shift) or catchup-source="something" then should be used query format described in catchup tags. I have seen lists with both tags. Theoretically this should be inconsistent, but practically there is no ISO for this :)

[phunkyfish]

I agree, catchup tag gets preference. If there is no catchup tag then a timeshift tag.

However a catchup-sourcetag would still be ignored without a catchup tag. I have seen examples where just a catchup-source tag is in the entry where both default and append could be assumed.

[SerhiyRomanov]

I also find out next examples

http://fr01.spr24.net/364/timeshift_abs-1350274200.ts?token=key
http://fr01.spr24.net/364/archive-1350274200-4200.m3u8?token=key

This URL means that video should play 4200 second, starting from 1350274200 second, as a file accessed by HLS. 1350274200 is a UNIX time in the UTC time zone.

It's kinda weird but is there any idea how to deal with it?

[Mr-Groch]

This is flussonic standard - more complicated, but I know it. Simplest approuch for catchup="flussonic" (or "fs") is to use regex match to explode url into fields, for example this is channel live url:

http://list.tv:8888/325/mono.m3u8?token=secret

Regex with group match: ^(http[s]?://[^/]+)/([^/]+)/([^/]+)\.m3u8(\?.+=.+)?$

group1: host group2: channel_id group3: list_type (can be mono, video or index) group4: url_append (credentials or token)

Then we can build catchup url from this like this (yes, we can use list_type.m3u8 instead of archive.m3u8):

catchup-source="'.$group1.'/'.$group2.'/'.$group3.'-${start}-14400.m3u8'.$group4.'"'

As you can see I use const 14400 (s) as duration, but it works ;)

I have made php online converter script for flussonic backends lists to use in this addon (and for Archive Client for CoreELEV also):

https://gist.github.com/Mr-Groch/5b7eb468e7c31ba7e5b3fcc37c9e1743 (i'm also asking flussonic server api if catchup exists for each channel and then I'm adding catchup and catchup-source tags to list)

But if catchup="flussonic" (or "fs") could be supported by this addon internally it would be nice :)

PS. There is also catchup="xc" standard ;) And like for fussonic, I have made php online converter:

https://gist.github.com/Mr-Groch/64ba1696462a4537d7c3e38ee9d280ec

[phunkyfish]

Ok, I can add support for these after the first two. Assuming you can test any updates?

[SerhiyRomanov]

@Mr-Groch Thank you for pointing this out.

@phunkyfish Yes, I can test it. I even can provide a playlist with such types of catchup (flussonic) for you for testing during development.

What do you think about to have common catchup types in settings? I mean, to add an option like "Catchup type" with values "default", "append", "flussonic", "xc", shift", ..., "Custom format string". So, users will be able to choose one from common types OR to provide their custom format string.

[phunkyfish]

You mean assume by default all channels are of a certain type if nothing is in the M3U?

Maybe explain what you mean a bit more. And maybe in a new issue 😉

[Mr-Groch]

I can also test all of mentioned here types: shift (also with just timeshift tag), flussonic (fs) and xc.

[phunkyfish]

@Mr-Groch could you breakdown and explain the xc standard similar to how you explained flussonic?

[Mr-Groch]

catchup="xc" is more complicated than flussonic. First - live url format is different for HLS and TS lists, but fortunately it can be detected easily - HLS URL ends with .m3u[8] and contains /live/ string.

This is universal regex with group match for XC (notice non-capturing group between group1 and group2): ^(http[s]?://[^/]+)/(?:live/)?([^/]+)/([^/]+)/([^/\.]+)(\.m3u[8]?)?$

group1: host group2: username group3: password group4: channel_id group5: extension (only for HLS lists)

Then we can build catchup url from this like this:

unlike live URL, catchup URL must always end with extension $extension = ($group5 ? $group5 : '.ts') catchup-source="'.$group1.'/timeshift/'.$group2.'/'.$group3.'/240/{Y}-{m}-{d}:{H}-{M}/'.$group4.$extension.'"'

/240/ in above URL means minutes to play from provided datetime.

Example of TS URL: http://list.tv:8080/my@account.xc/my_password/1477

Catchup query URL: http://list.tv:8080/timeshift/my@account.xc/my_password/240/2020-02-15:18-30/1477.ts

Example of HLS URL: http://list.tv:8080/live/my@account.xc/my_password/1477.m3u8

Catchup query URL: http://list.tv:8080/timeshift/my@account.xc/my_password/240/2020-02-15:18-30/1477.m3u8

[Sorien]

it looks like it would be easier to have same scriptable extension point than implement all these custom providers and combinations in core

[SerhiyRomanov]

@phunkyfish Seems helpful https://flussonic.com/doc/digital-video-recording-dvr/accessing-dvr-archives-via-various-protocols

[Mr-Groch]

@SerhiyRomanov Flussonic API have many possibilities (XC also), but I think better is to keep it as simple as possible and use one universal archive query string that works and it is simple to build from channel URL.

[phunkyfish]

@Mr-Groch thanks

Does this mean that xc format does not allow you specify seconds, only minutes?

[phunkyfish]

it looks like it would be easier to have same scriptable extension point than implement all these custom providers and combinations in core

Please expand. How do you mean in core?

[Mr-Groch]

Does this mean that xc format does not allow you specify seconds, only minutes?

Unfortunately yes, it allows streaming only from full minutes according to my knowledge based on studying XC API docs when it was available (currently xtream codes website is down, so documentation is hard to find)

EDIT: There is also another/alternative URL format for live and catchup from old XC API version 1, but I haven't seen any provider using it last year, so IMHO we can skip it... It looks like: http://iptvserver.url:port/streaming/timeshift.php?username=xxxx&password=yyyy&stream=streamnumber&start=2018-7-19:23-00&duration=60

[phunkyfish]

Unfortunately yes, it allows streaming only from full minutes according to my knowledge based on studying XC API docs when it was available (currently xtream codes website is down, so documentation is hard to find)

Ok, no worries.

[phunkyfish]

@Mr-Groch what is the /240/ in the xtream codes catchup url?

EDIT: NVM it’s already in your post.

[phunkyfish]

Ok, please test:

iptv simple: https://jenkins.kodi.tv/blue/organizations/jenkins/kodi-pvr%2Fpvr.iptvsimple/detail/PR-327/3/artifacts ffmpeg direct: https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Finputstream.ffmpegdirect/detail/PR-14/7/artifacts

You need both addons. Has ${timestamp}, shift, fs/flussonic and xc.

[Mr-Groch]

I have only Linux(Debian) x86_64 or Linux(Debian) on armv7. Can you point to repo with this changes? I can clone and compile myself and then I can test it

[phunkyfish]

Sure, here are the PRs:

https://github.com/kodi-pvr/pvr.iptvsimple/pull/327

https://github.com/xbmc/inputstream.ffmpegdirect/pull/14

[Mr-Groch]

I have tested so far:

1. ${timestamp} placeholder (with catchup=“append” catchup-source="?utc=${start}&lutc=${timestamp}")
2. catchup="shift"
3. timeshift="x" without any other catchup tags
4. catchup="xc" with HLS playlists
5. catchup="xc" with TS playlists

Results:

1. OK, catchup from EPG plays correctly, seeking backward/forward works OK when playing from catchup
2. OK, catchup from EPG plays correctly, seeking backward/forward works OK when playing from catchup
3. OK, catchup from EPG plays correctly, seeking backward/forward works OK when playing from catchup
4. Partially OK, catchup from EPG plays correctly, seeking backward/forward works OK when playing from catchup, issues found:
   • {duration} for XC catchup query url should be in minutes, plugin inserts duration in seconds
5. Partially OK, catchup from EPG plays correctly, seeking backward/forward works OK when playing from catchup, issues found:
   • {duration} for XC catchup query url should be in minutes, plugin was inserting duration in seconds
   • displayed duration in KODI UI was invalid (for example - programme duration should be 5 minutes, KODI player showed more than 1 hour)
   • I was unable to seek backward in live - KODI player looked like timeshift was available, but when I try to seek backward, it allowed only to jump to time when I started watching live tv

I was also unable to seek backwards when playing live tv in other cases (HLS playlists), but in that cases KODI player allowed to do so. I have not tested that before this change, so maybe that is an old issue... I wanted to jump 3 minutes backwards, KODI UI looked like timeshift was succesfully - even playing time was incresing, but video and audio were freezed... Logs from this situation: https://pastebin.com/cYnNWqCu

cathup="flussonic" (or "fs") I will be able to test this weekend, because I need to refresh my subscription for that list.

[phunkyfish]

Ah, ok. I didn’t realise duration had to be minutes. I can add that.

Can you explain further on trying to seek backward in live? Did you enable catchup for all channels?

[Mr-Groch]

XC duration in minutes, Flussonic duration is seconds :)

Catchup for all channels in plugin config was disabled. Not all channels in my lists have archive.

Steps to reproduce for TS list:

1. Play any channel in live
2. Wait about 10 seconds
3. Press 4x left arrow key fast to seek backward 3 minutes and observe KODI UI - label under progress bar is showing only -10 second timeshift possible to rewind
4. After short time KODI succesfully jumps 10 seconds backwards and plays channel from the moment you have started watching

Steps to reproduce for HLS list:

1. Play any channel in live
2. Press 4x left arrow key fast to seek backward 3 minutes and observe KODI UI - label under progress bar is correctly updated after each key press and shows -3 minutes in the end
3. After short time KODI tries to jump 3 minutes backwars, video and audio freezes, but current position at progress bar moves 3 minutes backwards and label above progress bar with timeshifted hour is increasing as if video were playing

[phunkyfish]

Ok, your issues may be because of duration.

Both PRs updated. Please rebuild and try again.

[Mr-Groch]

I forgot about flussonic mpegts playlist... I think that we need another regex, because flussonic mpegts lists catchup query url is different... So if catchup="flussonic" (or "fs") then check channel URL if it is matching HLS regex, if not then check TS regex.

http://list.tv:8888/325/mpegts?token=secret

Regex with group match: ^(http[s]?://[^/]+)/([^/]+)/mpegts(\?.+=.+)?$

This regex has not been tested by me...

group1: host group2: channel_id group3: optional url_append (credentials or token)

catchup-source="'.$group1.'/'.$group2.'/timeshift_abs-${start}.ts'.$group3.'"'

[phunkyfish]

Spotted and added one on the latest PR. I actually used the same regex for both:

  static std::regex fsRegex("^(http[s]?://[^/]+)/([^/]+)/([^/\\?]+)(?:\\.m3u8)?(\\?.+=.+)?$");
  std::smatch matches;

  if (std::regex_match(m_streamURL, matches, fsRegex))
  {
    if (matches.size() == 5)
    {
      std::string fsHost = matches[1].str();
      std::string fsChannelId = matches[2].str();
      std::string fsListType = matches[3].str();
      std::string fsUrlAppend = matches[4].str();

      if (fsListType == "mpegts")
        m_catchupSource = fsHost + "/" + fsChannelId + "/" + "timeshift_abs-${start}.ts" + fsUrlAppend;
      else
        m_catchupSource = fsHost + "/" + fsChannelId + "/" + fsListType + "-${start}-{duration}.m3u8" + fsUrlAppend;

      return true;
    }
  }

Also, to enable timeshifting for TS streams is still a todo. Right now that will only work for M3U8 streams.

[phunkyfish]

Interesting is that we need a way to identify TS streams so it's possible to enable timeshift. Handily both fs/xc streams give us that. TS is the trickiest format as detecting it is hard.

Users can get around this by using a kodiprop to set the mimetype correctly if it's something we can't easily figure out.

I think once we are happy with this PR, i.e. timeshift working for M3U8 and plain catchup for TS streams that will be the next feature.

[Mr-Groch]

I have tested once again catchup="xc" on updated PRs (I have pulled and compiled about one hour ago).

I didn't pay big atention to TS playlist this time - just checked duration from catchup. Duration is OK this time - in both, catchup url and KODI player UI.

All OK with HLS playlist

[phunkyfish]

Excellent, thanks for testing. @SerhiyRomanov will test the flussonic streams and the all channels mode tomorrow.

Will let you know once I have figured out how to enable TS streams.

[phunkyfish]

@Mr-Groch added a new test commit for TS streams. Let me know if it works.

[Mr-Groch]

Checked XC TS playlist with newest commits - I see no difference - still timeshift from live dosn't work. Catchup from EPG works as earlier - OK.

Checked also XC HLS for regression - it's OK.

I have also noticed new option in plugin configuration - "All channel support catchup using mode" - but when I removed from m3u list all catchup tags, set this option to "Xtream Codes" - there is no catchup indicator in EPG, timeshift also is disabled.

[phunkyfish]

Ya, the new setting is not working, you’re correct. Will fix shortly.

I will need a test TS fs or xc stream to add TS support.

[Mr-Groch]

@phunkyfish there are no private messages on github - give me your mail, I will send URL

[phunkyfish]

Ok, new setting should be working now.

Will look at the TS streams tomorrow. Thanks for the test stream.

[phunkyfish]

Added in #327