Closed sergeyshulga closed 2 years ago
do a git pull
and then try:
python3 x9k3.py -i https://slo.me/longb.ts
output should look like
seg0.ts start: 1.433000 duration: 5.339
seg1.ts start: 6.772000 duration: 2.369
seg2.ts start: 9.141000 duration: 2.969
seg3.ts start: 12.110000 duration: 5.339
seg4.ts start: 17.449000 duration: 5.339
seg5.ts start: 22.788000 duration: 5.338
seg6.ts start: 28.126000 duration: 5.339
seg7.ts start: 33.465000 duration: 5.339
seg8.ts start: 38.804000 duration: 5.338
seg9.ts start: 44.142000 duration: 5.339
seg10.ts start: 49.481000 duration: 5.339
seg11.ts start: 54.820000 duration: 5.338
seg12.ts start: 60.158000 duration: 5.339
seg13.ts start: 65.497000 duration: 5.339
seg14.ts start: 70.836000 duration: 5.338
seg15.ts start: 76.174000 duration: 5.339
seg16.ts start: 81.513000 duration: 5.339
seg17.ts start: 86.852000 duration: 5.338
seg18.ts start: 92.190000 duration: 5.339
seg19.ts start: 97.529000 duration: 5.339
seg20.ts start: 102.868000 duration: 5.338
seg21.ts start: 108.206000 duration: 2.770
seg22.ts start: 110.976000 duration: 2.569
Let me know if that works.
I found what I believe to be the issue.
Please upgrade threefive to 2.3.33 , I fixed a weird bug in the threefive.Stream class.
python3 -mpip install --upgrade threefive
pypy3 -mpip install --upgrade threefive
Let me know if this fixes it.
Talk to me Goose.....
Sorry for late reply. )) Updated threefive to 2.3.33. Tested with your TS (https://slo.me/longb.ts) - all working fine, segments and m3u created. On my TS with Mpeg2 video its still not create segments and playlist. Sorry, i missed that x9k3 support only H.264/H.265. After switch video to H.264 its working ok, creates segments and m3u, but have some messages in log like "pid: 0x3e9 last cc:13 cc:4". What that mean? Paket loss?
Also when i have UDP TS on x9k3 input, created segments not spliced correctly - it have mssing/dropped frames on the start/end of segments. If i record my UDP TS to file (.ts) and put it to x9k3 input, it splices correctly - no drops or blury frames between segments. Is possible to setup input buffer size for x9k3?
Thanks for getting back to me man.
Is the MPEG2 video codec valid for HLS? Apple's hls page says only 264 and 265, but the actual RFC does not specify the video codecs so I really don't know. x9k3 is currently only setup to handle h264 and h265 video codecs. For now, the MPEG2 video codec is not going to work correctly.
Sort of. The messages are continuity counter checks. you're seeing mpegts packet loss, but a few packets should not be enough to negatively impact the video quality as you describe.
Are you using multiple video streams?
Why are you using UDP? Is it a requirement? I use gumd for multicast and it works really well.
What software are you using stream the video via UDP?
I'm thinking I should have x9k3 use a larger receive buffer for the UDP socket connection like it does for multicast streams. I'll try that later today and let you know.
Don't worry man, we'll figure it out.
Adrian
Thank you for your time. Serg.
We have two problems
Mpeg2 video codec support.
UDP
I was able to reproduce the screwed-up video that you described, as well as the bad continuity counters. Updates to the reader function resolved both the playback and continuity counters.
I just push threefive v.2.3.35 with the fix.
python3 -mpip install --upgrade threefive
python3 -mpip install --upgrade threefive
let me know what happens.
Adrian
Let me know what happens.
You are the best! Unicast and multicast H264 are splitting now without issues. But Mpeg2 not working on my side - tested with udp or .ts file Example of Mp2 stream - https://www.dropbox.com/s/v5vrnlbqpqsc7xn/test2_mp2.ts?dl=0
Also - is possible to add two features?
Do a git pull
on x9k3, just to make sure were in sync
test2_mp2.ts works perfectly for me.
a@fumatica:~/scte35-hls-segmenter-x9k3$ pypy3 x9k3.py -i udp://192.168.1.154:5005
seg0.ts start: 162.917311 duration: 2.000
seg1.ts start: 164.917311 duration: 2.000
seg2.ts start: 166.917311 duration: 2.000
seg3.ts start: 168.917311 duration: 2.000
seg4.ts start: 170.917311 duration: 2.000
seg5.ts start: 172.917311 duration: 2.000
seg6.ts start: 174.917311 duration: 2.000
seg7.ts start: 176.917311 duration: 2.000
seg8.ts start: 178.917311 duration: 2.000
seg9.ts start: 180.917311 duration: 2.000
seg10.ts start: 182.917311 duration: 2.000
seg11.ts start: 184.917311 duration: 2.040
seg12.ts start: 186.957311 duration: 2.000
seg13.ts start: 188.957311 duration: 2.000
seg14.ts start: 190.957311 duration: 2.000
seg15.ts start: 192.957311 duration: 2.000
seg16.ts start: 194.957311 duration: 2.000
seg17.ts start: 196.957311 duration: 2.000
seg18.ts start: 198.957311 duration: 2.000
It's on my 'to do' list. I don't really know Windows, and I have been concerned about handling Windows file paths, but since you're here, you can help me test it. I'll get on it.
I hadn't considered this, but it makes a lot of sense, I'll add it.
Do a
git pull
on x9k3, just to make sure were in sync test2_mp2.ts works perfectly for me.
Strange. I used last version of x9k3, but it not splice mpeg2 TS.
git pull
x9k3$ pypy3
import x9k3 x9k3.version() '0.0.93'
Yes, its working now. Splices Mpeg2 as needed. Thank you.
That is super cool that the threads work on Windows.
Sorry about the "hey" everywhere, I was checking when a method was getting called,
Output dir option also workin fine ))
Output dir option also workin fine ))
Cool. I tried something new to handle Windows file paths and I was hoping it would work. I'm still thinking about how I want to handle deleting files for live stuff.
As variant - delete segments (if exist) with number < MEDIA_SLOTS, when create each new segment. Maybe code like that os.remove(f"seg{self.seg_num-MEDIA_SLOTS}.ts")
I think deleting would be best done externally. especially when the source is UDP, it's just too easy to drop packets. Deleting will have to interrupt x9k3 from reading from the socket, parsing the data does the same thing and it was dropping packets, and that's why I have it parse in a separate thread now.
x9k3 not dropping packets when i try to use some my junk code before "seg_file = f"seg{self.seg_num}.ts"": _del_file = f"seg{(self.seg_num-MEDIA_SLOTS)}.ts" del_uri = self.mk_uri(self.output_dir, del_file) try: os.remove(del_uri) except OSError as error: print("File can not be removed:"+deluri) But its duct tape for my case ))
Don't call your code junk, I like how you did that actually.
I was thinking about how to clear up segments left by previous runs, something like
import os
def del_segs():
with open('index.m3u8') as m3u8:
active = [line.strip() for line in m3u8.readlines() if line[0] not in ('#','\n')]
avail = [f for f in os.listdir() if f.endswith('.ts')]
[os.unlink(f) for f in avail if f not in active]
I think I need to make a segment class, make a new instance when I start a new segment, and put instances in X9K3.queue. that would make it easier and help me clean up the code a bit.
I'll get the delete in there for you, just give me a minute.
Have you tried the 0.0.97? I put it up yesterday.I replaced threading with multiprocess, that way it uses multiple cores. It works pretty well.
Hmm, build 97 not working. Tested on file and udp.
No big deal, I saw something about pickling and multiprocessing on Windows,
You see on the last line where it says cant pickle <function C9K3.<locals>.reader ......
it calls functools.partial, which I believe uses pickling to cache function args
def readr(): # 340425 ~64MB
if not self._find_start():
return
for chunk in iter(
# functools.partial call here
partial(self._tsdata.read, self._PACKET_SIZE * self._NUM_PKTS), b""
):
work_queue.put(chunk)
return
Give me a minute and I'll put up another version without pickling.
0.0.98 -> Back to threads, added --delete.
git pull and tell me how it goes man.
FYI,
I modeled the delete after how you did it.
I just used self.queue[0][1].rsplit(",")[1].strip()
instead of f"seg{(self.seg_num-MEDIA_SLOTS)}.ts"
so I could keep deleting the segment in sync with removing
the segment data from self.queue.
( I really need to rename self.queue because I import the queue module. )
def _write_manifest(self):
if self.live:
if len(self.queue) > MEDIA_SLOTS:
if self.delete_segs:
drop = self.mk_uri(
self.output_dir, self.queue[0][1].rsplit(",")[1].strip()
)
os.unlink(drop)
print(f"deleting {drop}")
self.queue = self.queue[1:]
self.start_seg_num = self.queue[0][0]
< snip >
Hello.. Tested b.98, its working ok with TS file, but not work with UDP.
C:\PyPy>pypy3.exe x9k3.py --live -delete -i udp://192.168.0.20:1234
Traceback (most recent call last):
File "x9k3.py", line 457, in
MSDN about that error: Message too long. A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram was smaller than the datagram itself.
Change _NUM_PKTS to 57425 and see if that works
https://github.com/futzu/x9k3/blob/main/x9k3.py#L57
_NUM_PKTS = 57425
Do me a favor and run this while x9k3 is running
tcpdump udp port <port_num>
replace
What you're looking for is the length
18:07:04.992922 IP 192.168.1.66.1046 > 239.255.255.250.8082: UDP, length 876 <--- length
18:07:04.992927 IP 192.168.1.65.1039 > 239.255.255.250.8082: UDP, length 1637
After changing _NUM_PKTS = 57425 its working. But what that value mean? Option --delete also looks working OK...
TCPDump show length 1316 bytes of each packet. 19:18:03.258022 IP Serg-ПК.51348 > ospf-all.mcast.net.1234: UDP, length 1316 19:18:03.261023 IP Serg-ПК.51348 > ospf-all.mcast.net.1234: UDP, length 1316 19:18:03.264023 IP Serg-ПК.51348 > ospf-all.mcast.net.1234: UDP, length 1316
NUM_PKTS is the number of mpegts packets x9k3 reads at a time. That's what I changed to get rid of your original problem. I was trying to lower it in 0.0.98 to reduce initial lag of segment creation,
If that is your stream is on port 1234, it looks like multicast, you may want to try it with udp://@ instead of udp:// in the uri.
Man the bitrate on that stream is almost twice what it should be, you should really consider running it through ffmpeg first to reduce the bitrate and then piping it to x9k3. Cut it down from 6000 to about 3500, and your hls playback with improve dramatically without a loss in quality.
I use high bitrate only for test, currently it ~3Mbps. No difference in x9k3 behavior. With _NUM_PKTS = 57425 its worked some time (not too long) and crash with error 10040. I try reduce NUM_PKTS, less value -> crash earlier with the same error 10040. Current workaround - not use NUM_PKTS value in chunk reading. I replaced string *partial(self._tsdata.read, self._PACKET_SIZE self._NUM_PKTS), b"" to partial(self._tsdata.read, self._PACKET_SIZE), b""** and its seems working fine.
But sometime have error and thread stops: C:\PyPy>pypy3.exe x9k3.py --live --delete -i udp://192.168.0.20:1234 seg120.ts start: 888.053322 duration: 5.300 Exception in thread Thread-1: Traceback (most recent call last): File "C:\PyPy\Lib\threading.py", line 973, in _bootstrap_inner self.run() File "C:\PyPy\Lib\threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "x9k3.py", line 383, in workr self._parse(item[i : i + self._PACKET_SIZE]) File "x9k3.py", line 365, in _parse self._mk_segment(pid) File "x9k3.py", line 268, in _mk_segment self._write_manifest() File "x9k3.py", line 240, in _write_manifest os.unlink(drop) OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: './CUE-O UT=YES\n#EXT-X-SCTE35:CUE="/DAgAAAAAAAAAP/wDwUAAAAAf/9+AAAAAAAAAAAAAD4xJP8="'
C:\PyPy>pypy3.exe x9k3.py --live --delete -i udp://192.168.0.20:1234 Splice Insert Splice Point Splice Insert@159.429322 seg44.ts start: 159.193322 duration: 0.244 deleting ./seg34.ts seg45.ts start: 159.437322 duration: 2.576 deleting ./seg35.ts seg46.ts start: 162.013322 duration: 2.280 deleting ./seg36.ts seg47.ts start: 164.293322 duration: 2.360 deleting ./seg37.ts seg48.ts start: 166.653322 duration: 2.160 deleting ./seg38.ts Splice Insert seg49.ts start: 168.813322 duration: 2.240 deleting ./seg39.ts seg50.ts start: 171.053322 duration: 2.060 Exception in thread Thread-1: Traceback (most recent call last): File "C:\PyPy\Lib\threading.py", line 973, in _bootstrap_inner self.run() File "C:\PyPy\Lib\threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "x9k3.py", line 383, in workr self._parse(item[i : i + self._PACKET_SIZE]) File "x9k3.py", line 365, in _parse self._mk_segment(pid) File "x9k3.py", line 268, in _mk_segment self._write_manifest() File "x9k3.py", line 240, in _write_manifest os.unlink(drop) OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: './CUE-OUT=YES\n#E XTINF:0.92'
C:\PyPy>pypy3.exe x9k3.py --live --delete -i udp://192.168.0.20:1234 seg43.ts start: 157.053322 duration: 2.140 deleting ./seg33.ts Splice Insert Splice Point Splice Insert@159.429322 seg44.ts start: 159.193322 duration: 0.244 deleting ./seg34.ts seg45.ts start: 159.437322 duration: 2.576 deleting ./seg35.ts seg46.ts start: 162.013322 duration: 2.280 deleting ./seg36.ts seg47.ts start: 164.293322 duration: 2.360 deleting ./seg37.ts seg48.ts start: 166.653322 duration: 2.160 deleting ./seg38.ts Splice Insert seg49.ts start: 168.813322 duration: 2.240 deleting ./seg39.ts seg50.ts start: 171.053322 duration: 2.060 Exception in thread Thread-1: Traceback (most recent call last): File "C:\PyPy\Lib\threading.py", line 973, in _bootstrap_inner self.run() File "C:\PyPy\Lib\threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "x9k3.py", line 383, in workr self._parse(item[i : i + self._PACKET_SIZE]) File "x9k3.py", line 365, in _parse self._mk_segment(pid) File "x9k3.py", line 268, in _mk_segment self._write_manifest() File "x9k3.py", line 240, in _write_manifest os.unlink(drop)The filename, directory name, or volume label syntax is incorrect: './CUE-OUT=YES\n#E XTINF:0.92'
This will make it easier.
get v.0.0.99 and use ffmpeg to get the stream
# replace input.ts with your input stream
ffmpeg -re -copyts -i input.ts -map 0 -c copy -f mpegts - | python3 x9k3.py --delete
--delete implies --live
--output-dir will now create the directory if it does not exist.
You can use anything, curl, multicat, ffmpeg, tsduck, whatever you like and pipe it into x9k3. --live and --delete and __--output_dir__ all work
This issue is resolved.
Hello. Sorry for delay with answers. I'm not use file for test, always use UDP live stream with --live option. No difference unicast or multicast in x9k3 behavior. I tested v.0.0.99, and have some issues - dont know is it related to Windows or anything else... Sometime its crashes - but i need more testing to understand when it happened. Some time its skips SCTE35 messages or not correctly write to manifest(but when i catch UDP stream it with TSDuck - all messages seen). Example - i have live stream with clips and only 2 messages - CUE-OUT and CUE-IN immediatly. In firrst time its writtent to m3u Ok, but second occurence x9k3 not correctly write CUE-OUT command to m3u (Command totally the same as first entry). Sorry if i too annoying ))
seg1.ts
seg2.ts
seg3.ts
seg4.ts
seg5.ts
seg6.ts
seg7.ts
seg8.ts
seg9.ts
seg10.ts
seg11.ts
seg12.ts
Hello. I try to use x9k3 segmenter but have no success.
First problem - its not create .m3u manifest file. OS enviroment - Win7. I try to use Pypy3 or Python3, with admin rights - no success. Any suggestion? Also its possible to select target directory for m3u and ts segments?
Second - it crash after some time with the same error.
Thank you.