Possibility to modify project to be used as a video capture source?

[jibsaramnim]

Hello!

I stumbled upon your project as I have a device that uses the same IT9910HD drivers. I don't have experience with Rust, but I was able to modify and build your project to get its HDMI input to work with my particular device (it has a different USB product identifier). I wasn't able to get other sources to work, I'm guessing that might be due to a difference in mapping or so, but I was wondering if there would be any way to have your fantastic work be adopted so that the device can be used as an input device, as supposed to current method of a fuse-mounted path that has a ts "file" in there. That way it could even support being used with tools like OBS.

Thank you kindly!

[marek-g]

Hi Dave!

It's great to hear that you were able to run my code! I think that you may be now the only user of this code on the planet (or at least the only one who contacted with me :)) as my device burned down in an unfortunate electronic experiment :/ So I cannot continue developing the code for it. I only enjoyed it for two weeks...

I was only using the first HDMI input and nothing else, so my code for changing inputs was never tested. But it could be easy to fix. You can use USB sniffer on Windows and watch for the content of the SetSource packet (0x99100003) during switching inputs. See: https://github.com/marek-g/it9910hd_fusefs/blob/master/doc/IT9910_programming_guide.md. If you have any questions about that document, feel free to ask. The sniffer I have used on Windows was Wireshark.

So, the project already allows you to fuse-mount path with a ts file. But there are may be some problems with it:

• The first one is I have not implemented threading and buffering of data so currently there may be only one process that can open the file at the same time (it could be modified to allow more clients but it need some changes in the code, does not involve more communication with the device but rather caching the captured data and keeping track of multiple file positions - it may be difficult to do for example if clients read at different speeds, you may want to drop some data or have really huge buffers). So try to make OBS the only client (don't run the mpv command).

• The second problem is there is no correct file size and seek implementation, because it's a live stream. So not all the tools will work with it. Fortunately, mpv works great with it. But I assume OBS is not if you are asking (I haven't tested it). Maybe you can add logging to fuse's seek() call too see if OBS is calling it and try to return different error code, I'm not sure.

There is always another way. Instead of using FUSE, you can implement the v4l driver. It's good for the live capture streams and will work for sure with OBS and other capturing programs. The reason I haven't go that path was that v4l is (was?) not designed for devices that return compressed stream (other than motion jpeg). So the implementation would be pretty easy except you will have to decompress the stream in your driver (for example using libffmpeg). But then your device will behave like web camera. And I didn't like that solution, because capturing data will require full recompression of the stream while now it is just data copy process.

BTW. There are capture devices that work like usb camera and do not need custom drivers on Linux (they do not provide compression but the latency is much much smaller) and work on Linux with OBS out of the box.

[marek-g]

So try to make OBS the only client.

Be careful with that, because sometimes even having the folder open with file manager may block the file, because the file manager already opened the file :)

Oh, an I have another idea. Instead of writing your own v4l driver maybe you can search for an existing solution that simulates web camera based on the video file. Something like this: https://www.linuxfordevices.com/tutorials/linux/fake-webcam-streams

You can just try first if you can play the file with ffplay or convert it with ffmpeg command. If you can, the above driver should work also :)

[jibsaramnim]

Thank you very much for the reply! I think I might only be able to debug about half of what you said -- and that would be after some extensive and frantic searching around. I'm really out of my league here :D.

Oh, an I have another idea. Instead of writing your own v4l driver maybe you can search for an existing solution that simulates web camera based on the video file.

This actually works, that's pretty awesome! Unfortunately it seems like in order to get audio to work too, you need to run a separate ffmpeg instance for the audio stream, which wouldn't work in this case.

edit: I was also able to get it to work by using "Media source", de-selecting Local file, and providing the input path as file:///path/to/video/hdmi_stream.ts, although no audio works through this method either, oddly enough.

BTW. There are capture devices that work like usb camera and do not need custom drivers on Linux (they do not provide compression but the latency is much much smaller) and work on Linux with OBS out of the box.

I think this is honestly the wiser route. I should have researched this device before purchasing it a year or two ago. At the time I didn't expect I would use it connected up to a PC, as it can record directly to a USB drive/stick too, but the latter comes at great cost, as it seems to record at possibly the worst bitrates/settings possible. I originally bought this one as it supports both HDMI input as-well as component and composite, but it seems like you can't actually disable its own interlacing and rely on whatever software you might want to use, and the built-in interlacing is.. less than ideal.

I know this is now very much outside the scope of this repository, but do you have any device recommendations that work natively with Linux, too?

Thank you so much for taking the time to respond!

[marek-g]

Unfortunately it seems like in order to get audio to work too, you need to run a separate ffmpeg instance for the audio stream, which wouldn't work in this case.

What if you use cp command to copy the .ts file somewhere else and use 2 ffmpegs to access the destination file without any restrictions? :) Haha. We are building Rube Goldberg machine here ;)

However, I'm not really sure why the audio is not working. Maybe it's muted in the device (need another USB command to unmute it?) or wrong audio source is chosen (try to play with different --audio_src values). Please verify if audio is playing with mpv command because it was for me.

you can't actually disable its own interlacing

I know that pain :) These kinds of devices are also changing framerates to 30 or 60 fps. In my country the standard is 25 / 50 fps so also the smoothness is not as good as it should be with 25/50 fps signal.

do you have any device recommendations that work natively with Linux, too?

I was using some Chinese no-name YK9xx devices from Aliexpress and they worked (cannot find them now, sorry, probably out of sale) but the very important thing to verify is to check if it doesn't do the compression. I believe most (if not all) the USB devices that does not compress the signal are working on the web-cam protocol. Maybe look for one of these: https://www.reddit.com/r/linux_gaming/comments/c5m1at/i_tested_4_linuxcompatible_usb_30_hdmi_capture/

[jibsaramnim]

What if you use cp command to copy the .ts file somewhere else and use 2 ffmpegs to access the destination file without any restrictions? :) Haha. We are building Rube Goldberg machine here ;)

Hoboy, what a beast that would be haha. :D

However, I'm not really sure why the audio is not working. Maybe it's muted in the device (need another USB command to unmute it?) or wrong audio source is chosen (try to play with different --audio_src values). Please verify if audio is playing with mpv command because it was for me.

I have no idea either. I suppose it's comparatively easier to create a virtual machine and pass through this device just so I can rely on its actual driver, although as I no longer have extra GPUs there will be some hardware encoding constraints to add some more challenge there.

I was using some Chinese no-name YK9xx devices from Aliexpress and they worked (cannot find them now, sorry, probably out of sale) but the very important thing to verify is to check if it doesn't do the compression. I believe most (if not all) the USB devices that does not compress the signal are working on the web-cam protocol. Maybe look for one of these: https://www.reddit.com/r/linux_gaming/comments/c5m1at/i_tested_4_linuxcompatible_usb_30_hdmi_capture/

Thanks for the link! I have thrown in the towel for now. If I ever get the need for trying to more properly solve this conundrum I might try a new hardware device.

For now though, I want to thank you so much for what you've built, and for taking the time to reply to me. I got very excited as your solution got so close to giving a working solution with the hardware I already have. Almost! :)

[marek-g]

Dave, don't give up yet! Please consider two things:

1. [Easier] Comment out the "set_source" call (it9910hd_driver.rs, line 53). The device remembers its settings until reset. You can use the VM to setup the device (just start and stop playing) and then you can turn off the VM and it should still have the correct source selected (audio & video) in Linux until you restart the device. Not very convenient but better than using VM all the time.

2. I really encourage you to use usb sniffer (for example Wireshark) and compare the commands send by original software and my code (especially if step 1 works). Maybe you need to change only a single byte in 0x99100003 command or send a single new command that I'm not sending (you don't even need to understand the content, just send the same bytes). Also look at my doc. If you succeed you will get that feeling like you did something impossible :)

[jibsaramnim]

I love your enthusiasm! To be honest, if source switching was the only issue, I'd spend some extra time working on this when I have some free time. But I'd first have to find a way to get audio to work, as it's not very useful without this. At least not for my use-case, anyway. For some reason I can't remember if audio was working when playing back the "file" through mpv, I should first see if that's working.

[marek-g]

Dave, by "source switching" I mean both: video & audio source setup done by the same packet. I was mostly keeping in mind your audio problem and the above steps should help you with both.

[jibsaramnim]

Dave, by "source switching" I mean both: video & audio source setup done by the same packet. I was mostly keeping in mind your audio problem and the above steps should help you with both.

I just re-tried, and with mpv I'm actually getting audio from the HDMI source. So I switched back to OBS and wouldn't you know it, audio is working there now too!

The device remembers its settings until reset.

If I had to make a wild guess, this might actually be why I had trouble the last time around, as I had been poking around trying to get different sources to work. Perhaps the setting is preserved a longer than a quick power cycle (re-plugging the USB cable, basically).

So that leaves source switching. I have zero experience with a tool like Wireshark, but if there's a way to log changes I might be able to share some results with you? I don't think the maker of this device (MyGica, though it seems to just be a rebranded ezcap product of some sort) has any software of their own, but fortunately OBS under Windows supports switching source with this device, so hopefully that would work too.

[jibsaramnim]

1. [Easier] Comment out the "set_source" call (it9910hd_driver.rs, line 53).

Small follow-up about this; This line only runs if the device_model is returned as 2 by your get_hw_grabber method, but mine is returned as 0. It seems like the specific position you're looking for in the received array is actually 0 in my case. I'm not sure if it helps, but here's what's returned in it9910hd_driver.rs:238 for my device:

[
  0, 2, 0, 0, 2, 240, 16, 153, 1, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 26, 208,
  35, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 36, 51, 208, 0, 24,
  223, 28, 0, 42, 231, 48, 0, 24, 40, 248, 0, 24, 40, 248, 0, 0, 0, 0, 0, 42,
  231, 188, 0, 42, 231, 212, 0, 36, 49, 132, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  29, 0, 0, 0, 29, 0, 0, 0, 0, 0, 0, 0, 0, 0, 26, 208, 35, 0, 42, 232, 192, 0,
  0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0, 0, 0, 0, 115, 0, 0, 0, 0, 0, 26, 234, 137,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  0, 0, 0, 0, 0, 0, 0, 26, 208, 35, 0, 42, 231, 224, 0, 0, 0, 0, 0, 0, 0, 0, 0,
  26, 208, 35, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 1, 0, 42, 232, 128, 0, 0, 0, 4,
  0, 0, 0, 0, 0, 42, 232, 12, 0, 0, 0, 10, 0, 0, 0, 48, 0, 25, 31, 60, 0, 42,
  232, 36, 0, 44, 26, 160, 0, 0, 0, 48, 0, 44, 26, 160, 0, 36, 46, 48, 0, 25,
  36, 8, 0, 42, 232, 56, 0, 38, 5, 84, 0, 36, 46, 48, 0, 44, 26, 112, 0, 24, 67,
  48, 0, 42, 232, 72, 0, 36, 49, 132, 0, 0, 0, 48, 0, 23, 156, 140, 0, 42, 232,
  96, 0, 0, 0, 48, 0, 36, 49, 132, 0, 44, 26, 112, 0, 36, 46, 48, 0, 23, 79, 28,
  0, 42, 232, 120, 0, 36, 46, 48, 0, 36, 49, 132, 0, 42, 232, 188, 0, 0, 0, 1,
  0, 42, 232, 160, 0, 26, 208, 10, 0, 26, 246, 30, 0, 0, 0, 0, 0, 42, 232, 176,
  0, 26, 208, 10, 0, 26, 246, 30, 0, 0, 0, 0, 0, 42, 232, 240, 0, 36, 247, 164,
  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 23, 144, 24, 0, 42, 232,
  184, 0, 8, 59, 16, 0, 26, 246, 30, 0, 27, 11, 220, 0, 0, 0, 0, 0, 42, 232,
  224, 208, 80, 0, 0, 0, 42, 232, 244, 2, 171, 77, 152, 0, 36, 46, 48, 0, 0, 0,
  1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 42, 232,
  244, 0, 42, 232, 248
]

Edit: My device also seems to automatically switch to whatever input it notices activity on. That would suggest that with the set_source() method never being triggered and the device auto-switching it should natively just pick up on whatever is being sent to the computer -- unless something else is throwing a wrench in the way of course.

Edit #2: It's entirely possible the device defaults back to not automatically picking input sources if it's being used over USB, it might simply assume the USB host always picks the source. That might be the aforementioned wrench :D.

[marek-g]

audio is working there now too

That's great!! :)

Perhaps the setting is preserved a longer than a quick power cycle (re-plugging the USB cable, basically).

Yes. It has its own power supply. It must be turned off and on to reset settings. And it sometimes can be in a wrong state.

This line only runs if the device_model is returned as 2 by your get_hw_grabber method, but mine is returned as 0 My device also seems to automatically switch to whatever input it notices activity on.

Hm... That part of code comes from reverse engineering the player app included with my unit (also rebranded ezcap called Ria if I remember correctly). My unit was returning 2. If your device switches inputs automatically the command may do nothing on your device.

[marek-g]

I have zero experience with a tool like Wireshark, but if there's a way to log changes I might be able to share some results with you?

Yes. If you still have any problems with it, feel free to share logs with me. Wireshark can export logs to the file.

[jibsaramnim]

If your device switches inputs automatically the command may do nothing on your device.

My device definitely supports switching sources from software, as I distinctly remember doing that before with OBS under windows. I am not sure if its auto-switching capabilities is limited to its self-recording feature (it can record straight to a USB storage device too, though with poor and unchangeable quality settings) or if it also applies to when its being used with a USB host.

My unit was returning 2. I had previously attempted actually force-skipping this check, basically forcibly enabling the set_source method, but doing just that wasn't enough. Without relying on something like Wireshark as you mentioned I'd just be poking in the dark, really.

I am installing Windows in a VM right now as it's been a while since I had one, when I have a chance to try out Wireshark I'll report back with my findings :).

[marek-g]

0 in my case

The original reverse engineered code expected only 0x17, 0x27 and 0x37 values here. So you can always try what happens when you enable that set_source() call.

[jibsaramnim]

1. You can use the VM to setup the device (just start and stop playing)

Update time: I was able to get this to work! I mounted the USB device to the VM, used OBS to set the right source, then disconnected the USB device and ran your tool again. Launched mpv and boom, composite in working. With the wrong aspect ratio and audio seems to skip out every now and then, which is possibly related to an error mpv is spamming the logs with, as it says Invalid audio PTS: followed by two numbers that seem close to one another but not the same.

AV: 00:00:02 / 00:00:03 (63%) A-V:  0.277 ct:  0.460 Cache: 0.7s/1MB
Invalid audio PTS: 2.827667 -> 2.667000
AV: 00:00:03 / 00:00:04 (79%) A-V:  0.006 ct:  0.570 Cache: 0.7s/1MB
Invalid audio PTS: 3.925667 -> 3.767000
AV: 00:00:04 / 00:00:05 (85%) A-V: -0.040 ct:  0.458 Cache: 0.6s/1006KB
Invalid audio PTS: 5.025667 -> 4.833000
AV: 00:00:05 / 00:00:06 (90%) A-V: -0.113 ct:  0.338 Cache: 0.5s/787KB
Invalid audio PTS: 6.155667 -> 6.000000
AV: 00:00:07 / 00:00:07 (92%) A-V: -0.155 ct:  0.224 Cache: 0.5s/679KB
Invalid audio PTS: 7.237333 -> 7.067000
AV: 00:00:08 / 00:00:08 (95%) A-V: -0.195 ct:  0.094 Cache: 0.4s/512KB
Invalid audio PTS: 8.325667 -> 8.133000
AV: 00:00:09 / 00:00:09 (95%) A-V: -0.195 ct: -0.099 Cache: 0.4s/512KB
Invalid audio PTS: 9.562333 -> 9.400000
AV: 00:00:10 / 00:00:10 (97%) A-V: -0.197 ct: -0.259 Cache: 0.2s/290KB
Invalid audio PTS: 10.658667 -> 10.467000
AV: 00:00:11 / 00:00:12 (97%) A-V: -0.202 ct: -0.445 Cache: 0.3s/346KB
Invalid audio PTS: 11.789667 -> 11.633000
AV: 00:00:12 / 00:00:13 (98%) A-V: -0.202 ct: -0.602 Cache: 0.2s/234KB
Invalid audio PTS: 12.891667 -> 12.733000
AV: 00:00:13 / 00:00:13 (96%) A-V: -0.248 ct: -0.715 Cache: 0.4s/346KB
(etc..)

But regardless, progress? :)

The original reverse engineered code expected only 0x17, 0x27 and 0x37 values here. So you can always try what happens when you enable that set_source() call.

I had already tried that yes (vaguely mentioned earlier on too), but I had little luck with getting it to work so far. That, too, could be due to not knowing exactly what sources to actually enter, mind you.

---

Update #1: Another update: By looking at the IDs OBS under Windows lists as available input sources, if I forcibly modify get_hw_grabber to always return device id 2 I can actually successfully use your script to select the correct composite video + audio sources (--video_src 3 --audio_src 7). So if Ijust map out which IDs are what that theoretically that would mean I don't even need the VM to pre-configure the device. Nice!

Still not entirely sure what the audio issue is though, and to elaborate on that; it's not just the audio that has the occasional hiccup, it seems like the video stream temporarily freezes along with it. I'm not sure if that's just how mpv handles video if audio is having an issue though.

---

Edit #2: It seems like the actual resolution setting isn't having any effect, so I thought I'd try to see if I can get a usable Wireshark log for that. Unfortunately OBS only seems to show two supported resolutions by this device (720x480 and 160x120) I switched to the awkward 160x120 resolution while having Wireshark on, which is hopefully included in the log. I am trying but so far the log is not making any sense to me whatsoever :joy:

enabling-and-attempting-resolution-switch.pcapng.gz

Forgot to add; if I manually type in a resolution like 720x576 it seems to not want to work at all, with OBS not showing anything coming in from the device. It might be possible that my particular device has a very limited set of resolutions available.

---

Edit #3: Consolidated my spam comments into one, sorry for not doing that right away.

Another update; I had been testing with composite only so far, so I grabbed a PS2 just now to try component input. It seems like the resolutions are actually properly sent! Testing with 640x480 I'm actually getting a square video coming in:

With this test I am forcing get_hw_grabber to always return 2, which seems to do the trick.

Your tool is blowing my mind. Reverse engineered using one device, it seems to be virtually perfectly compatible with this completely different brand device I've got here. Fantastic!

[marek-g]

Hi Dave! Only a short reply (before I will be able to look at Wireshark logs) as I'm in the middle of my daily job right now :)

It's amazing you didn't give up. It would be a pity - you were so close :))

1. I remember that I had the same audio errors with mpv, however the sound sounded good. Maybe bad sound is related with #2.

2. Your country historically were using NTSC system, in my country it was PAL. So correct resolution/fps for me was 720x576 and 25/50 fps. And for you it is 720x480 and 30/60 fps. You definitely should stick with these values. Your composite signal has only 480 lines, that's why it cannot work with 576 setting. Please set 720x480 resolution. But please also change fps to 30! I think having it at 25 may cause the video freezes and hopefully audio hiccups too. Please change default fps in the code or pass a new value by parameter.

Edit: Actually it's 640x480 for NTSC and 30/60 fps of course.

Reverse engineered using one device, it seems to be virtually perfectly compatible with this completely different brand device I've got here. Fantastic!

Yes :) It's very common that many companies just order hardware in Chinese factory and asks for re-branding. The same is for example with e-ink readers. I was able to move firmware between different manufacturers (Kobo and Tolino share the same hardware, you can even take Android from Tolino and use with Kobo which originally is based on plain Linux only and most of other e-ink devices are clones of Boyoue/Likebook).

And thanks for the screenshots!!!! It's really fantastic to see real images :)))))

[marek-g]

Please set 720x480 resolution.

I think the correct aspect ratio should be 640x480, but maybe device can capture only 720x480. I believe mpv and OBS have settings to rescale it.

Edit: Oh, sorry - you've mentioned that 640x480 works for you. Great!! Just change the FPS and you should be good.

[jibsaramnim]

Your country historically were using NTSC system, in my country it was PAL.

Actually, our home countries are the same in that regard. My situation is a little less common, in that I brought my childhood PAL SNES and N64 with me to the country I now live in, which is NTSC as you already noticed. That's why I was using those resolutions/framerates for the SNES and N64 tests, and 640x480 when testing with the NTSC region Dreamcast I have here. It's a bit confusing, I know :D.

I haven't had time to look more into it yet, but as I mentioned in my recent-most edit, I think most stuff actually seems to be working, which is really cool. All I had to do in the end was forcibly return device id 2 just so that the rest of the code can do things like send the resolution setting trigger and whatnot, and I had to test a few different resolutions to find the ones that worked.

Of course, a cleaner solution would be to have the device ID not be hard-set. I'm not sure if you're able to recognize anything useful in what I shared in this comment, I would love to get a pointer or two so I know what to look for. If I'm able to wrap this up into a nice PR we could end up with official support of this particular device too, and presumably several others that have the same internals. Please do let me know if you've got any ideas as to how to make that happen, I'm all ears! :)

Edit: Because sharing is caring, I've uploaded two test recordings I made, just in case you're curious to see what the results look like with this particular device. I haven't yet tested recording something in progressive mode, that would be next up on my todo list for when I have a moment again.

• Tekken Tag Tournament, (PS2, Component, NTSC): https://youtu.be/wEN4g-JeqJI
• Legend of Zelda: Ocarina of Time (N64, Composite, PAL): https://youtu.be/PzPbcJJHtRA

[marek-g]

I'm really glad it finally works for you, Dave!! Even if you are the only user it was worth to write the code :)

I remember that I have written to the producer of the chip asking for the documentation because I would like to use it on Linux and they denied my request. So I spent two weeks hard working (about 80 hours in total) because of their response, I had a few points of resignation (especially when trying to understand meaning of the random data sent to device because someone was not clearing his buffers). But I did it and I though - ok, now it is my hard work - I haven't seen your documentation at all, so the results are my property and I can do whatever I want to do with it. So I published it as public domain :)) Let everyone benefit from it.

I think the reason why they have denied my request may be related with the fact that this chip decrypts HDCP and they cannot officially say this. This is another story why DRM is so bad..... I cannot write my own convenient video player for DRM protected web stream even when I'm paying for the subscription. I have to use web browser with inconvenient window which is four time CPU more hungry and has 10 times as needed memory usage, a few times slower startup and un-configurable OSD.... So, at least thanks to the producer of IT chip that you can at least watch HDMI stream from your own device.

I share your sentiment for SNES and Nintendo. I'm still playing old Amiga games (like Alien Breed, Cannon Fodder, Lemmings)... but only on emulator. I have digital joystick from '90-ies connected through old keyboard's USB chip (I was so happy when I had that idea and it started working).

Your videos look very good! Beside the low resolution I would not say it's from component/composite input!! Looks fantastic and smooth!!

It did a small change in the source code. In the case of "0x00" or unknown device type I'm assuming the device has ability to configure inputs. I think this is a much safer assumption. If it has multiple inputs, it will work. If it has not, sending additional command will probably not hurt.

Thank you for your perseverance and helping to improve this code and have a good usage of the device!!