search

JuliaIO / VideoIO.jl

Reading and writing of video files in Julia via ffmpeg
https://juliaio.github.io/VideoIO.jl/stable
Other
128 stars 53 forks source link

Camera show almost working on Windows 10 #146

Closed hearnsj closed 3 years ago

hearnsj commented 5 years ago

I am very pleased to report that VideoIO 0.5.3 builds correctly on Windows 10. Thanks guys! But sadly on my HP laptop the viewcam function almost works

julia> VideoIO.viewcam() ERROR: Could not open file HP TrueVision FHD RGB-IR Stacktrace: [1] error(::String) at .\error.jl:33 [2] open_avinput at C:\Users\hearn.julia\packages\VideoIO\zRxWD\src\avio.jl:174 [inlined] [3] #AVInput#1(::Int64, ::Type, ::String, ::Ptr{VideoIO.AVFormat.AVInputFormat}) at C:\Users\hearn.julia\packages\VideoIO\zRxWD\src\avio.jl:198 [4] Type at C:\Users\hearn.julia\packages\VideoIO\zRxWD\src\avio.jl:183 [inlined] [5] #opencamera#7 at C:\Users\hearn.julia\packages\VideoIO\zRxWD\src\avio.jl:701 [inlined] [6] opencamera at C:\Users\hearn.julia\packages\VideoIO\zRxWD\src\avio.jl:701 [inlined] [7] viewcam(::String, ::Ptr{VideoIO.AVFormat.AVInputFormat}) at C:\Users\hearn.julia\packages\VideoIO\zRxWD\src\VideoIO.jl:45 (repeats 2 times) [8] top-level scope at none:0

ffmpeg report this device ffmpeg -list_devices true -f dshow -i dummy ffmpeg version N-92086-gd702769213 Copyright (c) 2000-2018 the FFmpeg developers built with gcc 8.2.1 (GCC) 20180813 configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth libavutil 56. 19.101 / 56. 19.101 libavcodec 58. 31.102 / 58. 31.102 libavformat 58. 18.103 / 58. 18.103 libavdevice 58. 4.105 / 58. 4.105 libavfilter 7. 33.100 / 7. 33.100 libswscale 5. 2.100 / 5. 2.100 libswresample 3. 2.100 / 3. 2.100 libpostproc 55. 2.100 / 55. 2.100 [dshow @ 00000186d54ba100] DirectShow video devices (some may be both video and audio devices) [dshow @ 00000186d54ba100] "HP TrueVision FHD RGB-IR" [dshow @ 00000186d54ba100] Alternative name "@devicepnp\?\usb#vid_064e&pid_3401&mi_00#6&2045b01b&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global" [dshow @ 00000186d54ba100] DirectShow audio devices [dshow @ 00000186d54ba100] "Microfoonmatrix (Realtek High Definition Audio(SST))" [dshow @ 00000186d54ba100] Alternative name "@devicecm{33D9A762-90C8-11D0-BD43-00A0C911CE86}\wave_{4B2CCB96-C32D-4B85-BC58-B1C623A1372E}"

hearnsj commented 5 years ago

How can I help with debugging?

hearnsj commented 5 years ago

I can display the video stream from my camera using ffmpeg -f dshow -i video="HP TrueVision FHD RGB-IR" -pix_fmt yuv420p -f sdl "SDL Output"

hearnsj commented 5 years ago

I think the problem is with the default video format for my camera. In src/ffmpeg/AVFormat there are directories: Mode LastWriteTime Length Name

d----- 11/03/2019 11:28 src d----- 11/03/2019 11:28 v53 d----- 11/03/2019 11:28 v54 d----- 11/03/2019 11:28 v55 d----- 11/03/2019 11:28 v56 d----- 11/03/2019 11:28 v57 d----- 11/03/2019 11:28 v58

The version of VideoIO I have is 0.5.3 I guess there is an 0.5.8 version which uses the code in 0.58 ???

hearnsj commented 5 years ago

julia> VideoIO.get_camera_devices("ffmpeg","dshow","dummy") 2-element Array{String,1}: "HP TrueVision FHD RGB-IR" "@devicepnp\\?\usb#vid_064e&pid_3401&mi_00#6&2045b01b&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global"

kmsquire commented 5 years ago

I guess there is an 0.5.8 version which uses the code in 0.58 ???

The versions here are the versions of the individual libraries within ffmpeg:

julia> VideoIO.versioninfo()
Using ffmpeg
AVCodecs version 58.35.100
AVFormat version 58.20.100
AVUtil version 56.22.100
SWScale version 5.3.100
AVDevice version 58.5.100
AVFilters version 7.40.101
julia> VideoIO.viewcam()
ERROR: Could not open file HP TrueVision FHD RGB-IR

I was going to give some hints on how to debug this, but while I was exploring it, I discovered some obvious issues that might be causing this.
When you get the chance, can you see if #147 fixes your issues?

(On a Mac, it gets close, although I get an error about the selected framerate not being supported by my hardward.)

kmsquire commented 5 years ago

147 is now merged, and I'm going to tag a new version. I can't test if it actually fixes this issue--when you get the chance, can you either checkout master, or wait until the tag gets merged and update? Thanks!

hearnsj commented 5 years ago

Arrgggh. I tried that branch ulia> using VideoIO [ Info: Recompiling stale cache file C:\Users\hearn.julia\compiled\v1.1\VideoIO\tZxJ7.ji for VideoIO [d6d074c3-1acf-5d4c-9a43-ef38773959a2] ┌ Warning: Package VideoIO does not have Libdl in its dependencies: │ - If you have VideoIO checked out for development and have │ added Libdl as a dependency but haven't updated your primary │ environment's manifest file, try Pkg.resolve(). │ - Otherwise you may need to report an issue with VideoIO └ Loading Libdl into VideoIO from project dependency, future warnings for VideoIO are suppressed. WARNING: Method definition zero(Type{VideoIO.AVUtil.AVRational}) in module AVUtil at C:\Users\hearn.julia\packages\VideoIO\gjR7u\src\ffmpeg\AVUtil\v56\libavutil_h.jl:644 overwritten at C:\Users\hearn.julia\packages\VideoIO\gjR7u\src\ffmpeg\AVUtil\src\AVUtil.jl:7.

julia> VideoIO.viewcam() ERROR: UndefVarError: viewcam not defined Stacktrace: [1] getproperty(::Module, ::Symbol) at .\sysimg.jl:13 [2] top-level scope at none:0

kmsquire commented 5 years ago

Did you import ImageView before VideoIO? It's an optional dependency, and viewcam won't be defined if it isn't loaded first.

hearnsj commented 5 years ago

I hope I added the master branch correctly Pkg.status [d6d074c3] VideoIO v0.5.4+ #master (https://github.com/JuliaIO/VideoIO.jl)

Now I get:

julia> VideoIO.viewcam() [dshow @ 000000002595b700] Malformed dshow input string. ERROR: Could not open file HP TrueVision FHD RGB-IR Stacktrace: [1] error(::String) at .\error.jl:33 [2] open_avinput at C:\Users\hearn.julia\packages\VideoIO\gjR7u\src\avio.jl:178 [inlined] [3] #AVInput#1(::Int64, ::Type, ::String, ::Ptr{VideoIO.AVFormat.AVInputFormat}) at C:\Users\hearn.julia\packages\VideoIO\gjR7u\src\avio.jl:201 [4] Type at C:\Users\hearn.julia\packages\VideoIO\gjR7u\src\avio.jl:187 [inlined] [5] #opencamera#7 at C:\Users\hearn.julia\packages\VideoIO\gjR7u\src\avio.jl:673 [inlined] [6] opencamera at C:\Users\hearn.julia\packages\VideoIO\gjR7u\src\avio.jl:673 [inlined] [7] viewcam(::String, ::Ptr{VideoIO.AVFormat.AVInputFormat}) at C:\Users\hearn.julia\packages\VideoIO\gjR7u\src\VideoIO.jl:95 (repeats 2 times) [8] top-level scope at none:0

hearnsj commented 5 years ago

Forgive me if I have got something fouled up. I have removed and added this package quite a few times

kmsquire commented 5 years ago

No, I'm pretty sure it isn't your fault--there just haven't been many recent contributors who use Windows.

Can you report the values of VideoIO.CAMERA_DEVICES, VideoIO.DEFAULT_CAMERA_DEVICE, and VideoIO.DEFAULT_CAMERA_FORMAT?

kmsquire commented 5 years ago

Actually, @hearnsj, can you try #149? A little googling around suggests that, for dshow, we need to prepend video= to the camera name, and quote the name.

Since you seem to be on master right now, you should just be able to check out kms/fix_dshow_video_device_spec, and then build VideoIO.

hearnsj commented 5 years ago

Wii do

On Thu, 21 Mar 2019 at 04:42, Kevin Squire notifications@github.com wrote:

Actually, @hearnsj https://github.com/hearnsj, can you try #149 https://github.com/JuliaIO/VideoIO.jl/pull/149? A little googling around suggests that, for dshow, we need to prepend video= to the default camera name, and quote the name.

Since you seem to be on master right now, you should just be able to check out kms/fix_dshow_video_device_spec, and then build VideoIO.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JuliaIO/VideoIO.jl/issues/146#issuecomment-475111644, or mute the thread https://github.com/notifications/unsubscribe-auth/ASXgxe-N4DgnM0hsSLpZ2X3idunAHPkbks5vYw2ggaJpZM4boadW .

hearnsj commented 5 years ago

:-( With branch kms/fix_dshow_video_device_spec

julia> VideoIO.viewcam() [dshow @ 000000001e4fc580] Could not find video device with name ["HP TrueVision FHD RGB-IR"] among source devices of type video. ERROR: Could not open file video="HP TrueVision FHD RGB-IR" Stacktrace: [1] error(::String) at .\error.jl:33 [2] open_avinput at C:\Users\hearn.julia\packages\VideoIO\0q6II\src\avio.jl:178 [inlined] [3] #AVInput#1(::Int64, ::Type, ::String, ::Ptr{VideoIO.AVFormat.AVInputFormat}) at C:\Users\hearn.julia\packages\VideoIO\0q6II\src\avio.jl:201 [4] Type at C:\Users\hearn.julia\packages\VideoIO\0q6II\src\avio.jl:187 [inlined] [5] #opencamera#7 at C:\Users\hearn.julia\packages\VideoIO\0q6II\src\avio.jl:673 [inlined] [6] opencamera at C:\Users\hearn.julia\packages\VideoIO\0q6II\src\avio.jl:673 [inlined] [7] viewcam(::String, ::Ptr{VideoIO.AVFormat.AVInputFormat}) at C:\Users\hearn.julia\packages\VideoIO\0q6II\src\VideoIO.jl:95 (repeats 2 times) [8] top-level scope at none:0

hearnsj commented 5 years ago

Using the magic of the MagneticReadHead debugger

reakpoint Hit: opencamera(device, format, args...) in VideoIO at C:\Users\hearn.julia\packages\VideoIO\0q6II\src\avio.jl:673

➧ function opencamera(device=DEFAULT_CAMERA_DEVICE, format=DEFAULT_CAMERA_FORMAT, args...; kwargs...) camera = AVInput(device, format) VideoReader(camera, args...; kwargs...)

Vars: device, format, args Commands: CC (Continue), SI (Step In), SN (Step Next), SO (Step Out), XX (Abort) iron>args () iron>device "video=\"HP TrueVision FHD RGB-IR\"" iron>format Ptr{VideoIO.AVFormat.AVInputFormat} @0x00007ff858644000 iron>args () iron>

kmsquire commented 5 years ago

I wonder if it needs the quotes. Can you try removing them from here? Another thing you can try is to simply set the name to video=0 (without extra quotes).

I've been testing some of these changes on my Mac as well, and found that my camera couldn't handle the default frame rate (29.97--NTSC). The kms/avdictionary_framerate branch (#150) starts to address this problem (although this would be after fixing your current issue).

hearnsj commented 5 years ago

We almost have it working. It is the quotes. VideoIO.viewcam("video=HP TrueVision FHD RGB-IR")

I get a single frame still picture and lots of these messages

ze: 3041280 [rtbufsize parameter])! frame dropped! [dshow @ 0000000026e51600] real-time buffer [HP TrueVision FHD RGB-IR] [video input] too full or near too full (102% of size: 3041280 [rtbufsize parameter])! frame dropped! [dshow @ 0000000026e51600] real-time buffer [HP TrueVision FHD RGB-IR] [video input] too full or near too full (102% of size: 3041280 [rtbufsize parameter])! frame dropped! [dshow @ 0000000026e51600] real-time buffer [HP TrueVision FHD RGB-IR] [video input] too full or near too full (102% of size: 3041280 [rtbufsize parameter])! frame dropped! [dshow @ 0000000026e51600] real-time buffer [HP TrueVision FHD RGB-IR] [video input] too full or near too full (102% of size: 3041280 [rtbufsize parameter])! frame dropped! [dshow @ 0000000026e51600] real-time buffer [HP TrueVision FHD RGB-IR] [video input] too full or near too full (102% of size: 3041280 [rtbufsize parameter])! frame dropped! [dshow @ 0000000026e51600] real-time buffer [HP TrueVision FHD RGB-IR]

kmsquire commented 5 years ago

Okay, progress! For this one, you'll need to switch to the kms/avdictionary_framerate branch, and then run something like

VideoIO.viewcam("video=HP TrueVision FHD RGB-IR", VideoIO.DEFAULT_CAMERA_FORMAT, VideoIO.AVDict("rtbufsize" => "4M"))

You can change this size to whatever you need (I saw someone set it to 1024M online). You also might want to lower the framerate ("framerate" => 20).

In theory, you should be able to set whatever parameters you set on the commandline here, so you could also change the resolution, etc.

(The interface needs to be cleaned up--e.g., you shouldn't need to specify the DEFAULT_CAMERA_FORMAT, and it would be nice to just pass extra parameters as part of the function call to viewcam. Will try to make that change.

zygmuntszpak commented 5 years ago

Hi all,

I'll be doing a lot more video processing in the near future so I am keen to help out here as well. I work on both Windows and Linux.

I've just started catching up on some of the discussions happening here. I think instead of 

VideoIO.viewcam("video=HP TrueVision FHD RGB-IR", VideoIO.DEFAULT_CAMERA_FORMAT, VideoIO.AVDict("rtbufsize" => "4M"))

you mean:

VideoIO.opencamera("video=HP TrueVision FHD RGB-IR", VideoIO.DEFAULT_CAMERA_FORMAT, VideoIO.AVDict("rtbufsize" => "4M"))

I checked out kms/avdictionary_framerate and viewcam is not defined for those sets of arguments.

kmsquire commented 5 years ago

@hearnsj @ianshmean and @SimonDanisch have done some work which might help this. Can you give it another go when you get the chance? Thanks.

hearnsj commented 5 years ago

Almost there....
julia> VideoIO.CAMERA_DEVICES 2-element Array{String,1}: "HP TrueVision FHD RGB-IR" "@devicepnp\\?\usb#vid_064e&pid_3401&mi_00#6&2045b01b&0&0000#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global"

I am not sure where the second video device is coming from. My laptop is plugged into a Thunderbolt docking station

SimonDanisch commented 5 years ago

The second seems to be the same camera. ffmpeg shows it "alternative name: "@devicepnp...."

hearnsj commented 5 years ago

Yes, sorry. I just disconnected the docking station and rebooted. The device list remains the same.

On Thu, 9 May 2019 at 14:32, Simon notifications@github.com wrote:

The second seems to be the same camera. ffmpeg shows it "alternative name: "@devicepnp...."

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JuliaIO/VideoIO.jl/issues/146#issuecomment-490906357, or mute the thread https://github.com/notifications/unsubscribe-auth/AES6BRI4E2C7INIRUU53GATPUQRYVANCNFSM4G5BU5LA .

SimonDanisch commented 5 years ago

What I meant is, that it's fine... Or what's the problem with it?

IanButterworth commented 5 years ago

@hearnsj What's the error when you run:

using Makie, VideoIO
VideoIO.viewcam()
hearnsj commented 5 years ago

VideoIO.viewcam() ...long pause.... Unable to decode APP fields (cant see the complete message)

…camera light comes on... Many reports of dshow @ 000000000b139bc0] real-time buffer [HP TrueVision FHD RGB-IR] [video input] too full or near too full (101% of size: 3041280 [rtbufsize parameter])! frame dropped!

Yes, we have discussed in th epast how to alter the rtbufsize

hearnsj commented 5 years ago

I captured the output

[mjpeg @ 000000001e6d0e00] unable to decode APP fields: Invalid data found when processing input [dshow @ 00000000210190c0] real-time buffer [HP TrueVision FHD RGB-IR] [video input] too full or near too full (70% of size: 3041280 [rtbufsize parameter])! frame dropped!

SimonDanisch commented 5 years ago 
Many reports of
dshow @ 000000000b139bc0] real-time buffer [HP TrueVision FHD RGB-IR] [video input] too full or near too full (101% of size: 3041280 [rtbufsize parameter])! frame dropped!

That's likely because during compilation of the viewer, we don't consume any frames....We could get around this if we reorganize the code, to first show an image, so that we are sure that everything got compiled, and only then start streaming.

IanButterworth commented 5 years ago

I’m wondering if a quick fix to this would be if it’s possible to control how ffmpeg buffers frames. NewestOnly would be simplest, or for trying to capture every frame, an OldestFirst order

hearnsj commented 5 years ago

I am very pleased to report: Windows 10 Julia 1.1.1 Makie v0.9.4 VideoIO 0.6.5

VideoIO.viewcam() starts up and displays the output from my laptop camera! There is still a lot of output in the background saying Unable to decode APP fields

IanButterworth commented 3 years ago

Should be fixed now. I'll close but please report back if there are issues with v0.9 and julia 1.6