Native MJPEG,YUV420p , YUY2 support?

[eix128]

Hello , i have logitech c910 webcam , it gives MJPEG support natively. Java Webcam Capture API may have native MJPEG support with webcams?

Also some webcams natively support YUV420p output. That will be good for VP8 , H264 encoding because all require YUV420P.

there is also YUY2 encoding support natively by webcams. That is easier to convert YUY2 => YUV420p instead of RGB.

These are the things required much for realtime webcam processing.

For example :

Webcam webcam = Webcam.getDefault();
WebcamFormat formats[] = webcam.getNativeFormats();
WebcamFormat formatFound;
for(WebcamFormat format : formats) {
       System.out.println(format);
       if(format == WebcamFormat.YUV420P) {
               formatFound = WebcamFormat.YUV420P;
       }
}
ByteBuffer yuv420pData;
if(formatFound == WebcamFormat.YUV420P) {
       yuv420pData = webcam.getImageByteBuffer(formatFound);

} else if(formatFound == WebcamFormat.RGB24) {
       ByteBuffer rgbData = webcam.getImageByteBuffer(formatFound);
       yuv420pData = GPUUtils.RGB24toYUV420P(rgbData);
}

//send YUV420p data to FFMPEG
VideoFrame frame = encoder.encodeVideo(yuv420pData,picture);

Thanks :)

[sarxos]

Hi @jduke32,

This would be indeed very useful feature :) The problem currently lies in the format detection - on Linux this can be achieved pretty easily (and it's already done on top of v4l4j, please see V4l4jDevice to check how it finds the most optimal format), but on Windows it's paint in the arse so I'm considering to move from BridJ to javacpp to gain possibility to access low-level videoinput API via videoinput-preset (on Windows) which is a must if we ever consider adding formats on this platform. I still have to find out how to translate format detection from C++ to Java without a need of JNI.

[eix128]

Need GPUUtils with the help of AMD Aparapi. Future is GPU its better to use AMD Aparapi for that. JavaCPP is very powerful and most promising JNI API its far better then BridJ because it has zero overhead java and jni

[eix128]

Windows needs DirectShow for that i have some source code i can share with you

[eix128]

#define MIN(x,y) (x)<(y) ? (x) : (y);

void YUY2toRGB24(BYTE *pDst, BYTE *pSrc) {
    for(long lSrc=0,lDst=0; lSrc<320*240*2; lSrc=lSrc+4,lDst=lDst+6) {
        int y,u,v;
        y = *(pSrc+lSrc+0);
        u = *(pSrc+lSrc+1);
        v = *(pSrc+lSrc+3);
        *(pDst+lDst+2) = MIN((9535 * (y - 16) + 13074 * (v - 128)) >> 13, 255);
        *(pDst+lDst+1) = MIN((9535 * (y - 16) - 6660 * (v - 128) - 3203 * (u - 128)) >> 13, 255);
        *(pDst+lDst+0) = MIN((9535 * (y - 16) + 16531 * (u - 128)) >> 13, 255);
        y = *(pSrc+lSrc+2);
        *(pDst+lDst+5) = MIN((9535 * (y - 16) + 13074 * (v - 128)) >> 13, 255);
        *(pDst+lDst+4) = MIN((9535 * (y - 16) - 6660 * (v - 128) - 3203 * (u - 128)) >> 13, 255);
        *(pDst+lDst+3) = MIN((9535 * (y - 16) + 16531 * (u - 128)) >> 13, 255);
    }
}

This needs GPU Accellerated solution , can someone help with that for AMD Aparapi ?

[eix128]

INT InitWebcam( INT id ) {
    g_bWebcamInit   = FALSE;
    SampleGrabberEx*    sample_grabber = NULL;

    HRESULT hr;

    hr = CoCreateInstance (CLSID_FilterGraph, NULL , CLSCTX_INPROC , IID_IGraphBuilder , (void **) &g_pGraph );

    if (FAILED(hr)) {
        return 1;
    }

    // Create the capture graph builder
    hr = CoCreateInstance (CLSID_CaptureGraphBuilder2 , NULL, CLSCTX_INPROC,IID_ICaptureGraphBuilder2, (void **) &g_pCapture);

    if (FAILED(hr)) {
        g_pGraph->Release();
        return 2;
    }

    // Obtain interfaces for media control and Video Window
    hr = g_pGraph->QueryInterface(IID_IMediaControl,(LPVOID *) &g_pMC);
    if (FAILED(hr)) {
        g_pCapture->Release();
        g_pGraph->Release();
        return 3;
    }

    hr = g_pGraph->QueryInterface(IID_IVideoWindow, (LPVOID *) &g_pVW);
    if (FAILED(hr)) {
        g_pMC->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        return 4;
    }

    hr = g_pGraph->QueryInterface(IID_IMediaEvent, (LPVOID *) &g_pME);
    if (FAILED(hr)) {
        g_pVW->Release();
        g_pMC->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        return 5;
    }

    g_pVW->put_Visible(OAFALSE);

    hr = NULL;
    g_pME->SetNotifyFlags( 0 );

    IBaseFilter *pSrcFilter=NULL;
    hr = g_pCapture->SetFiltergraph(g_pGraph);

    if(FAILED(hr)) {
        g_pVW->Release();
        g_pMC->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        return 2;
    }

    hr = FindCaptureDevice(&pSrcFilter,id);
    if(FAILED(hr) || pSrcFilter == NULL) {
        g_pVW->Release();
        g_pMC->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        return 3;
    }

    // Add Capture filter to our graph.
    hr = g_pGraph->AddFilter(pSrcFilter, L"Video Capture");
    if(FAILED(hr)) {
        g_pVW->Release();
        g_pMC->Release();
        pSrcFilter->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        return 4;
    }

    //aaaaa

    hr = CoCreateInstance(CLSID_SampleGrabber, NULL,CLSCTX_INPROC, IID_ISampleGrabber, (void**)&g_pGrabber);
    CComQIPtr< IBaseFilter, &IID_IBaseFilter > pGrabberBase( g_pGrabber );
    if(FAILED(hr)) {
        g_pVW->Release();
        g_pMC->Release();
        pSrcFilter->Release();
        g_pGrabber->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        return 5;
    }

    g_pGrabber->SetBufferSamples(FALSE);
    g_pGrabber->SetOneShot(FALSE);

    g_pGraph->AddFilter(pGrabberBase, L"Sample Grabber");
    if(g_lpSampleGrabber)
        delete g_lpSampleGrabber;

        sample_grabber = new SampleGrabberEx();//grab();
            g_lpSampleGrabber = sample_grabber;
        //sample_grabber = grab;
        //0: SampleCB (the buffer is the original buffer, not a copy)
        //1: BufferCB (the buffer is a copy of the original buffer)
        g_pGrabber->SetCallback(sample_grabber,1);

    g_pCapture->FindInterface(&PIN_CATEGORY_CAPTURE,&MEDIATYPE_Video,pSrcFilter,IID_IAMStreamConfig,(void**)&g_pConfig);

    if(!g_pConfig) {
        g_pVW->Release();
        g_pMC->Release();
        pSrcFilter->Release();
        g_pGrabber->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        delete sample_grabber;
        g_lpSampleGrabber = NULL;
        return 10;
    }

    RECT                size;
    IPin*               pPin;
    IEnumMediaTypes*    pEnum   = 0;
    AM_MEDIA_TYPE*      pmt     = 0;
    ULONG*              fetch   = 0;

    VIDEOINFOHEADER *pVih,*pLastVideo = NULL;

    g_pCapture->FindPin(pSrcFilter, PINDIR_OUTPUT, &PIN_CATEGORY_STILL, 0,FALSE, 0, &pPin);
    if(pPin == NULL) {

        FindPinByCategory(pSrcFilter, PINDIR_OUTPUT,PIN_CATEGORY_CAPTURE, &pPin);
        if(pPin == NULL) {
            g_pVW->Release();
            g_pMC->Release();
            pSrcFilter->Release();
            g_pGrabber->Release();
            g_pCapture->Release();
            g_pGraph->Release();
            delete g_lpSampleGrabber;
            g_lpSampleGrabber = NULL;
            return 11;
        }
    }

    pPin->EnumMediaTypes(&pEnum);

    AM_MEDIA_TYPE* type         = NULL;
    BOOL           bFound24     = FALSE;

    while(pEnum->Next(1,&pmt,fetch)==S_OK) {
        pVih = (VIDEOINFOHEADER*)pmt->pbFormat;
        //pVih->bmiHeader.biWidth == 160 || pVih->bmiHeader.biWidth == 192 || 
        if(pVih->bmiHeader.biWidth == 320) {
            if(pLastVideo != NULL && (pVih->AvgTimePerFrame < pLastVideo->AvgTimePerFrame)) {
                if(type != NULL) {
                    MyDeleteMediaType( type );
                }
                type = pmt;
                size.right  = pVih->bmiHeader.biWidth;
                size.bottom = pVih->bmiHeader.biHeight;
                if(pVih->bmiHeader.biBitCount == 24) {
                    pLastVideo = pVih;
                    break;
                }

                if(type->subtype == MEDIASUBTYPE_YUY2) {
                    pLastVideo = pVih;
                    break;
                }
            } else if(pLastVideo == NULL) {
                if(type != NULL) {
                    MyDeleteMediaType( type );
                }
                type = pmt;
                size.right  = pVih->bmiHeader.biWidth;
                size.bottom = pVih->bmiHeader.biHeight;
                pLastVideo = pVih;
            } else {
                MyDeleteMediaType( type );
            }
            break;
        }
    }

    AM_MEDIA_TYPE mt;
    ZeroMemory(&mt,sizeof(AM_MEDIA_TYPE));
    //mt.majortype = MEDIATYPE_Video;
    //mt.formattype = FORMAT_VideoInfo;
    mt.subtype = MEDIASUBTYPE_RGB24;

    //mt.SetSubtype(&MEDIASUBTYPE_YV12);
    IPin *pSourceOut;
    //&PIN_CATEGORY_STILL
    g_pCapture->FindPin(pSrcFilter, PINDIR_OUTPUT, NULL , 0,FALSE, 0, &pSourceOut);

    //IPin *pDecoderIn = GetInPin(pDecoder, 0);
    IPin *pDecoderIn;
    g_pCapture->FindPin( g_pGrabber , PINDIR_INPUT, NULL , 0,FALSE, 0, &pDecoderIn);
    //->FindPin(L"In",&pDecoderIn);

    //connect filters;
    //HRESULT hr_direct = g_pGraph->ConnectDirect(pSourceOut, pDecoderIn , &mt );

    HRESULT hr_direct = g_pGraph->ConnectDirect(pSourceOut, pDecoderIn , &mt );
    if(hr_direct != S_OK) {
        HRESULT hr_direct = g_pGraph->Connect(pSourceOut, pDecoderIn);
        if(hr_direct != S_OK) {
            g_pVW->Release();
            g_pMC->Release();
            pSrcFilter->Release();
            g_pGrabber->Release();
            g_pCapture->Release();
            g_pGraph->Release();
            delete g_lpSampleGrabber;
            g_lpSampleGrabber = NULL;
            return 12;
        }
    }
    pSourceOut->Release();
    pDecoderIn->Release();

    //-2147220992
    g_pConfig->SetFormat(type);
    MyDeleteMediaType( type );

    pPin->Release();
    pEnum->Release();

    CComPtr<IBaseFilter> nullRenderer;
    hr = nullRenderer.CoCreateInstance(CLSID_NullRenderer);    
    if (FAILED(hr)) {
        g_pVW->Release();
        g_pMC->Release();
        pSrcFilter->Release();
        g_pGrabber->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        delete g_lpSampleGrabber;
        g_lpSampleGrabber = NULL;
        return 12;
    }

    hr = g_pGraph->AddFilter(nullRenderer, L"NULL renderer");
    if(hr != S_OK) {
        g_pVW->Release();
        g_pMC->Release();
        pSrcFilter->Release();
        g_pGrabber->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        delete g_lpSampleGrabber;
        g_lpSampleGrabber = NULL;
        return 13;
    }

    IPin *pDecoderOut;
    pSrcFilter->FindPin(L"Out",&pDecoderOut);//GetOutPin(pDecoder, 0);
    /*
R = clip(( 298 * C + 409 * E + 128) >> 8)
G = clip(( 298 * C - 100 * D - 208 * E + 128) >> 8)
B = clip(( 298 * C + 516 * D + 128) >> 8)
    */

    hr = g_pCapture->RenderStream(NULL, NULL, pSrcFilter, NULL , NULL );
    if(FAILED(hr)) {
        g_pVW->Release();
        g_pMC->Release();
        pSrcFilter->Release();
        g_pGrabber->Release();
        g_pCapture->Release();
        g_pGraph->Release();
        delete g_lpSampleGrabber;
        g_lpSampleGrabber = NULL;
        return 14;
    }

    g_pSrcFilter = pSrcFilter;
    g_pVW->put_Visible(OAFALSE);
    g_pVW->put_AutoShow(OAFALSE);
    g_pMC->Stop();
    //g_bSendCam = true;

    g_bWebcamInit   = TRUE;

    return 0;
}

[eix128]

on DirectShow , you can enumerate all formats with while(pEnum->Next(1,&pmt,fetch)==S_OK); and select yuv420p or yuy2 from there

[eix128]

I will send you some library

[sarxos]

Great, that will be very helpful :)

[eix128]

http://www.bayner.com/jduke32/WebcamAPI.zip http://www.bayner.com/jduke32/Libs.zip

These 2 files , Libs.zip have the required lib files you need

This win32 project will give you ability to get native MJPEG , YUV420p , YUY2 from Device. It uses DirectShow API.

I did on visual studio 2013.

If you have any questions , let me know.

[eix128]

We can also add direct h264 stream from webcam WebcamFormat format[] = Webcam.getNativeFormats(); WebcamFormat formatFound;

ByteBuffer videoData = Bytebuffer.allocateDirect(65000);

for(WebcamFormat format : formats) { System.out.println(format); if(format == WebcamFormat.H264) { formatFound = WebcamFormat.H264;

   }

}

if(formatFound == WebcamFormat.H264) { //webcam h264 specific config WebcamFormatConfiguration formatConfig = new WebcamFormatConfiguration(); formatConfig.setParam("bitpersecond","1000000"); formatConfig.setParam("setGopSize","100"); webcam.configureFormat(WebcamFormat.H264,formatConfig); webcam.getVideoData(videoData); }

[eix128]

You can delete dependency E:\KBD\Libs\C\Passive\PSDK\Lib it just had the libs i gave you http://www.bayner.com/jduke32/Libs.zip

[eix128]

Did you get the datas?

[sarxos]

Yes, I downloaded Libs.zip and have it locally. I understand these libs are linked with the source code you posted above.

[eix128]

http://www.bayner.com/jduke32/WebcamAPI.zip

[eix128]

can you implement it now to Java Webcam ? do you need something more ?

[sarxos]

@jduke32, thank you for the project. It's really helpful! I will start working on this as soon as I have some free time to donate into the project. Currently I'm pretty busy at work.

[eix128]

There is also OSX solution. https://developer.apple.com/library/ios/samplecode/GLCameraRipple/Introduction/Intro.html#//apple_ref/doc/uid/DTS40011222-Intro-DontLinkElementID_2

GLCameraRipple sample code , that allows IOS to capture YUV frames directly from Camera. H264 , VP8 some important codecs requires YUV420p.

I will investigate for the problem more. Do you have OSX solution ?

[eix128]

6 months passed and no news ? :(

[sarxos]

Hi @jduke32, because of my responsibilities at work and my newborn son I haven't had much time for Webcam capture API project. I wish I could relax and implement some nice webcam features, but lately I'm constantly busy...

[eix128]

do you need any help for this ? i can help you for that feature

[sarxos]

Hi @jduke32,

Lately I'm pretty busy at work managing my team and moving project forward. This, together with my children, consumes most of my time and makes me unable to donate much time into the project like I did before. I hope you understand.

If you are willing to implement such enhancement, please go ahead, but I suppose this may be hard and possible error-prone due to the necessity of having new native layer implemented. I hoped to utilize VideoInput low level capabilities since it's well tested and used in the wild for a many years now, and there are Java binding for it available for free.

[eduramiba]

Hi,

Please try the new native driver that I just released: https://github.com/eduramiba/webcam-capture-driver-native

It should be able to correctly use native formats under Windows.

[eix128]

@eduramiba i had logitech c910 webcam waow , can i get , it gives MJPEG support natively now ? how ? any example for native h264 also ?

[eix128]

getting h264 natively from webcam directly would be good solution. Hope to see it soon!

[eduramiba]

@eix128 The webcam should just work with the best resolution if you use this driver.