Android FFmpeg video rotation

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Record video with android
2. Rotate by 90 or 270 deg
3.

What is the expected output? What do you see instead?

Look at original.mp4 and modified.mp4  both of them are in H264 see code 
attached, modified should take all the width, i'm pretty sure this is linked to 
the way Android manage bitmap using ARGB_8888 but at this point i'm not sure if 
my problem is in my own code or the way opencv/ffmpeg use iplimage to rotate 
it. 

What version of the product are you using? On what operating system?
0.6, Android 4.2 on a Galaxy Nexus

Please provide any additional information below.

        FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(videoFile);     
        grabber.start();

        FrameRecorder recorder = new FFmpegFrameRecorder("ORIGINAL_FILE.mp4",grabber.getImageHeight() ,getImageWidth() , grabber.getAudioChannels());
        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
        recorder.setFrameRate(grabber.getFrameRate()<30.0?grabber.getFrameRate():30.0);
        recorder.setSampleFormat(grabber.getSampleFormat());
        Log.d("onr", "pixel format"+recorder.getPixelFormat());
        //grabber.getPixelFormat return 3 which is format AV_PIX_FMT_BGR24
        //H264 supposedly support only AV_PIX_FMT_YUV420P/AV_PIX_FMT_YUV422P
        recorder.setSampleRate(grabber.getSampleRate());
    recorder.start();
        Frame frame;
            while ((frame = grabber.grabFrame()) != null) {
                if(frame.image!=null){
                IplImage rotateImage = rotate(frame.image, 270);
                    frame.image=rotateImage;
                    recorder.record(rotateImage);
                    opencv_core.cvReleaseImage(rotateImage);
                }else{
                    recorder.record(frame);
                }
                mProgressDialog.setProgress((int)currentFrame);
            }
            recorder.stop();
            grabber.stop();

    public static IplImage rotate(IplImage image, double angle) {        
        IplImage copy = opencv_core.cvCloneImage(image);

        IplImage rotatedImage = opencv_core.cvCreateImage(opencv_core.cvGetSize(copy), copy.depth(), copy.nChannels()); 
        CvMat mapMatrix = opencv_core.cvCreateMat( 2, 3, opencv_core.CV_32FC1 );

        //Define Mid Point
        CvPoint2D32f centerPoint = new CvPoint2D32f();
        centerPoint.x(copy.width()/2);
        centerPoint.y(copy.height()/2);

        //Get Rotational Matrix
        opencv_imgproc.cv2DRotationMatrix(centerPoint, angle, 1.0, mapMatrix);
        //opencv_core.cvReleaseImage(copy);

        //Rotate the Image
        opencv_imgproc.cvWarpAffine(copy, rotatedImage, mapMatrix, opencv_imgproc.CV_INTER_CUBIC +  opencv_imgproc.CV_WARP_FILL_OUTLIERS, opencv_core.cvScalarAll(170));
        opencv_core.cvReleaseImage(copy);
        opencv_core.cvReleaseMat(mapMatrix);        
        return rotatedImage;
    } 

Original issue reported on code.google.com by math...@gmail.com on 21 Oct 2013 at 9:44

Attachments:

• original.mp4
• modified.mp4

[GoogleCodeExporter]

If you don't try to rotate them, do the images get written to the video file 
properly?

Original comment by samuel.a...@gmail.com on 3 Nov 2013 at 1:55

[GoogleCodeExporter]

Yes they are written properly and the encoding is respected (well as planned)

Btw, great job you did with javacv

Original comment by math...@gmail.com on 3 Nov 2013 at 1:57

[GoogleCodeExporter]

Thanks! I guess I'll need to look a bit more into that, but it looks to me like 
an incorrect assumption of how the OpenCV functions work. To figure this out, 
we could try to change the matrix in random ways, for example, to understand 
how it affects the result...

Original comment by samuel.a...@gmail.com on 3 Nov 2013 at 2:02

[GoogleCodeExporter]

That was my first try also, at first i changed 90 deg to 85 just to see if it 
was linked to the fact of 90 deg, it is not. I'll be glad to help, let me know!

Original comment by math...@gmail.com on 3 Nov 2013 at 2:10

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Try setting the size to the oposite of the original image, something like: 

CvSize size = new CvSize(copy.height(), copy.width());
IplImage rotatedImage = opencv_core.cvCreateImage(size, copy.depth(), 
copy.nChannels());

And set the pivot x and y to height/2

// 320x240
CvPoint2D32f centerPoint = new CvPoint2D32f();
centerPoint.x(height/2);
centerPoint.y(height/2);

Original comment by jo...@sparklingzoo.se on 11 Nov 2013 at 3:30

[GoogleCodeExporter]

Issue 380 has been merged into this issue.

Original comment by samuel.a...@gmail.com on 17 Nov 2013 at 8:09

[GoogleCodeExporter]

Issue 382 has been merged into this issue.

Original comment by samuel.a...@gmail.com on 17 Nov 2013 at 8:18

[GoogleCodeExporter]

Maybe to do a follow up for jo...@sparklingzoo.se. It did not work, honestly 
the problem is in the way pixel is managed in Android, the effect is the same 
as if you rotate a bitmap taken from the camera and don't use the good "type" 
of pixel to enable the rotation

Original comment by math...@skex.ca on 17 Nov 2013 at 3:26

[GoogleCodeExporter]

Well I was able to flip the video you provided. Using this. What result do you 
get?

    private void FlipVideo(String original, String flip) throws FrameGrabber.Exception, FrameRecorder.Exception {

        File f = new File(original);
        FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(f.getPath());
        grabber.start();

        FrameRecorder recorder = new FFmpegFrameRecorder(flip,grabber.getImageHeight() ,grabber.getImageWidth() , grabber.getAudioChannels());
        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
        recorder.setFrameRate(grabber.getFrameRate()<30.0?grabber.getFrameRate():30.0);
        recorder.setSampleFormat(grabber.getSampleFormat());

        recorder.setSampleRate(grabber.getSampleRate());
        recorder.start();
        Frame frame;
        mProgress.setMax(grabber.getLengthInFrames());
        while ((frame = grabber.grabFrame()) != null) {
            if(frame.image!=null){
                IplImage rotateImage = rotate(frame.image, 90);
                frame.image=rotateImage;
                recorder.record(rotateImage);
                opencv_core.cvReleaseImage(rotateImage);
            }else{
                recorder.record(frame);
            }
            mProgress.setProgress((int)grabber.getFrameNumber());
        }
        recorder.stop();
        grabber.stop();
    }

    public static IplImage rotate(IplImage image, double angle) {
        IplImage copy = opencv_core.cvCloneImage(image);

        double radians = Math.toRadians(angle);
        double sin = Math.abs(Math.sin(radians));
        double cos = Math.abs(Math.cos(radians));

        int newWidth = (int) (copy.width() * cos + copy.height() * sin);
        int newHeight = (int) (copy.width() * sin + copy.height() * cos);

        int[] newWidthHeight = {newWidth, newHeight};

        CvSize size = new CvSize(copy.height(), copy.width());

        IplImage rotatedImage = opencv_core.cvCreateImage(size, copy.depth(), copy.nChannels());
        CvMat mapMatrix = opencv_core.cvCreateMat( 2, 3, opencv_core.CV_32FC1 );

        //Define Mid Point
        CvPoint2D32f centerPoint = new CvPoint2D32f();
        centerPoint.x(copy.width()/2);
        centerPoint.y(copy.width()/2);

        //Get Rotational Matrix
        opencv_imgproc.cv2DRotationMatrix(centerPoint, angle, 1, mapMatrix);

        opencv_imgproc.cvWarpAffine(copy, rotatedImage, mapMatrix, opencv_imgproc.CV_INTER_CUBIC +  opencv_imgproc.CV_WARP_FILL_OUTLIERS, opencv_core.cvScalarAll(170));
        opencv_core.cvReleaseImage(copy);
        opencv_core.cvReleaseMat(mapMatrix);
        return rotatedImage;
    }

Original comment by jo...@sparklingzoo.se on 6 Dec 2013 at 12:29

[GoogleCodeExporter]

Does anyone else get a green video when trying the rotate method above?

Thanks,
K

Original comment by klomm...@gmail.com on 29 Jan 2014 at 7:25

[GoogleCodeExporter]

klomm...@gmail.com i get

Original comment by ov.sa...@gmail.com on 12 Feb 2014 at 9:00

[GoogleCodeExporter]

Does FFmpeg output any warning on the console? We may not be able to access 
easily those messages on Android, so you might want to try to debug that on the 
desktop.

Original comment by samuel.a...@gmail.com on 12 Feb 2014 at 9:14

[GoogleCodeExporter]

My guess is that this is due to not properly set the height and width after 
frame rotation
PS sorry for my bad english 

Original comment by ov.sa...@gmail.com on 12 Feb 2014 at 9:19

[GoogleCodeExporter]

Help pls

Original comment by ov.sa...@gmail.com on 12 Feb 2014 at 9:56

[GoogleCodeExporter]

Yes, try a different image width. It's quite possible that the codec you want 
to use doesn't support the width you are trying to use, and some codecs, 
instead of throwing an error, simply encode green images.

Original comment by samuel.a...@gmail.com on 12 Feb 2014 at 11:57

[GoogleCodeExporter]

Thks. I'm using h264 and 640x480 frame sizes. It's very common combination 
isn't it? 

Original comment by ov.sa...@gmail.com on 12 Feb 2014 at 12:01

[GoogleCodeExporter]

Yes, that should be fine.

If anyone could send the log messages of what they get from FFmpeg, we might be 
able to understand a bit better what is going on here.

Original comment by samuel.a...@gmail.com on 12 Feb 2014 at 12:08

[GoogleCodeExporter]

that's just the problem that ffmpeg does not write anything

Original comment by ov.sa...@gmail.com on 12 Feb 2014 at 12:16

[GoogleCodeExporter]

Yes, so we should try it out on the desktop with Java SE, first. This way, we 
can see easily see all the warning and error messages on the console.

Original comment by samuel.a...@gmail.com on 15 Feb 2014 at 1:52

[GoogleCodeExporter]

Hi samuel im getting the same problem while trying to merge a video recorded 
from camera with video created from combining images and audio...the recorded 
video gets rotated automatically when merged with other videos,,,

Original comment by kalee...@gmail.com on 1 Apr 2014 at 8:59

[GoogleCodeExporter]

@kaleemim Does the code from comment #10 works for you or not?

Original comment by samuel.a...@gmail.com on 1 Apr 2014 at 1:44

[GoogleCodeExporter]

Yes it partially works,more than half of the video is blank(showing gray color 
strip) and remaining part is ok,,i tried changing the codec but cdnt fnd any 
solution.I also tried above code with different height and width,but this time 
the rezsultd video is showing black strip instead.and this is case with only 
the video recorded from camera.other videos when merged dsnt behave like this. 

Original comment by kalee...@gmail.com on 3 Apr 2014 at 4:00

[GoogleCodeExporter]

Hi Samuel,One more thing to mention is dat when I try to rotate it by 90 
degrees it works fine,now the video is upside down, but when I try it with 270 
degrees den the grey strip appears..

Original comment by kalee...@gmail.com on 3 Apr 2014 at 5:24

[GoogleCodeExporter]

Hi everyone! 
All kinds of strips and other interferences are result of incorrected 
widht/height proportion of rotated frame. 

Slampy.

Original comment by ov.sa...@gmail.com on 3 Apr 2014 at 8:11

[GoogleCodeExporter]

Thanx for your reply,,,
I got the desired result when i replaced this line from math...@gmail.com 's 
code

 opencv_imgproc.cvWarpAffine(copy, rotatedImage, mapMatrix, opencv_imgproc.CV_INTER_CUBIC +  opencv_imgproc.CV_WARP_FILL_OUTLIERS, opencv_core.cvScalarAll(170));

with this
opencv_imgproc.cvWarpAffine(copy, rotatedImage, mapMatrix,
                opencv_imgproc.CV_INTER_CUBIC
                        + opencv_imgproc.CV_GAUSSIAN,
                opencv_core.cvScalarAll(170));

Original comment by kalee...@gmail.com on 3 Apr 2014 at 10:44

[GoogleCodeExporter]

Issue 473 has been merged into this issue.

Original comment by samuel.a...@gmail.com on 24 May 2014 at 4:43

[GoogleCodeExporter]

Hi Samuel, Thanks for Javacv library at first. Right now, I am also facing the 
same issue as others did, the greenish output. The method mentioned above and 
also the method below,

    IplImage myImage = cvCloneImage(image);
        if (myImage.width() > 0 && myImage.height() > 0) {
            CvMat mat = new CvMat();
            cvGetMat(myImage, mat, null, 1);
            IplImage trans = cvCreateImage(cvSize(mat.rows(), mat.cols()), myImage.depth(), myImage.nChannels());
            cvTranspose(myImage, trans);
            cvTranspose(myImage, trans);
            cvFlip(trans, trans, 1);
            myImage = null;
            myImage = cvCreateImage(cvSize(trans.width(), trans.height()), trans.depth(), trans.nChannels());
            myImage = cvCloneImage(trans);
        }

both gives the same greenish output. My video size is 640 * 480. Is there any 
other way to rotate the IplImage stream? I am trying to do 180 degree rotation.

Original comment by intrepidkarthi@gmail.com on 3 Jun 2014 at 11:47

[GoogleCodeExporter]

@intrepidkarthi Could you please post the output of FFmpeg? It could help 
diagnose the problem, thank you.

Original comment by samuel.a...@gmail.com on 3 Jun 2014 at 12:08

[GoogleCodeExporter]

Hi Samuel,
   I tried to merge multiple videos through FrameGrabber. The result video is generated with some frames missing.
The code snippet is 
try
        {
            //get files
            File pF = new File(OUTPUT_DIR );
            File files[] = pF.listFiles();

            Log.d("amlan", Integer.toString(files.length));

            if(files.length <= 1)
                return;

            List<FrameGrabber> fgs = new ArrayList<FrameGrabber>();
            for(File f : files)
            {
                FrameGrabber grabber = new FFmpegFrameGrabber(f);
                grabber.start();
                fgs.add(grabber);
            }

            FrameRecorder recorder = new FFmpegFrameRecorder(OUTPUT_DIR + File.separator + "VIDEO_COMBINED_TEST.mp4",
                    fgs.get(0).getImageWidth(), fgs.get(0).getImageHeight() , 1);
//          recorder.setFormat("mp4");
//          recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
            recorder.setAudioChannels(1);
//          recorder.setPixelFormat(avcodec.AV_CODEC_ID_H264);

            RecorderParameters recorderParameters = new RecorderParameters();
            recorder.setFormat(recorderParameters .getVideoOutputFormat());
            recorder.setSampleRate(recorderParameters.getAudioSamplingRate());
            recorder.setFrameRate(recorderParameters.getVideoFrameRate());
            recorder.setVideoCodec(recorderParameters.getVideoCodec());
            recorder.setVideoQuality(0/*recorderParameters.getVideoQuality()*/); 
            recorder.setAudioQuality(recorderParameters.getVideoQuality());
            recorder.setAudioCodec(recorderParameters.getAudioCodec());
            recorder.setVideoBitrate(recorderParameters.getVideoBitrate());
            recorder.setAudioBitrate(128000/*recorderParameters.getAudioBitrate()*/);

            recorder.start();
            Frame frame;

            for(FrameGrabber fg : fgs)
            {
                while ((frame = fg.grabFrame()) != null) {
                    recorder.record(frame);
                }
                fg.stop();
            }

            recorder.stop();
        }
        catch(Exception ex)
        {
            ex.printStackTrace();
        }

Thanks
Amlan.

Original comment by amlan....@gmail.com on 22 Aug 2014 at 2:02

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

@Override
    public void onPreviewFrame(byte[] data, Camera camera) {  
         //IplImage newImage = cvCreateImage(cvGetSize(yuvIplimage), IPL_DEPTH_8U, 1);   
         if (recording) {    
             videoTimestamp = 1000 * (System.currentTimeMillis() - startTime);                
             yuvimage = IplImage.create(imageWidth, imageHeight * 3 / 2, IPL_DEPTH_8U,1); 
             yuvimage.getByteBuffer().put(data); 

             rgbimage = IplImage.create(imageWidth, imageHeight, IPL_DEPTH_8U, 3); 
             opencv_imgproc.cvCvtColor(yuvimage, rgbimage, opencv_imgproc.CV_YUV2BGR_NV21);      

             IplImage rotateimage=null;
                  try {
                        recorder.setTimestamp(videoTimestamp);   
                        int rot=0;
                        switch (degrees) {
                        case 0:
                            rot =1;
                            rotateimage=rotate(rgbimage,rot);
                        break;
                        case 180:
                            rot = -1;
                            rotateimage=rotate(rgbimage,rot);
                            break;                     
                        default:
                            rotateimage=rgbimage;
                    }                      
                        recorder.record(rotateimage);

                  } catch (FFmpegFrameRecorder.Exception e) {
                     e.printStackTrace();
                  }  
            }

        }
IplImage rotate(IplImage IplSrc,int angle) {
        IplImage img= IplImage.create(IplSrc.height(), IplSrc.width(), IplSrc.depth(), IplSrc.nChannels());
        cvTranspose(IplSrc, img);
        cvFlip(img, img, angle);        
        return img;
        }    
    }

after many searches this works for me on android note 3 (türkay biliyor)

Original comment by tbili...@gmail.com on 19 Sep 2014 at 7:06

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Can I crop the video to a square one?

Original comment by amlan....@gmail.com on 19 Sep 2014 at 9:01

[GoogleCodeExporter]

@tbiliyor Thanks for confirming that everything works fine! If you'd like to 
make a sample out of that to help others, let me know, and we'll add it to the 
"samples" folder, thanks!

Original comment by samuel.a...@gmail.com on 20 Sep 2014 at 12:43

• Changed state: Done
• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

Thanks you @tbiliyor, Finally i figured out a solution. But i dont understand 
this line: 
yuvimage = IplImage.create(imageWidth, imageHeight * 3 / 2, IPL_DEPTH_8U,1); 
Why do imageHeight * 3 / 2 make everything work. 

Original comment by top...@gmail.com on 26 Feb 2015 at 9:45