Audio Video overlay.

[cathalmccabe]

In the Audio Video overlay, I can't get the HDMI to pass through to the VGA. I'm using the Documentation workbook, 7_audio_video_overlay.ipynb, to test. Does this work for anyone else? This is the same issue I had a while ago, but as this overlay was being updated, the issue was closed. It seems that the issue is still not resolved.

Also, there is a crackle with the audio pass through. It sounds like it may be an issue with the sampling frequency or the read/write rate from input to output.

[cathalmccabe]

In addition, what resolutions are supported by the HDMI in? I just tried my laptop, and hdmi_start could not initialize with an input of 1280x720. It worked with 1600x900. I connected a HDMI output from another FPGA board which should be 1080i but it is detected as 1920 x 540, and the input image is squashed. Looks like it is not handling the interlaced and expects progressive?

I tried the grayscale conversion on the 1600x900 and it took ~100s. Sobel filter took ~200s. While there is a caveat at the top of the notebook that the PS is doing the processing and will take some time, this is far to slow. The way the memory is used is obviously the issue, but I don't understand how much of this is due to the way we have implemented the design and how Python is managing the arrays.

I think we need to optimize this, or remove it.

[schelleg]

My board is working with latest release with following tests:

audio - all py.test's passing - sound is good. video - HDMI input at 1280x720 passing. VGA py.test's passing.

Agreed that sobel filtering is slow, but I think that might be a good thing to highlight programmable logic video pipeline we'll be building.

[cathalmccabe]

Can you actually see the HDMI image on the VGA display?

I think the image processing is so slow that it just reflects really poorly on our platform/implementation. Is the implementation just too Naive? I don't know the overlay or the Python implementation to determine this.

Arduino etc seems to be able to do this in a few seconds.

I know we expect to loose some performance with Python, but this is maybe 50x or 100x slower than an inferior platform.

For ref, our friend Derek Molloy has a tutorial/video on arduino and opencv. Around 26 mins seems to show edge detection in a few seconds: http://derekmolloy.ie/beaglebone/beaglebone-video-capture-and-image-processing-on-embedded-linux-using-opencv/

[schelleg]

Yes, I can see the HDMI image on the VGA display - I am running through the audio_sobel code snippets to see it.

We definitely did lose a lot of performance when we moved to CPython - Linux and Python object structure are pretty burdensome (virtual addressing, Python object overhead). I'm sure we can speed it up ...

[cathalmccabe]

Ok. Hdmi -> vga doesn't work for me. How much of an issue is this?

Yes, must be a way to get some speed up. Can anyone take this on?

[schelleg]

@yunqu - can you verify you can read from HDMI to VGA?

On Wed, May 11, 2016 at 1:36 PM, cathalmccabe notifications@github.com wrote:

Ok. Hdmi -> vga doesn't work for me. How much of an issue is this?

Yes, must be a way to get some speed up. Can anyone take this on?

— You are receiving this because you were assigned. Reply to this email directly or view it on GitHub https://github.com/Xilinx/Pynq/issues/73#issuecomment-218566245

Graham

[yunqu]

@schelleg I do not have the HDMI interface on my laptop. Patrick has the cable adapter from micro HDMI to HDMI, but I don't have it now.

[schelleg]

@yunqu - when possible, verify HDMI ... just need to understand whether I have the lucky VGA or Cathal is seeing what others will experience.

[schelleg]

I just checked in audiovideo_sobel notebook that is much faster:

gray scale: 60 seconds --> 20 seconds sobel: 180 seconds --> 30 seconds

we'll still get added verification from Rock on HDMI issue Cathal is seeing

[yunqu]

@schelleg

Hi , I have just tested the audiovideo overlay. The video works for me. I (1) tested the new sobel filter notebook Graham uploaded, and (2) passed the pytest.

Graham, a small thing is, when HDMI is initiating, it would be better to use sleep(20), since sleep(10) sometimes does not work for me...

hdmi=HDMI('in') sleep(20) hdmi.start() sleep(1)

[schelleg]

okay, we can revert back to 20 ... but that is a long time for synchronization ... it takes me 5 seconds. Is there something unique about your HDMI cable or laptop?

On Thu, May 12, 2016 at 5:20 PM, Yun Rock Qu notifications@github.com wrote:

@schelleg https://github.com/schelleg

Hi , I have just tested. The video works for me. I (1) tested the new sobel filter notebook Graham uploaded, and (2) passed the pytest.

[image: image] https://cloud.githubusercontent.com/assets/14114006/15233387/21d56050-185d-11e6-93bf-35f3e73bf386.png

Graham, a small thing is, when HDMI is initiating, it would be better to use sleep(20), since sleep(10) sometimes does not work for me...

hdmi=HDMI('in') sleep(20) hdmi.start() sleep(1)

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Xilinx/Pynq/issues/73#issuecomment-218913081

Graham

[yunqu]

@schelleg ,

I think Cathal is talking about the audiovideo overlay notebook in Getting started folder. I tried that one. I am not able to see a smooth VGA display from HDMI input for that notebook.

All I see is a black screen after I started VGA and HDMI.

[yunqu]

I made a few changes on audio_video_overlay notebook. Basically I added some delays, switched the overlay back to pmod.bit (otherwise the system crashes sometimes), saved the images into /Getting_Started/images. @cathalmccabe , feel free to change it if you think there is any issue with that.

[cathalmccabe]

Agree about sleeping for 20 seconds. It takes a variable time for the HDMI to sync. Rather than go for worst case (20 seconds) I suggest a small time, and if it fails, put a note underneath to tell the user to rerun the command. I did this for one of the HDMI notebooks.

I just tried the audiovideo_sobel.ipynb with 1080p hdmi took me 139.66871637999998 to do the grayscale.

import time
last_time = time.time()
for y in range(0, height):
    for x in range(0, width):
        gray = round((0.299*frame[x,y][0]) + (0.587*frame[x,y][1]) 
                      + (0.114*frame[x,y][2]))
        frame[x,y] = (gray, gray, gray)
now = time.time() - last_time
print(now)
gray_img_path = '/home/xpp/jupyter_notebooks/Examples/data/gray.jpg'
frame.save_as_jpeg(gray_img_path)
Image(filename=gray_img_path)

@schelleg what resolution were you using? Perhaps there is a bigger overhead when we go over a threshold for an array size?

I also tried HDMI -> VGA at home to test a different set of HDMI/VGA. I used my TV box (HDMI out), and TV (VGA input) and I can see the HDMI frame in the notebook, but no VGA.

[schelleg]

That is the old code. Can you update for repo and try getting times again? I was doing 1280x720.

Sent from my iPad

On May 13, 2016, at 5:22 AM, cathalmccabe notifications@github.com wrote:

Agree about sleeping for 20 seconds. It takes a variable time for the HDMI to sync. Rather than go for worst case (20 seconds) I suggest a small time, and if it fails, put a note underneath to tell the user to rerun the command. I did this for one of the HDMI notebooks.

I just tried the audiovideo_sobel.ipynb with 1080p hdmi took me 139.66871637999998 to do the grayscale.

import time last_time = time.time() for y in range(0, height): for x in range(0, width): gray = round((0.299_frame[x,y][0]) + (0.587_frame[x,y][1])

• (0.114*frame[x,y][2])) frame[x,y] = (gray, gray, gray) now = time.time() - last_time print(now) gray_img_path = '/home/xpp/jupyter_notebooks/Examples/data/gray.jpg' frame.save_as_jpeg(gray_img_path) Image(filename=gray_img_path) @schelleg what resolution were you using? Perhaps there is a bigger overhead when we go over a threshold for an array size?

I also tried HDMI -> VGA at home to test a different set of HDMI/VGA. I used my TV box (HDMI out), and TV (VGA input) and I can see the HDMI frame in the notebook, but no VGA.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[schelleg]

I'm going to close this issue and open new ones that are more specific to various issues raised.