search

espressif / esp32-camera

Apache License 2.0
1.79k stars 622 forks source link

Any support for ov5640 possible? #18

Closed searchingforcode closed 4 years ago

searchingforcode commented 5 years ago

Hi is it possible to integrate ov5640 driver? I found drivers of ov5640 for linux. Can esp32 process ov5640 images? Thanks

Niek commented 5 years ago

+1 for this

I know there is OV3660 support in the corresponding branch (https://github.com/espressif/esp32-camera/tree/ov3660-support), but the OV5640 is more widely available on Taobao/AliExpress and supports 30fps @ 1080p resolution.

Linux driver for reference: https://github.com/torvalds/linux/blob/master/drivers/media/i2c/ov5640.c

ArduCAM code: https://github.com/ArduCAM/Arduino/blob/master/ArduCAM/ov5640_regs.h

me-no-dev commented 5 years ago

don't believe frame per second numbers like this :) while the camera might be able to do it, it does not mean that the ESP can handle it :) this is mainly where we hit a wall. Also Adding a new driver takes time and effort that should have some sort of positive outcome in therms of business and OV5640 does not look like a prime time candidate. If any of you is willing to put some time into it, I can guide you through getting the driver to work.

Plus, I think we will have enough ov3660 cameras on the market soon :P

Niek commented 5 years ago

OK, that makes sense - looking forward to see those OV3660 sensors/updated ESP-EYE board đź‘Ť

searchingforcode commented 5 years ago

Hi great. Waiting for the ov3660 modules. And more thing, there is an micro-rtsp implementation for esp32 on platformio. Can it be ported/implemented to esp-idf? Thanks a lot :)

Niek commented 5 years ago

@searchingforcode ADF will (very?) soon have SIP/VoIP added: https://github.com/espressif/esp-adf/issues/106 / https://github.com/espressif/esp-adf/issues/81 - I guess it will be possible to add a MJPEG video stream in there as well, which will be more powerful than RTSP.

me-no-dev commented 5 years ago

MJPEG we already do :) example is in ESP-WHO repository.

Niek commented 5 years ago

I am aware of that, I meant MJPEG + (microphone) audio combined over SIP so it's possible to create a true "webcam" video+audio stream.

me-no-dev commented 5 years ago

Out SIP engineer has left the company, but I will poke arround and see if I can finish it. That said, SIP uses h263 and h264 for video transmission. Voice options are a couple, but all feasible. h263/h264 on ESP32 .... not a thing yet and maybe never a thing :)

searchingforcode commented 5 years ago

I dont think h263/h264 is feasible on esp32. Any idea about streaming to a cloud of this mjpeg streaming?

Niek commented 5 years ago

Out SIP engineer has left the company, but I will poke arround and see if I can finish it. That said, SIP uses h263 and h264 for video transmission. Voice options are a couple, but all feasible. h263/h264 on ESP32 .... not a thing yet and maybe never a thing :)

I thought SIP allowed different video codecs such as MJPEG as well, but I might be mistaken. Anyway, having the SIP support in ADF would be awesome.

me-no-dev commented 5 years ago

for actual streaming to a server, MJPEG should be encapsulated in some sort of video container in order to keep the timing between frames correct, else you will watch a video with speed comparable to the speed of the connection to your server :)

h263/h264 and other good video compressions work by having multiple frames at the same time and transmitting the differences (simplified). The ESP32 does not have enough memory or speed to acquire and keep a few frames in RGB format and then calculate and transmit their differences. Maybe some really low res thing can be done... but not 100% sure.

me-no-dev commented 5 years ago

@Niek honestly, I think that SIP should be it's own component. We will not use the compressions used by ADF, because 90% of SIPs out there use G711 audio compression (100% doable on ESP32).

asmalldev commented 5 years ago

Hi @me-no-dev are there any higher resolution cameras that may be better candidates than OV5640? I am looking to add more resolution and may have time to work on a driver :)

me-no-dev commented 5 years ago

@asmalldev honestly I am not sure if it makes sense to look at higher res cameras than 5MP. We do not have enough RAM available :) The sensor must support JPEG compression or there is no chance at all even for 3MP sensors :)

asmalldev commented 5 years ago

@me-no-dev thanks for the input :) If the limiting factor is the RAM space, would it be possible to interface with higher resolution cameras by adding more SPI/PSRAM to the hardware? I think this could be interesting for projects that need high resolution + low power but don't have frame rate requirements/ only need to take still images.

me-no-dev commented 5 years ago

You are limited to 4MB PSRAM access at a time, so that is the maximum.

asmalldev commented 5 years ago

Ah right, bank switching stuff is needed to access the higher memory. Thanks for the info, @me-no-dev I think I will try writing a driver for OV5640 to get 5MP w/ JPEG compression. If you know of any other 5MP sensors that would be good candidates, I'd love to hear about them :)

xxxajk commented 5 years ago

I would be happier with any camera doing autofocus myself. I only need VGA resolution anyway. I have a bunch OV5648 I pulled from dead tablets, but it has a different connector. I hope to be able to make my own board for it soon. Another thought on using higher-res cameras, why not send the raw data and use javascript on the PC side to process it? :-) Also, it seems that if the ESP is sending data there is corruption, and I get streaks in the image. If I delay 500ms before each frame, these don't happen. 2FPS is probabbly OK for my project too.

me-no-dev commented 5 years ago

Autofocus is a feature of the camera itself. If it can do it and provides control, it can be added. You can use "raw" data if you want (check the BMP sending in the README) but that means many times more data that needs to go through the network. Therefore it's not good for streaming. Please show the corruption in an image and post your camera config.

xxxajk commented 5 years ago

My setup is: CameraWebServer Arduino example. ESP32-CAM board with OV2640 fixed focus camera.

I hope that the bare-bones SDK will work better, as it just could be an outdated version in Arduino-ESP32. I've recently downloaded the SDK, etc and will try the bare metal example as this issue is possibly already solved.

Project goal is to use a couple of these for vision. It will be nice to have them wireless v.s. adding more cabling.

The main processing and computation will be using OpenCV on an Android tablet or Linux PC. I haven't decided which yet.

xxxajk commented 5 years ago

Adding example screen captures from tablet. Screenshot_2019-05-27-19-07-53 Screenshot_2019-05-27-19-07-30 Screenshot_2019-05-27-19-02-27 Screenshot_2019-05-27-18-47-36

xxxajk commented 5 years ago

Happens on the bare-metal variant as well, providing I turn off the debugging. Has to be a DMA conflict.

me-no-dev commented 5 years ago

What XCLK frequency are you setting?

xxxajk commented 5 years ago

20MHz. Also tried 10, 12, and a few others. Result is the same. From what I can tell, somehow, there is a race condition or DMA conflict. I wonder if it might be a good idea to pause I2S between getting the FB and returning it back to the pool? According to the ESP32 RM, you shouldn't try to DMA and operate on a bank at the same time if you want 100% speed. Since I tried much lower speeds, that shouldn't be happening either way. Checked the power supply and voltages on my 'scope, and they are solid. No power problems that could perturb the camera module, but... Seems to happen only if there is WiFi traffic. I am about to try tests using high-speed serial. If that works, then I could use an ESP8266 to do the actual transmitting. Would be awesome if it worked out-of the box though.

me-no-dev commented 5 years ago

I have not seen any issues with 20MHz XCLK and OV2640. Have tested quite a few samples. Some were bad, but presented in another way. You can set the frame buffers to 1, which will start I2S and DMA when you request a frame and then stop it once done. That will limit the FPS by half though.

xxxajk commented 5 years ago

Not worried about FPS at this point. I'll try that and let you know how it works out in a few minutes.

xxxajk commented 5 years ago

Still happens with a singular fb. :-( Any other good ideas?

me-no-dev commented 5 years ago

try another sensor?

xxxajk commented 5 years ago

Have 4 of them, tried 2, they both the same thing. :-/ I'll try the other two.

xxxajk commented 5 years ago

Tried 4 cams, on 4 boards, all showing same issue.

Niek commented 5 years ago

@xxxajk I had similar issues on the ESP32-CAM boards with some power input sources. If you're using that board, make sure to use a 5V 2A input to avoid image distortions.

me-no-dev commented 5 years ago

Since you say that WiFi is causing it, maybe it is power...

xxxajk commented 5 years ago

I've got a 2A supply.... I'll try it.

xxxajk commented 5 years ago

No difference using a different 5V supply. I'll rig up a 3.3 supply, if you think that would help.

jjbubka commented 5 years ago

I am aware that you need that the only task that must be running in the core 1 is the dma_filter_task of camera.c, every other task related to wifi must be in core 0. That way you will prevent of skipping some lines of the dma being processed. Maybe that is your issue. Best Regards

xxxajk commented 5 years ago

~/.arduino15/packages/esp32/hardware/esp32/1.0.2/tools/sdk at sdkconfig | grep CORE | grep '=y' CONFIG_ARDUINO_RUN_CORE1=y CONFIG_ARDUINO_EVENT_RUN_CORE1=y CONFIG_ARDUINO_UDP_RUN_CORE1=y CONFIG_BTDM_CONTROLLER_PINNED_TO_CORE_0=y CONFIG_BLUEDROID_PINNED_TO_CORE_0=y CONFIG_ESP32_ENABLE_COREDUMP_TO_NONE=y CONFIG_ESP32_WIFI_TASK_PINNED_TO_CORE_0=y CONFIG_CAMERA_CORE1=y CONFIG_FREERTOS_CORETIMER_0=y

xxxajk commented 5 years ago

Possibly crap cam module? IMG_20190528_044014

me-no-dev commented 5 years ago

are they all the same batch? I've stumbled on bad batches before (and we returned them to the seller). In any way... OV2640 is EOL sensor and no longer supported/manufactured by OmniVision. Look for OV3660 (which we support and is not EOL). There should be good ones out there as opposed to "whatever is left" in the case of 2640.

xxxajk commented 5 years ago

Been digging everywhere, and apparently either I'm stupid, or google is stupid, or both. Got a source for these? Adjustable focus, auto focus preferred. Fixed focus is useless because I need to focus from about 3cm-12cm distance. Most auto focus units can actually handle the close-up. The ones I have can autofocus @ 6.5, which is perfect for most of what I need to do.

me-no-dev commented 5 years ago

ov2640 auto focus? what is that? None of those have it... the most basic explanation is that focus is done through optics and not through the sensor ;) You need a sensor that supports such optics and can control them.

xxxajk commented 5 years ago

I mean for OV3660.

me-no-dev commented 5 years ago

same story :) focus is a thing that is done with lenses. You can not focus the sensor itself (this is true for every camera sensor). Some sensors have IO that is used to connect to lenses that have motors which allow them to change focus and then the software detects wether the focus is good or not.

xxxajk commented 5 years ago

Ok, well I am seeing some ov2640, but so far, only fisheye... way too much for them too. I've ran it in gray scale mode, and it isn't missing anything, or if it is corrupting a pixel, I'm not seeing it. Going to try single buffer + color and see what happens. If that works, even if it is slower, I'm all good with that.

xxxajk commented 5 years ago

More testing shows that it happens, but that it is a rare occasion. Funny how the esp32 can't keep up with a 25Hz freq with a 4.64ms low pulse... Should be able to do that with ease.

xxxajk commented 5 years ago

One more thing, bit busy here, but what I should do is output a pulse on an unused pin to see if we really are missing the vertical pulse. Not sure how to do that for the arduino side, but...

xxxajk commented 5 years ago

Seems to be missing the horizontal pulses. They are visible on the scope, so it has to be something else going on. I'm not entirely sure how DMA works on ESP32, such as arbitration, etc, or if something else is getting in the way. One thing I can validate is that it isn't the camera module. I only say this because if I delay a long while between frames, the pictures stream fine. If I ensure that there is no WiFi activity, it is fine too. I still need to have my scope do missing pulse detection, to be totally sure. Meanwhile I'm going to go another step farther and use the runt feature on the 'scope and see if the signal is indeed clean.

xxxajk commented 5 years ago

Got in some new modules form a totally different vendor (Arducam), same problem. So, something isn't right here. It is either missing the IRQ, or there is DMA contention.

xxxajk commented 5 years ago

Good news, found the problem! Seems the ESP32-CAM is VERY sensitive to any minor voltage droop. My fix, Add 3 1000uF tantalum caps on the 5v rail. Problem solved! Woo hoo!

RahulYadav08 commented 4 years ago

Is there anyway to implement h264 encoding on esp32 board?

mitchins commented 4 years ago

@err-Rahul it simply can’t handle it. MJPG is the best you can reasonably hope for. Transcode it externally if you need.