[Page Add][Enhanced Function]Support the camera interface for XIAO ESP32S3 Sense, by Zephyr

MatthewJeffson commented 7 months ago

Overview

We need your help to support the camera interface for ESP32 device, and support XIAO ESP32S3 Sense at the following part, using Zephyr.
Reference by Tim, the Seeed Contributor:
- There's an open issue for further extending zephyr support for the camera and microphone on the device tree. https://github.com/zephyrproject-rtos/zephyr/issues/60148 May be worth circling back here once someone has a chance to tackle that issue to further augment it.

How do we work together?

Leave a comment with your GitHub name below as you accept this assignment and will complete the contents about what we are trying to present.
The maintainer will assign and record you as our only contact for this assignment.
You can first Deploy this Wiki Platform Locally (The entire source code .zip file is only 30 M)
You will be invited as our "Contributor" once you "Pull Request".
The maintainer will review the PR and then merge.
What do we offer?

We want to express our gratitude to our contributors by offering a range of rewards. Depending on the difficulties(Tier 0/1/2/3), the time we expect(Days 1/3/7/15), the actual submission time and the actual submission contents, we will offer different levels of rewards.
1. We will provide our products or cash bonus to our contributors as a token of appreciation.
2. For those contributors who join our "Ranger Program"(announced soon), we offer even more exciting rewards, including access to our prototype products.
3. In addition, all contributors and their contributions will be acknowledged and showcased on our wiki platform.
  FAQ
For any question, the contributor can directly commend here.
I think I can complete the assignment, but I need more time?
The timeframe here is based on our old experience, from people who has done it or something like that.
However, many projects will take much more time under actual circumstances (including the main language of the contributors receiving the task and the actual creative difficulty)
You can leave the time you need and a brief reason, we will evaluate the situation accordingly.

Priyanshu0901 commented 7 months ago

@MatthewJeffson I would love to take up on this project.

MatthewJeffson commented 7 months ago

NICE, and thank you! @Priyanshu0901

havoc-dlt commented 6 months ago

Any progress on this one? I am kinda torn between waiting for it and trying something myself right now.

Priyanshu0901 commented 6 months ago

Yes, still a work in progress (still a bit of testing and documentation is left). ETA 1week.

havoc-dlt commented 6 months ago

Thanks a lot for your effort!

epc-ake commented 5 months ago

Hi, I'm also interested in this. Do you have already something to share?

MatthewJeffson commented 5 months ago

Hi! @epc-ake Yes! Thank you for considering it! @Priyanshu0901 Roy is currently not available on this assignment. If you can help us that would be really great! Here is my Discord account: matthew_yu0310 and we can discuss more on the Discord. Best Regards, Matthew

bienxanh1901 commented 4 months ago

Hello Team, Any progress on this task? I can only use Xiao ESP32S3 Sense SD card and Micro on Zephyr. Can you share something about working with camera interface? Can we just only define device tree, or need further work?

Thank you, Hai Pham

epc-ake commented 4 months ago

The problem is that there is currently no driver for the lcd_cam component of the esp32s3. I'm actually writing one now and i already was able to receive some images. However, there are also changes to the DMA driver needed. For those I made a PR yesterday. Once they are merged I'll continue improving the camera driver and make a PR for it as well.

MatthewJeffson commented 4 months ago

Wow! Thank you for the update!

bienxanh1901 commented 4 months ago

The problem is that there is currently no driver for the lcd_cam component of the esp32s3. I'm actually writing one now and i already was able to receive some images. However, there are also changes to the DMA driver needed. For those I made a PR yesterday. Once they are merged I'll continue improving the camera driver and make a PR for it as well.

Thank you so much for your hard working @epc-ake!

epc-ake commented 4 months ago

Hi all,

I made two drafts for a pull requests: zephyrproject-rtos/zephyr#75331 and zephyrproject-rtos/zephyr#74974. I can also provide the overlay file needed for the xiao_esp32s3, but I'm not sure where to place it yet. Note the limitations on external memory mentioned in the PR.

Feel free to test it out and leave your comments.

MatthewJeffson commented 4 months ago

Thank you so much! @epc-ake We will check it very soon! Best Regards, Matthew

pblxptr commented 2 months ago

@epc-ake how did it end up? I saw that support for ESP32S3 CAM interface has been merged to mainline, as well as the PR that allows to allocate video buffer in SPIRAM. How you tried to run these changes with xiao_esp32s3? Possibly with streaming cam over WIFI? I tried myself but did not manage to get any sensible results.

epc-ake commented 2 months ago

HI @pblxptr Yes we made good progress! As you mentioned, the video driver has been merged. To be able to capture in full resolution this PR is needed. Looks like this is merged soon too. But there is also this PR which updates sample/drivers/video/capture and lets you chose a lower resolution for the video stream that fits into internal ram. After that we probably need to add a conf file for the xiao_esp32 board to the sample to make it work out of the box. However, with the right configs set, I'm able to capture video in full resolution now. I also managed to stream video over wifi, but only with a hacky application I did myself.

To be able to run the sample/drivers/video/capture sample, try the following:

check out newest zephyr main
apply (this PR )
apply this PR
then copy the samples/drivers/video/capture/boards/esp32s3_eye_procpu.conf to a new file named samples/drivers/video/capture/boards/xiao_esp32s3_procpu.conf
Build and run the sample/drivers/video/capture sample

pblxptr commented 2 months ago

HI @pblxptr Yes we made good progress! As you mentioned, the video driver has been merged. To be able to capture in full resolution this PR is needed. Looks like this is merged soon too. But there is also this PR which updates sample/drivers/video/capture and lets you chose a lower resolution for the video stream that fits into internal ram. After that we probably need to add a conf file for the xiao_esp32 board to the sample to make it work out of the box. However, with the right configs set, I'm able to capture video in full resolution now. I also managed to stream video over wifi, but only with a hacky application I did myself.

To be able to run the sample/drivers/video/capture sample, try the following:
1. check out newest zephyr main

2. apply ([this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-16cf59216520b76034171d2c94fc72dc3bdcf0af722255a2c226c58cd35de6df) )

3. apply [this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-737bb0ca878e20bfaaf493aae9eb42d3de0e40e22a0fd17ac08d826050c1d41b)

4. then copy the `samples/drivers/video/capture/boards/esp32s3_eye_procpu.conf` to a new file named `samples/drivers/video/capture/boards/xiao_esp32s3_procpu.conf`

5. Build and run the `sample/drivers/video/capture` sample

Hi thanks, basically I've already done everything what you just suggested and it looks good, however I'm struggling with sending the stream over wifi (I guess there are some issues with sharing SPIRAM between video and wifi, anyway I keep tracking some ongoing PRs with WiFi changes, mostly related to heap and SPIRAM), perhaps I also need to make some more hacks in my app.

epc-ake commented 2 months ago

HI @pblxptr Yes we made good progress! As you mentioned, the video driver has been merged. To be able to capture in full resolution this PR is needed. Looks like this is merged soon too. But there is also this PR which updates sample/drivers/video/capture and lets you chose a lower resolution for the video stream that fits into internal ram. After that we probably need to add a conf file for the xiao_esp32 board to the sample to make it work out of the box. However, with the right configs set, I'm able to capture video in full resolution now. I also managed to stream video over wifi, but only with a hacky application I did myself. To be able to run the sample/drivers/video/capture sample, try the following:
1. check out newest zephyr main

2. apply ([this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-16cf59216520b76034171d2c94fc72dc3bdcf0af722255a2c226c58cd35de6df) )

3. apply [this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-737bb0ca878e20bfaaf493aae9eb42d3de0e40e22a0fd17ac08d826050c1d41b)

4. then copy the `samples/drivers/video/capture/boards/esp32s3_eye_procpu.conf` to a new file named `samples/drivers/video/capture/boards/xiao_esp32s3_procpu.conf`

5. Build and run the `sample/drivers/video/capture` sample
Hi thanks, basically I've already done everything what you just suggested and it looks good, however I'm struggling with sending the stream over wifi (I guess there are some issues with sharing SPIRAM between video and wifi, anyway I keep tracking some ongoing PRs with WiFi changes, mostly related to heap and SPIRAM), perhaps I also need to make some more hacks in my app.

Okay so you are able to run the sample/driver/video/capture sample? I have to admit, I've never tried to stream over WIFI from external ram. Maybe there are some problems I didn't run over jet. But I was able to stream from internal ram without any problems.

pblxptr commented 2 months ago

HI @pblxptr Yes we made good progress! As you mentioned, the video driver has been merged. To be able to capture in full resolution this PR is needed. Looks like this is merged soon too. But there is also this PR which updates sample/drivers/video/capture and lets you chose a lower resolution for the video stream that fits into internal ram. After that we probably need to add a conf file for the xiao_esp32 board to the sample to make it work out of the box. However, with the right configs set, I'm able to capture video in full resolution now. I also managed to stream video over wifi, but only with a hacky application I did myself. To be able to run the sample/drivers/video/capture sample, try the following:
1. check out newest zephyr main

2. apply ([this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-16cf59216520b76034171d2c94fc72dc3bdcf0af722255a2c226c58cd35de6df) )

3. apply [this PR](https://github.com/zephyrproject-rtos/zephyr/pull/77879/files#diff-737bb0ca878e20bfaaf493aae9eb42d3de0e40e22a0fd17ac08d826050c1d41b)

4. then copy the `samples/drivers/video/capture/boards/esp32s3_eye_procpu.conf` to a new file named `samples/drivers/video/capture/boards/xiao_esp32s3_procpu.conf`

5. Build and run the `sample/drivers/video/capture` sample
Hi thanks, basically I've already done everything what you just suggested and it looks good, however I'm struggling with sending the stream over wifi (I guess there are some issues with sharing SPIRAM between video and wifi, anyway I keep tracking some ongoing PRs with WiFi changes, mostly related to heap and SPIRAM), perhaps I also need to make some more hacks in my app.
Okay so you are able to run the sample/driver/video/capture sample? I have to admit, I've never tried to stream over WIFI from external ram. Maybe there are some problems I didn't run over jet. But I was able to stream from internal ram without any problems.

Yep I was able to run this sample. To be honest I've stumbled upon your changes after I did mine, because I was looking how to combine all these things together namely wifi and camera using external ram. I will try to hack my app and put everything into internal ram and I'll see what will be the result.

MatthewJeffson commented 2 months ago

Wow! @epc-ake That sounds really amazing! Sorry our guys were busy on the Watcher and I was thinking the new framework of the Contributor Program. It is great to see the progress going well! Happy Mid-Autumn Festival to you guys! Best Regards, Matthew

github-actions[bot] commented 1 month ago

✅ @MatthewJeffson

This issue is closed, If you have any questions, you can comment and reply.

pblxptr commented 1 month ago

@MatthewJeffson why is this closed? Is the support for XIAO ESP32S3 Sense added to Zephyr?

MatthewJeffson commented 1 month ago

Oh! @pblxptr I am sorry I am looking the wrong issue.

Thanks for the notices!

pblxptr commented 1 month ago

@MatthewJeffson hey I submitted a PR with support for the Xiao ESP32 S3 sense. Any remarks will be highly appreciated. https://github.com/zephyrproject-rtos/zephyr/pull/79334

MatthewJeffson commented 1 month ago

Oh Hi! @pblxptr Patryk thanks for the PR! I really appreciated! Are you using XIAO ESP32S3 Sense enable all Zephyr functions? including the camera usage? Best Regards, Matthew

pblxptr commented 1 month ago

Oh Hi! @pblxptr Patryk thanks for the PR! I really appreciated! Are you using XIAO ESP32S3 Sense enable all Zephyr functions? including the camera usage? Best Regards, Matthew

Hi, enabling camera was my main focus, so camera and sd card available on XIAO ESP32S3 Sense are working. There is also a mic on this board as far as I know, but tbh I didn't need it so I haven't touched it.

MatthewJeffson commented 1 month ago

That is nice! I will have my colleagues contacting the Zephyr Official regarding the PR. By the way, would you mind upgrading the wiki about how to use it? https://wiki.seeedstudio.com/xiao_esp32s3_zephyr_rtos/

Refer to the Contributor Program, I really want to make some compensation for your efforts!

Best Regards, Matthew

pblxptr commented 1 month ago

That is nice! I will have my colleagues contacting the Zephyr Official regarding the PR. By the way, would you mind upgrading the wiki about how to use it? https://wiki.seeedstudio.com/xiao_esp32s3_zephyr_rtos/

Refer to the Contributor Program, I really want to make some compensation for your efforts!

Best Regards, Matthew

Sure, once they merge my changes I'll update the wiki

MatthewJeffson commented 1 month ago

NICE! That would be great! Thanks!

Seeed-Studio / wiki-documents

[Page Add][Enhanced Function]Support the camera interface for XIAO ESP32S3 Sense, by Zephyr #1060

Overview

How do we work together?

What do we offer?

FAQ

I think I can complete the assignment, but I need more time?