Images stop saving / error in image aftert ~ 1 hr..

[ebgoldstein]

This is from @efarq (in https://github.com/TinyCamML/Boron-and-OpenMV/issues/26#issuecomment-2243314197):

Well, I just ran it successfully for one hour, and then the same issue started occurring. I have attached the datalog.txt, I have also attached the images, which shows that the same thing was happening with the images becoming a 2in1 situation. So I think I'll try adding the sensor.reset information in the while loop to see if that helps.

images.zip DataLog.txt

[ebgoldstein]

ok, the last column of the datalog is the memory, which seems to stay constant/low..

this isuggests to me its not a memory issue, but maybe something else... what are th options on the table right now @efarq , @SUPScientist ?

[ebgoldstein]

the glitches start at the pictures where the memory gets larger for a cycle or two...

[SUPScientist]

Still curious about the info here (https://forums.openmv.io/t/sleep-mode-wakeup-with-external-interrupt/767/2), despite it being a bit old, as mentioned here (https://github.com/TinyCamML/Boron-and-OpenMV/issues/7#issuecomment-2236514945). The portion of the frame granted to the split image appears to grow over time which could be another symptom of lack of proper resetting?

[SUPScientist]

Does sensor shutdown/sleep need to happen? I see those are commented out:

https://github.com/TinyCamML/Boron-and-OpenMV/blob/19091607839c727c075aa3a415274c732d4bae33/Firmware/OpenMV/PublishOpenMV.py#L80-L89

[efarq]

So I added the sensor.reset() lines to the while loop, and that still doesn't seem to fix the issue. The sensor shutdown items that are commented out are the steps that the senior design team took before putting their camera into machine.deepsleep(). I can try them for machine.sleep() though and see if it helps. The one other thing I think could be happening is when the boron sends its datetime to the openmv, the timing could be off, so then when the openmv tries to write that information to the logs, it doesn't have the number to do so and glitches or something

[SUPScientist]

The main glitch here occurred before the addition of the uart.read() so I doubt it's that, but worth looking under every stone now I suppose. https://github.com/TinyCamML/Boron-and-OpenMV/blob/19091607839c727c075aa3a415274c732d4bae33/Firmware/OpenMV/PublishOpenMV.py#L47

I don't understand how it was decided what goes into the while loop vs. not in the entire OpenMV script, though. What if almost everything is in that loop? @ebgoldstein, have a ref for what OpenMV does upon wakeup that might help us diagnose?

[efarq]

Just tried adding the sensor.sleep(True) sensor.shutdown(True) lines to the end before sensor.sleep() and that does not seem to work either.

[ebgoldstein]

i see what you mean @SUPScientist - the whole camera might need to re-initialize in the while loop... the PR i link to above makes this change... @efarq , can you try it?

[efarq]

Still does not work

[ebgoldstein]

can you drop the txt file here? or did it not even make it to save the image?

[efarq]

DataLog.txt It looks like it did save the image and said "NoFLOOD" but did not send it over to the boron

[efarq]

Ahh okay so the issue is not quite resolved. So it was just running nicely for about 40 minutes and then started doing the thing again. The only difference from when Evan and I were looking at it successfully on zoom until now was that I put the sampling period back to 6 minutes and turned the publishing back on for the Boron. On the OpenMV, the images look fine, but a new ERROR.txt was created, which I've attached. Also, the OpenMV light started blinking nonstop, which is what caught my eye this time, which hasn't happened really much before

https://github.com/user-attachments/assets/c70c18a1-6da7-4265-9d93-b6eef2ce623c

DataLog.txt ERROR.LOG

[ebgoldstein]

shoot.. ok, looking at forums now...

[ebgoldstein]

can you try something simple, and just replace the jpg to bmp on this line:

https://github.com/TinyCamML/Boron-and-OpenMV/blob/19091607839c727c075aa3a415274c732d4bae33/Firmware/OpenMV/PublishOpenMV.py#L66

so img.save("./images/" + file_name + ".bmp")

(this error has popped up previously on the forums related to jpg stuff...

[efarq]

The .bmp change does not seem to do the trick. I also added a del img line because I saw that could help in this forum (https://forums.openmv.io/t/memory-allocation-error-comments/977/2) but it doesn't seem to do the trick either

[ebgoldstein]

ok, @efarq - can you change back to jpg, run the code till it breaks, and post all of the relevant details here (the exact OpenMV code, the folder of images, datalog, and error log)? and also maybe any details (how long it has run for)?

i think its time we post on the OpenMV forums, and I can do that

[ebgoldstein]

also, please note the firmware, and openMV details (at the bottom of the IDE)

[efarq]

Also, I have a question for @SUPScientist, the openmv issue seemed to be fixed until we turned on the cellular from the boron and entered the publish state. In the publish state, we have the following lines of code: // Clean out any residual junk in buffer and restart serial port Serial1.end(); delay(1000); Serial1.begin(9600); delay(500); Serial1.setTimeout(TIMEOUT_TINYCAM_MS);

My question is why do we begin the Serial1 again when we don't use it for the rest of the publish or sleep state? Could that be causing a mismatch between the boron and openmv?

[ebgoldstein]

just to clarify @efarq - if cellular is off, the OpenMV will run forever?

[efarq]

I only saw what we saw on zoom together yesterday but it looked like it was working great without cellular. I'll try it out again real quick

[SUPScientist]

OK, interesting. Eager to hear results of next test with publishing off (#define PUBLISHING 0). No idea how/when/from whom that chunk https://github.com/TinyCamML/Boron-and-OpenMV/blob/19091607839c727c075aa3a415274c732d4bae33/Firmware/Boron/src/TinyCamPublish.ino#L107-L112 crept into PUBLISH_STATE but you're correct that it doesn't make sense there. Good catch.

[ebgoldstein]

We need a very minimal reproducible example of the failure if we are going to post on OpenMV forum, so i am trying this (OpenMV CAM H7 Plus (STM32H743 w/SDRAM) Sensor:OV5640 Firmware 4.5.5):

Note that there is no sleep (nor poll, nor uart.read), but each loop it re-initializes then shuts down the camera...

import sensor, image, time, utime, pyb, tf, machine, gc, os, uselect
from pyb import UART, Pin, ExtInt
from machine import LED

uart = UART(1, 9600) # UART1, adjust baudrate as needed

net = tf.load('MNv2Flood_cat_CG.tflite', load_to_fb=True)
labels = ['Flood', 'NoFlood']

#make directory to save images
if not "images" in os.listdir():
    os.mkdir("images")  # Make a directory

pin = Pin("P7", Pin.IN, Pin.PULL_UP)
ext = ExtInt(pin, ExtInt.IRQ_FALLING, Pin.PULL_UP, callback)

curr_time = 0

while(True):
    #Reinitialize the sensor after sleep
    sensor.reset() # Initialize the camera sensor.
    sensor.set_pixformat(sensor.RGB565)
    sensor.set_framesize(sensor.QVGA)
    sensor.skip_frames(time = 2000)

    img = sensor.snapshot()

    TF_objs = net.classify(img)

    Flood = TF_objs[0].output()[0]
    NoFlood = TF_objs[0].output()[1]

    if Flood > NoFlood:
        print('Flood')
        uart.write('Flood')
        pyb.delay(1000);
        file_name = str(curr_time) +"_FLOOD"
        floodstate = "Flood"

    else:
        print('No Flood')
        uart.write('No Flood')
        pyb.delay(1000);
        file_name = str(curr_time) +"_NOFLOOD"
        floodstate = "NoFlood"

    with open("./DataLog.txt", 'a') as file:
        file.write(str(curr_time) + "," + floodstate + "," + file_name + "," + str(gc.mem_alloc()) + "\n")

    print(curr_time)
    curr_time = curr_time +1

    #print(file_name)
    img.save("./images/" + file_name + ".jpg")

    # run garbarge collection to prevent memory growth
    gc.collect()

    # Begin power down ------------------------------------
    # Shutdown the sensor (pulls PWDN high).
    sensor.shutdown(True)

[efarq]

Okay cool I'm trying it out with publishing off right now and then if that continues to work, I'll try commenting out those 108-112 lines and turning publishing back on to see if that helps anything.

[efarq]

And then if that doesn't work, I'll get you all those files/details for the openmv forum, Evan

[ebgoldstein]

i do think there is a timing issue, only because the datalog.txt files have strange moments where memory grows and there seems to be a 'skipped' uart read/write cycle... so the lines you identified might atleast help with that 🤞

[efarq]

Update: commenting out lines 108-112 on boron's code still did not work. I'm going to continue to mess with the Boron's code, but I think it would be good to post on the openmv forum too. Exact openmv code as a .txt: maintxt.txt folder of images: images.zip datalog file: DataLog.txt error log: ERROR.LOG Firmware: 4.5.5 Board: OpenMV Cam H7 Plus (STM32H743 w/ SDRam) Sensor: OV5640

Things we've tried: changing .jpg to .bmp, adding a gc.collect(), adding del img, making sure we have the sensor.shutdown and reinitializing the camera after every restart Symptoms: OpenMV cam starts blinking white

[ebgoldstein]

@efarq - I have read that sometimes long openMV runs where it's connected to the IDE have problems. Is it possible for you to try deploying this not connected to the computer and see what happens?

I saw this because the minimal example i posted above would error out after 50 mins when being run via the IDE, but i just ran it with the openMV connected to battery power and it was fine for several hours, no error...

[efarq]

Currently, the only thing connected to my computer is the boron, which is powering the rest of it - so the OpenMV is not connected to the IDE.

Just writing down this idea - but maybe the next best thing to do would be make some sort code that runs like:

1. Boron wakes up OpenMV and waits to move on until it receives something from Serial1.readString() aka flood or no flood.
2. The openmv has a delay while the serial1 is resetting in the boron and then waits to move on until it receives a value for the datetime from the boron. Then it sends an "all clear" to boron
3. Then the Boron won't enter publish mode and subsequently publish mode until it receives an "all clear"

I guess that opens the possibility of either one getting stuck in infinite loops but just thinking about how to get their timing lined up. The other thing I am wondering about is if there is a way to turn off the vusb power entirely on the boron each cycle, as that would give the openmv a fresh start every time. Because we use the A0 pin on the boron to toggle the openmv on, it seems like you may have thought about that before. Perhaps machine.lightsleep() doesn't reset the camera enough? Just brainstorming at this point

[ebgoldstein]

i like this idea, especially if we pair it with writing to a text file on OpenMV to help debug...

[ebgoldstein]

and i think we need this sort of brainstorming... i firmly believe there must be a way forward, and i think it involves being more careful with all the comms (and having a way to track it for debugging)..

[SUPScientist]

The other thing I am wondering about is if there is a way to turn off the vusb power entirely on the boron each cycle, as that would give the openmv a fresh start every time.

This merits more exploration. A FET switch tied to VUSB's output from Boron to OpenMV and another GPIO on Boron should do the trick (should have some in the lab, same as I think you used for SEACOWs). The "fresh start" idea is compelling.

Re: @efarq's other idea that slows down/makes more explicit the Boron<—>OpenMV handshake seems reasonable but I'm having a hard time guessing how that gets at the root problem here. Maybe serial.flush statements on both sides "clean out" any residual stuff in buffers if that's the hypothesized problem?

[efarq]

I will continue to investigate this week...

[efarq]

I saw @ebgoldstein's branch about cleaning up the uart communications and that it worked for you for at least one hour. Were you testing that with cellular on the boron? Your code is not quite working as nicely for me.

However, last week I did add the mosfet switch into the system and it seems to work great! I ran it for almost 12 hours with no issues except that it adds a � into the result, which I cannot figure out how to get rid of:

It does seem to publish the character with no issues though so I don't think the character is necessarily a huge issue but will just need to be fixed eventually. Here is the code within the branch that adds the on/off switch: https://github.com/TinyCamML/Boron-and-OpenMV/blob/OpenMVFix/Firmware/Boron/src/TinyCamPublish.ino

I'll test it again today but I feel pretty good about the consistency with the mosfet switch.

[ebgoldstein]

this is great news @efarq !

Did your second test run ok?

if so, i guess the action item is to make sure the boron & OpenMV are exactly the same as one the decives.. ( i see the new circuit diagram w/ mosfet).. then close the issue and we all have a big party in your honor for figuring it out!

[efarq]

Yes the second test seemed to run quite nicely! So maybe the mosfet switch is the way to go. What do you mean by, "the action item is to make sure the boron & OpenMV are exactly the same as one the decives" before closing the issue?

[ebgoldstein]

Sorry, was not super clear.. All I mean is- 'If the code on GitHub the exact same as the code on the devices, close the issue. If not, update code and close the issue'

[efarq]

Perfect, just wanted to make sure before I changed anything! Here is the merge: commit b141d4c