Can't upload script via MaixPy IDE

[alexjones85]

I am trying to upload a script that was developed by @AIWintermuteAI and posted here aXeleRate - Keras-Based Framework for AI on the Edge.

But there seems to be some issue where there is not an obvious error message that I can follow. I have uploaded the latest firmware to the M.A.R.K from the Codecraft web interface and then tried to run the script posted below:

import sensor,image,lcd, os, time, utime
from maix_motor import Maix_motor
import KPU as kpu

def find_center():
    img = sensor.snapshot()
    img = img.resize(224,224)
    img= img.rotation_corr(z_rotation=90.0)
    a = img.pix_to_ai()
    code = kpu.run_yolo2(task, img)
    if code:
        for i in code:
            a=img.draw_rectangle(i.rect(),color = (0, 255, 0))
            a = img.draw_string(i.x(),i.y(), classes[i.classid()], color=(255,0,0), scale=3)
            x_center = i.x()+i.w()/2
            print(x_center)
        a = lcd.display(img)
        return x_center
    else:
        a = lcd.display(img)
        return -1

def scan_pan():
    Maix_motor.motor_run(0, 0, 0)
    Maix_motor.servo_angle(2, 110)
    for i in range(0, 180, 5):
        pos = find_center()
        if pos >= 100 and pos <= 124:
            Maix_motor.servo_angle(1, 90)
            if i >= 90:
                Maix_motor.motor_right(40, 0)
                Maix_motor.motor_left(-40, 0)
                utime.sleep_ms(((i - 90) * 12))
            if i < 90:
                Maix_motor.motor_right(-40, 0)
                Maix_motor.motor_left(40, 0)
                utime.sleep_ms(((90 - i) * 12))
            Maix_motor.motor_motion(1, 1, 0)
            time.sleep(0.3)
            return True
        else:
            Maix_motor.servo_angle(1, i)

def scan_tilt():
    #Maix_motor.motor_run(0, 0, 0)
    Maix_motor.servo_angle(1, 90)
    for i in range(100, 140, 2):
        pos = find_center()
        if pos > 0:
            return True
        else:
            Maix_motor.servo_angle(2, i)

def scan_rotate():
    Maix_motor.servo_angle(1, 90)
    Maix_motor.servo_angle(2, 110)
    Maix_motor.motor_motion(1, 3, 0)
    while True:
        pos = find_center()
        if pos > 0:
            break

lcd.init()
sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.set_vflip(1)
sensor.run(1)
Maix_motor.servo_angle(1, 90)
Maix_motor.servo_angle(2, 110)
DEBUG = 0
classes = ["person"]
task = kpu.load('/sd/model_people_end.kmodel')
a = kpu.set_outputs(task, 0, 7,7,30)
anchor = (0.57273, 0.677385, 1.87446, 2.06253, 3.33843, 5.47434, 7.88282, 3.52778, 9.77052, 9.16828)
a = kpu.init_yolo2(task, 0.3, 0.3, 5, anchor)
while(True):
    x_center = find_center()
    if not DEBUG:
        if x_center >= 100 and x_center <= 124: Maix_motor.motor_motion(1, 1, 0)
        if x_center < 100 and x_center > 0: Maix_motor.motor_motion(1, 4, 0)
        if x_center > 124: Maix_motor.motor_motion(1, 3, 0)
        if x_center < 0:
            if not scan_tilt():
                scan_pan()
                if not scan_pan():
                    scan_rotate()
a = kpu.deinit(task)

I have trained the kmodel placed it on the SD card with the correct filename and inserted into the CyberEye module. I am able to connect to the board through MaixPy IDE but when trying to run the script the terminal displays the following before it automatically disconnects the serial port connection:

>>> init i2c:2 freq:100000
[MAIXPY]: find ov2640
[MAIXPY]: find ov sensor

Nothing is displayed on the lcd screen of the device except a black screen.

FYI as additional information: I have been able to upload other scripts to a seperate Maixduino device using MaixPy IDE. And have had the M.A.R.K kit working with code developed through the Codecraft graphical coding web interface such as the line following example using with the on board camera and object detection algorithm.

Is anyone able to assist in troubleshooting why this script isn't functioning?

[AIWintermuteAI]

Hello, Alex! Okay, so I think you get a segfault on this line

task = kpu.load('/sd/model_people_end.kmodel')

The reason you're not seeing a segfault message is that MaixPy IDE console doesn't show C segfault tracebacks. In order to see if it is a segfault you need to: 1) put a print statement on both sides of task = kpu.load('/sd/model_people_end.kmodel'). It doesn't matter what you print, can be 1 and 2 for example. We just want to see how far the code execution go, it is a standard debugging technique. 2) Open the terminal in MaixPy IDE (Tools -> Open Terminal - > New Terminal, default baud rate (115200), the port is specific to your OS and machine) 3) Copy all the code, then in terminal window press Ctrl + E, paste the code and press Ctrl + D. That will run the code. Paste the output here please.

[alexjones85]

Ok. I got the code to run in the serial port terminal. It looks like there is a memory issue: E (68524594969) SYSCALL: Out of memory. The full output is below. It looks like the code does not load the model as the 2nd print command does not execute. Is the model size too big for the device memory? I trained the model with input size 224x224 using Mobilenet7_5 architechture. The size of the kmodel file on disk is 1.9 MB.

{"board_info": {"WIFI_RX": 7, "ESP32_MOSI": 28, "MIC0_BCK": 18, "MIC0_WS": 19, "WIFI_TX": 6, "ESP32_RST": 8, "ESP32_MISO": 26, "ESP32_SCLK": 27, "PIN0": 4, "LED_G": 12, "ESP32_CS": 25, "PIN1": 5, "PIN3": 22, "PIN4": 23, "LED_B": 14, "PIN5": 24, "ESP32_RDY": 9, "PIN6": 32, "PIN7": 15, "PIN8": 14, "PIN9": 13, "PIN10": 12, "I2S_WS": 33, "I2S_DA": 34, "BOOT_KEY": 16, "MIC0_DATA": 20, "I2S_BCK": 35, "PIN11": 11, "PIN12": 10, "PIN13": 3, "PIN2": 21, "LED_R": 13, "WIFI_EN": 8}, "type": "duino"}
###free gc heap memory : 748 KB

###free sys heap memory: 3552 KB

MicroPython v0.5.0-395-gcb71b9c42-dirty on 2021-10-21; Sipeed_M1 with kendryte-k210
Type "help()" for more information.
>>> 
>>> 
>>> 
raw REPL; CTRL-B to exit
>OK
init i2c:2 freq:100000

[MAIXPY]: find ov2640
[MAIXPY]: find ov sensor
1
E (12095451000) SYSCALL: Out of memory

[alexjones85]

Update. It loads the model with the Mobilenet5_0 architechture but am still receiving an error message:

[MAIXPY]: find ov2640
[MAIXPY]: find ov sensor
1
2
w*h*ch!=3528

[MAIXPY]kpu: set_outputs arg value error: w,c,ch size not match output size

Traceback (most recent call last):
  File "<stdin>", line 82, in <module>
  File "<stdin>", line 10, in find_center
OSError: ERR_OUTPUTS_SET: need kpu.set_outputs() to set shape
>
MicroPython v0.5.0-395-gcb71b9c42-dirty on 2021-10-21; Sipeed_M1 with kendryte-k210
Type "help()" for more information.
>>> 

I will try to find the solution. I think maybe you already posted the answer here: https://github.com/AIWintermuteAI/aXeleRate/issues/10

So I checked the output values before reshape as suggested, see below:

Then change the section of code in MaixyPy IDE to a = kpu.set_outputs(task, 0, 14,14,18) as below:

task = kpu.load('/sd/model_people_end.kmodel')
a = kpu.set_outputs(task, 0, 14,14,18)
anchor = (0.57273, 0.677385, 1.87446, 2.06253, 3.33843, 5.47434, 7.88282, 3.52778, 9.77052, 9.16828)
a = kpu.init_yolo2(task, 0.3, 0.3, 5, anchor)
while(True):
    x_center = find_center()

And the output still says:

[MAIXPY]: find ov2640
[MAIXPY]: find ov sensor
w*h*ch!=3528

[MAIXPY]kpu: set_outputs arg value error: w,c,ch size not match output size

Traceback (most recent call last):
  File "<stdin>", line 82, in <module>
  File "<stdin>", line 11, in find_center
OSError: ERR_OUTPUTS_SET: need kpu.set_outputs() to set shape

I notice that the anchors are different on the device script to what I trained. The anchors I used in the config file for training are: anchor = [[[0.76120044, 0.57155991], [0.6923348, 0.88535553], [0.47163042, 0.34163313]],[[0.33340788, 0.70065861], [0.18124964, 0.38986752], [0.08497349, 0.1527057 ]]]

So I change the anchors in the device script to see if that fixes the issue as below:

task = kpu.load('/sd/model_people_end.kmodel')
a = kpu.set_outputs(task, 0, 14,14,18)
# anchor = (0.57273, 0.677385, 1.87446, 2.06253, 3.33843, 5.47434, 7.88282, 3.52778, 9.77052, 9.16828)
anchor = [[[0.76120044, 0.57155991], [0.6923348, 0.88535553], [0.47163042, 0.34163313]],[[0.33340788, 0.70065861], [0.18124964, 0.38986752], [0.08497349, 0.1527057 ]]]
a = kpu.init_yolo2(task, 0.3, 0.3, 5, anchor)
while(True):
    x_center = find_center()

And the script outputs the following:

w*h*ch!=3528

[MAIXPY]kpu: set_outputs arg value error: w,c,ch size not match output size

Traceback (most recent call last):
  File "<stdin>", line 80, in <module>
ValueError: requested length 10 but object has length 2

Do I need to retrain the model with the anchors the same as being run on the device? I try that in the Colab notebook from here: https://github.com/AIWintermuteAI/aXeleRate/blob/master/resources/aXeleRate_person_detector.ipynb

But get the following error:

Project folder /content/drive/MyDrive/Colab Notebooks/Sipeed_Maix_kmodel_trainer/person_dataset/person_detector already exists. Creating a folder for new training session.
Tflite Converter ready
Downloading K210 Converter
Downloading data from https://github.com/kendryte/nncase/releases/download/v0.2.0-beta4/ncc_linux_x86_64.tar.xz
7520256/7513544 [==============================] - 0s 0us/step
7528448/7513544 [==============================] - 0s 0us/step
/content/aXeleRate/axelerate/networks/common_utils/ncc_linux_x86_64.tar.xz
['person']
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-8-18434c67d1ce>](https://localhost:8080/#) in <module>()
      1 from keras import backend as K
      2 K.clear_session()
----> 3 model_path = setup_training(config_dict=config)

2 frames
[/content/aXeleRate/axelerate/networks/yolo/frontend.py](https://localhost:8080/#) in create_yolo(architecture, labels, input_size, anchors, obj_thresh, iou_thresh, coord_scale, object_scale, no_object_scale, weights)
     35 
     36     n_classes = len(labels)
---> 37     n_boxes = int(len(anchors[0]))
     38     n_branches = len(anchors)
     39     yolo_network = create_yolo_network(architecture, input_size, n_classes, n_boxes, n_branches, weights)

TypeError: object of type 'float' has no len()

Any ideas what I'm doing wrong here? Or suggestions on how to fix?

[AIWintermuteAI]

Okay, now I understand what is going on here. aXeleRate was upgraded to YOLOv3 - I made a PR to MaixPy, but it seems it is no longer in active development, so they never merged it. https://github.com/sipeed/MaixPy/pull/451 https://github.com/AIWintermuteAI/aXeleRate/issues/39 And of course it did not make its way to MARK firmware.

The best course for you is to train a model with legacy aXeleRate - you can do that by switching to legacy-yolov2 branch in Colab notebook right after cloning the repository. https://github.com/AIWintermuteAI/aXeleRate/tree/legacy-yolov2

Tell me how that goes! The model trained there should work on MARK.

[alexjones85]

Bugger! So you don't expect the PR to ever get merged now? Or is just extensively delayed?

I tried running with the legacy repo but it seems like there is still some issue as there are not inference images created after finishing the notebook cells. I think it has something to do with the deprecation warnings.

I'm loading the legacy repo like this:

%load_ext tensorboard
#we need imgaug 0.4 for image augmentations to work properly, see https://stackoverflow.com/questions/62580797/in-colab-doing-image-data-augmentation-with-imgaug-is-not-working-as-intended
!pip uninstall -y imgaug && pip uninstall -y albumentations && pip install imgaug==0.4
!pip install --upgrade --no-cache-dir gdown
!git clone -b legacy-yolov2 https://github.com/AIWintermuteAI/aXeleRate.git
import sys
sys.path.append('/content/aXeleRate')
from axelerate import setup_training, setup_inference

Visualising the dataset works fine when loading this cell:

%matplotlib inline
# !gdown https://drive.google.com/uc?id=1UWwxlJm5JH_JiBY9PoLgGyHsRDzBqRGU #dataset
# !gdown https://drive.google.com/uc?id=1-2fiBxykZVZBRcux9I6mKZaS3yAHq6hk #pre-trained model

# !unzip --qq person_dataset.zip

from axelerate.networks.common_utils.augment import visualize_detection_dataset

visualize_detection_dataset(img_folder='person_dataset/imgs_validation', ann_folder='person_dataset/anns_validation', img_size=None, augment=True)

My training config file looks like this:

config = {
        "model":{
            "type":                 "Detector",
            "architecture":         "MobileNet5_0",
            "input_size":           [224,224],
            "anchors":            [0.57273, 0.677385, 1.87446, 2.06253, 3.33843, 5.47434, 7.88282, 3.52778, 9.77052, 9.16828],
            # "anchors":              [[[0.76120044, 0.57155991], [0.6923348, 0.88535553], [0.47163042, 0.34163313]],
            #                         [[0.33340788, 0.70065861], [0.18124964, 0.38986752], [0.08497349, 0.1527057 ]]],
            "labels":               ["person"],
            "obj_thresh" :        0.7,
            "iou_thresh" :        0.5,
            "coord_scale" :         1.0,
            "class_scale" :         1.0,
            "object_scale" :        5.0,
            "no_object_scale" :     1.0
        },
        "weights" : {
            "full":                 "",
            "backend":              "imagenet"
        },
        "train" : {
            "actual_epoch":         1,
            "train_image_folder":   "person_dataset/imgs",
            "train_annot_folder":   "person_dataset/anns",
            "train_times":          1,
            "valid_image_folder":   "person_dataset/imgs_validation",
            "valid_annot_folder":   "person_dataset/anns_validation",
            "valid_times":          1,
            "valid_metric":         "mAP",
            "batch_size":           10,
            "learning_rate":        1e-3,
            "saved_folder":         F"/content/drive/MyDrive/Colab Notebooks/Sipeed_Maix_kmodel_trainer/person_dataset/person_detector",
            "first_trainable_layer": "",
            "augumentation":            True,
            "is_only_detect" :      False
        },
        "converter" : {
            "type":                 ["k210","tflite"]
        }
    }

The run the training:

from keras import backend as K 
K.clear_session()
model_path = setup_training(config_dict=config)

And gives the below output:

Project folder /content/drive/MyDrive/Colab Notebooks/Sipeed_Maix_kmodel_trainer/person_dataset/person_detector already exists. Creating a folder for new training session.
Tflite Converter ready
K210 Converter ready
['person']
Successfully loaded imagenet backend weights
Failed to load pre-trained weights for the whole model. It might be because you didn't specify any or the weight file cannot be found
Current training session folder is /content/drive/MyDrive/Colab Notebooks/Sipeed_Maix_kmodel_trainer/person_dataset/person_detector/2022-04-15_03-30-53

Model: "yolo"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 224, 224, 3)]     0         

 conv1_pad (ZeroPadding2D)   (None, 226, 226, 3)       0         

 conv1 (Conv2D)              (None, 112, 112, 16)      432       

 conv1_bn (BatchNormalizatio  (None, 112, 112, 16)     64        
 n)                                                              

 conv1_relu (ReLU)           (None, 112, 112, 16)      0         

 conv_dw_1 (DepthwiseConv2D)  (None, 112, 112, 16)     144       

 conv_dw_1_bn (BatchNormaliz  (None, 112, 112, 16)     64        
 ation)                                                          

 conv_dw_1_relu (ReLU)       (None, 112, 112, 16)      0         

 conv_pw_1 (Conv2D)          (None, 112, 112, 32)      512       

 conv_pw_1_bn (BatchNormaliz  (None, 112, 112, 32)     128       
 ation)                                                          

 conv_pw_1_relu (ReLU)       (None, 112, 112, 32)      0         

 conv_pad_2 (ZeroPadding2D)  (None, 114, 114, 32)      0         

 conv_dw_2 (DepthwiseConv2D)  (None, 56, 56, 32)       288       

 conv_dw_2_bn (BatchNormaliz  (None, 56, 56, 32)       128       
 ation)                                                          

 conv_dw_2_relu (ReLU)       (None, 56, 56, 32)        0         

 conv_pw_2 (Conv2D)          (None, 56, 56, 64)        2048      

 conv_pw_2_bn (BatchNormaliz  (None, 56, 56, 64)       256       
 ation)                                                          

 conv_pw_2_relu (ReLU)       (None, 56, 56, 64)        0         

 conv_dw_3 (DepthwiseConv2D)  (None, 56, 56, 64)       576       

 conv_dw_3_bn (BatchNormaliz  (None, 56, 56, 64)       256       
 ation)                                                          

 conv_dw_3_relu (ReLU)       (None, 56, 56, 64)        0         

 conv_pw_3 (Conv2D)          (None, 56, 56, 64)        4096      

 conv_pw_3_bn (BatchNormaliz  (None, 56, 56, 64)       256       
 ation)                                                          

 conv_pw_3_relu (ReLU)       (None, 56, 56, 64)        0         

 conv_pad_4 (ZeroPadding2D)  (None, 58, 58, 64)        0         

 conv_dw_4 (DepthwiseConv2D)  (None, 28, 28, 64)       576       

 conv_dw_4_bn (BatchNormaliz  (None, 28, 28, 64)       256       
 ation)                                                          

 conv_dw_4_relu (ReLU)       (None, 28, 28, 64)        0         

 conv_pw_4 (Conv2D)          (None, 28, 28, 128)       8192      

 conv_pw_4_bn (BatchNormaliz  (None, 28, 28, 128)      512       
 ation)                                                          

 conv_pw_4_relu (ReLU)       (None, 28, 28, 128)       0         

 conv_dw_5 (DepthwiseConv2D)  (None, 28, 28, 128)      1152      

 conv_dw_5_bn (BatchNormaliz  (None, 28, 28, 128)      512       
 ation)                                                          

 conv_dw_5_relu (ReLU)       (None, 28, 28, 128)       0         

 conv_pw_5 (Conv2D)          (None, 28, 28, 128)       16384     

 conv_pw_5_bn (BatchNormaliz  (None, 28, 28, 128)      512       
 ation)                                                          

 conv_pw_5_relu (ReLU)       (None, 28, 28, 128)       0         

 conv_pad_6 (ZeroPadding2D)  (None, 30, 30, 128)       0         

 conv_dw_6 (DepthwiseConv2D)  (None, 14, 14, 128)      1152      

 conv_dw_6_bn (BatchNormaliz  (None, 14, 14, 128)      512       
 ation)                                                          

 conv_dw_6_relu (ReLU)       (None, 14, 14, 128)       0         

 conv_pw_6 (Conv2D)          (None, 14, 14, 256)       32768     

 conv_pw_6_bn (BatchNormaliz  (None, 14, 14, 256)      1024      
 ation)                                                          

 conv_pw_6_relu (ReLU)       (None, 14, 14, 256)       0         

 conv_dw_7 (DepthwiseConv2D)  (None, 14, 14, 256)      2304      

 conv_dw_7_bn (BatchNormaliz  (None, 14, 14, 256)      1024      
 ation)                                                          

 conv_dw_7_relu (ReLU)       (None, 14, 14, 256)       0         

 conv_pw_7 (Conv2D)          (None, 14, 14, 256)       65536     

 conv_pw_7_bn (BatchNormaliz  (None, 14, 14, 256)      1024      
 ation)                                                          

 conv_pw_7_relu (ReLU)       (None, 14, 14, 256)       0         

 conv_dw_8 (DepthwiseConv2D)  (None, 14, 14, 256)      2304      

 conv_dw_8_bn (BatchNormaliz  (None, 14, 14, 256)      1024      
 ation)                                                          

 conv_dw_8_relu (ReLU)       (None, 14, 14, 256)       0         

/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/adam.py:105: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  super(Adam, self).__init__(name, **kwargs)
 conv_pw_8 (Conv2D)          (None, 14, 14, 256)       65536     

 conv_pw_8_bn (BatchNormaliz  (None, 14, 14, 256)      1024      
 ation)                                                          

 conv_pw_8_relu (ReLU)       (None, 14, 14, 256)       0         

 conv_dw_9 (DepthwiseConv2D)  (None, 14, 14, 256)      2304      

 conv_dw_9_bn (BatchNormaliz  (None, 14, 14, 256)      1024      
 ation)                                                          

 conv_dw_9_relu (ReLU)       (None, 14, 14, 256)       0         

 conv_pw_9 (Conv2D)          (None, 14, 14, 256)       65536     

 conv_pw_9_bn (BatchNormaliz  (None, 14, 14, 256)      1024      
 ation)                                                          

 conv_pw_9_relu (ReLU)       (None, 14, 14, 256)       0         

 conv_dw_10 (DepthwiseConv2D  (None, 14, 14, 256)      2304      
 )                                                               

 conv_dw_10_bn (BatchNormali  (None, 14, 14, 256)      1024      
 zation)                                                         

 conv_dw_10_relu (ReLU)      (None, 14, 14, 256)       0         

 conv_pw_10 (Conv2D)         (None, 14, 14, 256)       65536     

 conv_pw_10_bn (BatchNormali  (None, 14, 14, 256)      1024      
 zation)                                                         

 conv_pw_10_relu (ReLU)      (None, 14, 14, 256)       0         

 conv_dw_11 (DepthwiseConv2D  (None, 14, 14, 256)      2304      
 )                                                               

 conv_dw_11_bn (BatchNormali  (None, 14, 14, 256)      1024      
 zation)                                                         

 conv_dw_11_relu (ReLU)      (None, 14, 14, 256)       0         

 conv_pw_11 (Conv2D)         (None, 14, 14, 256)       65536     

 conv_pw_11_bn (BatchNormali  (None, 14, 14, 256)      1024      
 zation)                                                         

 conv_pw_11_relu (ReLU)      (None, 14, 14, 256)       0         

 conv_pad_12 (ZeroPadding2D)  (None, 16, 16, 256)      0         

 conv_dw_12 (DepthwiseConv2D  (None, 7, 7, 256)        2304      
 )                                                               

 conv_dw_12_bn (BatchNormali  (None, 7, 7, 256)        1024      
 zation)                                                         

 conv_dw_12_relu (ReLU)      (None, 7, 7, 256)         0         

 conv_pw_12 (Conv2D)         (None, 7, 7, 512)         131072    

 conv_pw_12_bn (BatchNormali  (None, 7, 7, 512)        2048      
 zation)                                                         

 conv_pw_12_relu (ReLU)      (None, 7, 7, 512)         0         

 conv_dw_13 (DepthwiseConv2D  (None, 7, 7, 512)        4608      
 )                                                               

 conv_dw_13_bn (BatchNormali  (None, 7, 7, 512)        2048      
 zation)                                                         

 conv_dw_13_relu (ReLU)      (None, 7, 7, 512)         0         

 conv_pw_13 (Conv2D)         (None, 7, 7, 512)         262144    

 conv_pw_13_bn (BatchNormali  (None, 7, 7, 512)        2048      
 zation)                                                         

 conv_pw_13_relu (ReLU)      (None, 7, 7, 512)         0         

 detection_layer_30 (Conv2D)  (None, 7, 7, 30)         15390     

 reshape (Reshape)           (None, 7, 7, 5, 6)        0         

=================================================================
Total params: 844,926
Trainable params: 833,982
Non-trainable params: 10,944
_________________________________________________________________
/content/aXeleRate/axelerate/networks/common_utils/fit.py:129: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.
  use_multiprocessing = True)
/usr/local/lib/python3.7/dist-packages/imgaug/imgaug.py:184: DeprecationWarning: Function `ContrastNormalization()` is deprecated. Use `imgaug.contrast.LinearContrast` instead.
  warn_deprecated(msg, stacklevel=3)
90/90 [==============================] - ETA: 0s - loss: 2.9472/content/aXeleRate/axelerate/networks/yolo/backend/batch_gen.py:85: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return imgs_list, np.array(anns_list)

person 0.0000
mAP: 0.0000
Saving model on first epoch irrespective of mAP
Epoch 00000: Learning rate is 3.0458649045211895e-07.

90/90 [==============================] - 39s 362ms/step - loss: 2.9472 - val_loss: 1.2407
48-seconds to train
/usr/local/lib/python3.7/dist-packages/keras/backend.py:450: UserWarning: `tf.keras.backend.set_learning_phase` is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the `training` argument of the `__call__` method of your layer or model.
  warnings.warn('`tf.keras.backend.set_learning_phase` is deprecated and '
Converting to tflite without Reshape
WARNING:absl:Buffer deduplication procedure will be skipped when flatbuffer library is not properly loaded
/content/drive/MyDrive/Colab Notebooks/Sipeed_Maix_kmodel_trainer/person_dataset/person_detector/2022-04-15_03-30-53/YOLO_best_mAP.kmodel
/content/aXeleRate/axelerate/networks/common_utils/ncc/ncc compile "/content/drive/MyDrive/Colab Notebooks/Sipeed_Maix_kmodel_trainer/person_dataset/person_detector/2022-04-15_03-30-53/YOLO_best_mAP.tflite" "/content/drive/MyDrive/Colab Notebooks/Sipeed_Maix_kmodel_trainer/person_dataset/person_detector/2022-04-15_03-30-53/YOLO_best_mAP.kmodel" -i tflite --weights-quantize-threshold 1000 --dataset-format raw --dataset "/content/aXeleRate/axelerate/networks/common_utils/tmp"
1. Import graph...

2. Optimize Pass 1...

3. Optimize Pass 2...

4. Quantize...

  4.1. Add quantization checkpoints...

  4.2. Get activation ranges...

  Plan buffers...

  Run calibration...

  [>                                                 ] 0% 0s

  [=====>                                            ] 10% 0.313s

  [==========>                                       ] 20% 0.634s

  [===============>                                  ] 30% 0.944s

  [====================>                             ] 40% 1.28s

  [=========================>                        ] 50% 1.579s

  [==============================>                   ] 60% 1.882s

  [===================================>              ] 69% 2.185s

  [========================================>         ] 80% 2.493s

  [=============================================>    ] 89% 2.8s

  [==================================================] 100% 3.128s

  [==================================================] 100% 3.128s

  4.5. Quantize graph...

5. Lowering...

6. Optimize Pass 3...

7. Generate code...

  Plan buffers...

  Emit code...

Working memory usage: 7352 B

SUMMARY

INPUTS

0   Input_0 1x3x224x224

OUTPUTS

0   StatefulPartitionedCall:0   1x30x7x7

0
Converting to tflite without Reshape
WARNING:absl:Buffer deduplication procedure will be skipped when flatbuffer library is not properly loaded

Is this legacy repo no longer compatible with the Keras training modules/syntax?

[AIWintermuteAI]

There is some chance it would be upstreamed by Maixpy, but if it is going to make it to MARK firmware you need to try reaching out to Seeed. For Maixpy, leave your comment about upstreaming YOLOv3 in https://github.com/sipeed/MaixPy/pull/451. From the logs, I do not see any problems, except that you train the model for just one epoch (actual epochs parameter in config). Try setting to 30-50 and mAP should rise.

[AIWintermuteAI]

Oh, and about inference images - check your project folder "saved_folder": F"/content/drive/MyDrive/Colab Notebooks/Sipeed_Maix_kmodel_trainer/person_dataset/person_detector",

[alexjones85]

Yeah I've checked and it's same thing training for 100 epochs. There's definitely no inference images there. And they do appear after training with the master branch of repo.

[AIWintermuteAI]

Okay, I'll check the issue about inference images. But does mAP go up?

[alexjones85]

Yes the mAP improves:

Still no inference images even though the cell seems to run fine:

[alexjones85]

The best model model with mAP of 1.0 was achieved at epoch 53 so model was not saved after that. The conversion of the kmodel and tflite format seem to work fine. At least both files are there. I will try the kmodel in the MARK and see if it works anyway.

Is the Summary section suppose to look like that?

[alexjones85]

Yes it works!!

Just not sure how to upload the script to the board so it works without being plugged in via serial port? I've tried saving file to board and transfer file to board from MaixPy IDE. I also tried just uploading script to SD card with filename boot.py and main.py, but it still just displays the TinkerGen logo when turning the device on.

Can you tell me how to get the device to automatically run the script when turning it on?

[AIWintermuteAI]

So, the best way to have a script run on reboot is to save it as user.py with MaixPy IDE (Tools -> Transfer file to the board, file should be named user.py). Burn the model directly to flash - theoretically running model from SD card should work, but in practice I am not sure what were the latest code changes. One more things about debugging problems - you mention

it still just displays the TinkerGen logo when turning the device on

you can see what the problem is by connecting to the device with Serial Monitor (Tools -> Open Terminal - > New Terminal, default baud rate (115200), the port is specific to your OS and machine) and restarting MARK - it will show the error message on Terminal.

[alexjones85]

Great, that works. Thanks for all your help!

[AIWintermuteAI]

Excellent! I'll close the issue then! If that was helpful, consider buying me a coffee as a token of support :) https://www.buymeacoffee.com/hardwareai

[stnridy]

Hey, I try to use theegacy-yolov2 branch

%load_ext tensorboard
#we need imgaug 0.4 for image augmentations to work properly, see https://stackoverflow.com/questions/62580797/in-colab-doing-image-data-augmentation-with-imgaug-is-not-working-as-intended
!pip uninstall -y imgaug && pip uninstall -y albumentations && pip install imgaug==0.4
!pip install --upgrade --no-cache-dir gdown
!git clone -b legacy-yolov2 https://github.com/AIWintermuteAI/aXeleRate.git
import sys
sys.path.append('/content/aXeleRate')
from axelerate import setup_training, setup_inference

and below is my config

config = {
        "model":{
            "type":                 "Detector",
            "architecture":         "MobileNet7_5",
            "input_size":           224,
            "anchors":              [[[0.76120044, 0.57155991], [0.6923348, 0.88535553], [0.47163042, 0.34163313]]],
            "labels":               ["licence"],
            "obj_thresh" :          0.7,
            "iou_thresh" :          0.5,
            "coord_scale" :           1.0,
            "class_scale" :         1.0,
            "object_scale" :          5.0,
            "no_object_scale" :     1.0
        },
        "weights" : {
            "full":                   "",
            "backend":              "imagenet"
        },
        "train" : {
            "actual_epoch":        100,
            "train_image_folder":   "/content/MyDrive/MyDrive/ALPR/im",
            "train_annot_folder":   "/content/MyDrive/MyDrive/ALPR/images",
            "train_times":          1,
            "valid_image_folder":   "/content/MyDrive/MyDrive/ALPR/validation",
            "valid_annot_folder":   "/content/MyDrive/MyDrive/ALPR/validation_annot",
            "valid_times":          1,
            "valid_metric":         "recall",
            "batch_size":           4,
            "learning_rate":        1e-4,
            "saved_folder":         F"/content/MyDrive/MyDrive/ALPR/ww44p3_result",
            "first_trainable_layer": "",
            "augmentation":               True,
            "is_only_detect" :        False
        },
        "converter" : {
            "type":                 ["k210"]
        }
}

However, I am seeing the following error. anyone have any idea?

[AIWintermuteAI]

Looks like something related to upgraded packages in Colab. Create an issue in axelerate repository and I'll have a look if I have time?

[stnridy]

Hey, seems like theres no box for me to fill in the issue details here?

[AIWintermuteAI]

There might be an issue with your browser. https://github.com/AIWintermuteAI/aXeleRate/issues

[stnridy]

yeap I saw the same but when I click new issue, it return to the google tab one. Ask a friend to try and seeing the same also. Is it not like that in your case?

[AIWintermuteAI]

Okay, it seems GH templates syntax has changed. I fixed it - you can file an issue now.