How can I add a button

[tcapelle]

Great tool! I want to add a button to perform inference on another model (parallel to dextr) to compare the generated mask. How can I do this? I know how to perform the inference, but don't know how to add a button on this template thing. This model is a simple Unet, so I only need to pass the full image, without any point selection. Sorry, I am pretty new to front-end. I want to do this on the flask version. I tried modifying the labeller_app.html file adding:

    <button class="btn btn-sm btn-primary w-25 mb-1" type="button" id="unet_button" data-toggle="tooltip"
        data-placement="top" title="Unet labelling algo">Unet</button>

but I don't know what I would need to do on the flask app to recover the click form this button and implement my custom callback.

[Britefury]

Thanks for your compliments!

It would require a bit of work... Do you have tools set up for Typescript development? Personally I use PyCharm Professional that compiles Typescript to Javascript for me. I haven't attempted to use Node.js for Typescript development at all.

The DEXTR button callback can be found here: https://github.com/Britefury/django-labeller/blob/b3ca96aeeb987cc3f413219213f1167c34d3f03f/image_labelling_tool/static/labelling_tool/main_labeller.ts#L803-L807

Given that you would not be developing a new tool with UI/interaction etc, I would suggest a simple button that sends a request to the flask server to perform the inference. The server should return the result as a vectorized label in the form if a list of polygons that form the regions. These should be inserted as a polygonal label. I'm going to go through the steps of working within the existing design of the tool.

1. Add a field to store the U-net callback function

We want to pass the labeller a callback that it can invoke to perform this inference step. Let's add a field below the dextr callback field which can be found at: https://github.com/Britefury/django-labeller/blob/b3ca96aeeb987cc3f413219213f1167c34d3f03f/image_labelling_tool/static/labelling_tool/main_labeller.ts#L136-L138

So something like:

private _unetCallback: any;

2. Add a parameter to the constructor to receive the U-net callback, then assign the received callback to the callback field we added in step 1

You would want to modify the constructor so that it accepts the callback as a parameter. It is at: https://github.com/Britefury/django-labeller/blob/b3ca96aeeb987cc3f413219213f1167c34d3f03f/image_labelling_tool/static/labelling_tool/main_labeller.ts#L182-L187 Let's add a U-net callback, so that it reads:

constructor(schema: LabellingSchemaJSON, tasks: TasksJSON[],
                    anno_controls_json: AnnoControlJSON[],
                    images: ImageModel[], initial_image_index: number,
                    requestLabelsCallback: any, sendLabelHeaderFn: any,
                    getUnlockedImageIDCallback: any, dextrCallback: any, dextrPollingInterval: number,
                    unetCallback: any, config: any) {});

Now assign the field that we created. Below: https://github.com/Britefury/django-labeller/blob/b3ca96aeeb987cc3f413219213f1167c34d3f03f/image_labelling_tool/static/labelling_tool/main_labeller.ts#L371-L374

Add:

this._unetCallback = unetCallback;

3. When our U-net button is clicked, send a request to the server. When the server replies, add the labels that it gives us

Above I showed you where the DEXTR button callback lives. Here, we will implement click handler for the new U-net button:

var unet_button: any = $('#unet_button');
unet_button.click(function (event: any) {
    var image_id = self._get_current_image_id();
    // invoke the U-net callback provided to the constructor.
    self._unetCallback(image_id, function(unet_result) {
        // Handle the U-net result
        // Let's assume that the result consists of a JSON object whose 'regions' attribute is an nested array of points.
        if (unet_result.regions !== undefined && unet_result.regions.length > 0) {
            // Create a new polygonal label model. We use the currently selected label class, and indicate that it was automatically
            // generated using a U-net
            var model = new_PolygonalLabelModel(self.get_label_class_for_new_label(), "auto:unet");
            model.regions = unet_result.regions;
            var entity = self.root_view.get_or_create_entity_for_model(model);
            self.root_view.add_child(entity);
            self.root_view.select_entity(entity, false, false);
        }
    });
    event.preventDefault();
});

4. Implement the U-net callback in the flask template. This will be a Javascript function that Django-labeller invokes that sends the request onto the server using an AJAX POST request.

Okay, now we need to implement the unet callback in the Flask template. The DEXTR callback is at: https://github.com/Britefury/django-labeller/blob/b3ca96aeeb987cc3f413219213f1167c34d3f03f/image_labelling_tool/templates/labeller_page.jinja2#L98-L120

// U-net callback function
// Note that our parameters -- the image ID and a callback that is invoked when the server replies -- are the parameters
var unet_callback = function(image_id, on_response) { 
   // Create the POST data that provides the image ID
   var post_data = { 
       image_id: image_id 
   }; 

   // Send request to server
   $.ajax({ 
       type: 'POST', 
       url: '/labelling/unet', 
       data: post_data, 
       success: function(msg) { 
           // The server has replied; if the message has a `unet` attribute, invoke `on_response`, that is the function that we
           // define in the block above that starts with ` self._unetCallback(image_id, function(unet_result) {...`
           if (msg.unet !== undefined) {
               on_response(msg.unet);
           }
       }, 
       dataType: 'json'
   }); 
}; 

Further down the labelling tool is constructed: https://github.com/Britefury/django-labeller/blob/b3ca96aeeb987cc3f413219213f1167c34d3f03f/image_labelling_tool/templates/labeller_page.jinja2#L123-L138

We need to pass our callback to the labeller constructor:

// Create the labelling tool 
         // Give it: label classes, dimensions, image descriptors, initial image ID and the callbacks above 
             var tool = new labelling_tool.DjangoLabeller( 
                 {{ label_class_groups | tojson | safe }}, 
                 {{ tasks | tojson | safe }}, 
                 {{ colour_schemes | tojson | safe }}, 
                 {{ anno_controls | tojson | safe }}, 
                 {{ image_descriptors | tojson | safe }}, 
                 {{ initial_image_index | safe }}, 
                 get_labels, 
                 set_labels, 
                 null, 
                 dextr_request, 
                 null, 
                 unet_callback,
                 {{ labelling_tool_config | tojson | safe }} 
             ); 

5. Handle the request in the Flask server

Now add a new URL to the flask server. Our DEXTR one lives at: https://github.com/Britefury/django-labeller/blob/b3ca96aeeb987cc3f413219213f1167c34d3f03f/image_labelling_tool/flask_labeller.py#L199-L219

Lets make a similar one for U-net:

@app.route('/labelling/unet', methods=['POST'])
def unet():
    image_id = request.form['image_id']
    image = images_table[image_id]
    regions_js = apply_unet_js(image)
    unet_response = dict(unet=dict(regions=regions_js))
    return make_response(json.dumps(unet_response))

Note that the above requires a apply_unet_js function, that you will need to implement. Please follow apply_dextr_js to get an idea as to how to go about doing this. I hope that gets you going! :)

[tcapelle]

wow man, what a wonderful answer. Wow, i did exactly as you described, compiled the TypeScript (without knowing nothing about ts or js, just doing tsc main_labeller.ts and IT WORKED!!!!)

Note: My Unet is so much slower than DEXTR...

[Britefury]

Excellent! I'm really glad it worked! :)