Distributed, collaborative labelling

[martarho]

I find labelling a very time consuming task, that's while a usual approach is to onboard more people to create both bigger datasets and consensus on labelled objects.

A way we could unite this with the current development of the tool is via creating a sort of "task" distribution, where a subset of the data can be added to a session and this sent to a specific labeler. That would free the need to have a user system (keep it lean and simple) and still distribute the tasks around. Could be potentially be set up through an API (but that's another feature request).

[SkalskiP]

Hi, @martarho! Could you describe your idea a little more precisely how these tasks would work? How would they be assigned? Is it possible without creating a backend?

[martarho]

Well, without a backend it's kind of hard, but let me explain a simplified version of what I meant and maybe that could be more feasible.

Disclaimer: I'm no frontend/backend developer.

Now, a way to simplify the request and adapt it to a front-end only project would perhaps be having makesense have an integration with an external API, so the queing/tasking end is on the user and makesense just handles the labelling interface.

Configuring the API connection would be done at installation level, where a config file is edited to add the API endpoint + set up the "API" mode for make-sense.

Then, I would imagine make-sense having some sort of ingestion/connection where makesense requests and receives the labelling task information where the project name and label list are received as well as a url with the image. For instance, something like this:

{ "unique_task_id" : "XXYYZZ", "project_name" : "test_project_remote", "label_list" : ["A","B","C"], "image" : "https://i.ytimg.com/vi/MBtJdkiEhBk/maxresdefault.jpg" }

This would allow it to render the labelling interface. Finally, the exporting step would be simplified to a simple api call to the custom backend, which then would handle the label information in whatever way its needed.

I find potential in having such an integration, as it would allow external developers to make more sophisticated ways to develop labelling projects while still using the amazing capabilities makesense has. What do you think?

[skysuka]

it's a great idea, i have been working on this recently~

[SkalskiP]

@martarho and @skysuka I agree with you - enabling joint work on labelling is a powerful functionality. It seems to me, however, that doing something like this without a dedicated backend is difficult / not possible. The @martarho (I'm sorry I didn't reply faster - I didn't notice the answer.) proposal in fact requires a backend. The only difference is that the backend is not provided by MakeSense, only the potential user has to write it himself.

I have an idea to create a full package - docker image, which will contain both backend and database and frontend. However, this is a time-consuming idea. But it would open up many possibilities.

[rasyidf]

What if, we just add GitHub login capability, and allow users to sync settings, and their project on GitHub repository. isn't it collaborative enough?

[rasyidf]

Perceptilabs already implemented this by using github repo as their project storage

[SkalskiP]

Hello, once again @rasyidf 👋 . But is it possible only with the front end or would we need to implement the backend for that? I'm looking for a solution, where we would not need any backend. I do not know if it is possible to use git using js in your brawser. I'd need to read about it.

[rasyidf]

Correct me if I'm wrong, but if what you referred as front end is this repo, then of course yes, we can add Oauth2 inside the react. see : medium - use github oauth as sso react

and then use github gist as bases of our storage. with api token. see : gist api docs

[SkalskiP]

@rasyidf of so you suggest we would use gists not repost to support collaboration?

[rasyidf]

integrating github repo would bring more work to be done. especially on creating commit and managing collaborators.

gist will only save the Metadata, not images but it's faster and easier to share.

but if the project-based system is implemented. then we can do it manually just like normal git folder.