Simple OpenCV Tracking Application

[alyhasansakr]

A simple pedestrian tracking application (or anything that moves) that uses OpenCV functions (CPU).

The application is divided into two parts: client and server, each part is a Docker application.

The client application reads video input (from a file) and is capable of doing all of the processing without the server. The client should provide the option to connect to a server or to just run locally. If connected to a server, the client should encode video frames before sending it to the server, then wait to receive the output from the server (detected objects).

The server application waits for a client connection, once connected it receives video frames from the client, decode it, run image processing on it then sends the result back to the client (detected objects).

This is the core implementation of the application, we don't need any connection to the master or a logger. Options such as client mode (whether to run locally or send to the server) should just be application parameters for now.

Image processing function has to be implemented in a separate class that is used in both client and server. Other than that you are free to implement any number of classes.

Don't forget about the unit tests. Also a manual test should be possible using Docker.

The original application can be found here, but you don't need to follow it exactly, it is really not good for what we want to do here.

[theodorelu27]

@alyhasansakr before starting to work with it, I want to confirm the requirements for this issue:

1. Create a Client application
   • can read video input
   • can switch between local and server for client modes
   • can encode video frames
   • can send the encoded frames to the server (POST Request)
   • can connect to a server
2. Create a Server application
   • can decode incoming video frames
   • send result back to the client (POST Request)

other / common traits:

• For the scope of this issue, we will only connect client with a known server, connection should nevertheless be abstracted
• The Client and Server should communicate with REST
• Both should have unit tests
• Both should have manual to test in Docker (put it in Wiki?)
• We would try to first implement this in Java, but if it's not possible, implementation with C++ is preferred

things I want to confirm:

• Can both the client and server each did more than one processing job?
• If a processing has already started in local and a server is connected, should it finish the current processing or should it move the processing to the server right away?

For the estimates, this is my first time doing this kind of application so I'd give it until Wednesday to confirm if it's viable to do in Java and finish it on Saturday, is it a realistic or otherwise estimate or is too long?

[alyhasansakr]

First of all, I like your approach of identifying the requirements.

Let me answer the question first:

• Both client and server should not have more than one application instance, that is one video input and one detections output, both of them however, may have more than one thread to handle this instance. In other words, we use microservice model instead of a standard multi-threaded server.
• Changing between local processing and remote processing should not interrupt the execution of the current video frame.

There is something else I notice here which may not be ideal, REST may not be the best option to send a video frame (or the server response), the delay is too much and it doesn't make use of the other benefits of REST. If you don't agree you may try yourself and see, otherwise think of something else or just use a UDP or TCP socket.

And don't worry about time, we can always push the dates here, the important thing is that we do it the right way.

[theodorelu27]

@alyhasansakr thank you I'm still not familiar with multi-threading, but I guess it is safe to say that at one time, one node can process one video, correct?

Alright, I'll keep that in mind about the protocl used for the streaming

By the way I still have problems installing OpenCV since they didn't really maintain the Java side of the project that much and their documentation still refers to the version 2.4 of the application when it was first included, I think I'm going to follow this steps because so far, they are the only one providing tutorial with a more recent version of OpenCV https://elbauldelprogramador.com/en/compile-opencv-3.2-with-java-intellij-idea/ For the Ant however, I currently have JDK 14, JDK 15, and JRE 8 installed, but they are all missing the tools.jar that Ant needed, I'm trying to install JDK 8 to see if they have it, or have you encountered this problem before? edit: never mind, Ant is working now, I'd proceed with this and I'd let you know if I find other problems

As an alternative, I tried to implement this repository https://github.com/quickbirdstudios/opencv-android as a dependency in my gradle.build, but it's not being imported by IntelliJ

[alyhasansakr]

correct, one node one video.

I know that OpenCV with Java is not a common configuration, you can try it for one more day, if it doesn't work then switch to C++.

[alyhasansakr]

Instead of doing a pedestrian tracking application, just Aruco marker detection.

[alyhasansakr]

protentional resources: https://hub.docker.com/r/asuprun/opencv-java https://github.com/joshuamarquez/docker-opencv-java

[theodorelu27]

@alyhasansakr I made some progress with this docker image https://github.com/eedkevin/opencv-java8 which is baed one https://github.com/joshuamarquez/docker-opencv-java only it's packaged with JDK8

I tried some sample codes to generate a Mat object and print its content, I don't know if it's relevant to proving whether OpenCV Java could work well with docker

the problem is these images came with the vanilla build for OpenCV, so Aruco's library was not included currently I'm trying to build my own image based on the references, on Ubuntu 16.04 with JDK8, but somehow apt-get was unable to locate ant and git

I've attached the Dockerfile that I used to build the image, I separated the install script like they did because I find it somewhat cleaner, I dumped all the build stuffs instead of using variable like in the reference because they kept on throwing unexpected "(" error

`FROM ubuntu:16.04

MAINTAINER akashafh20 "theodore.lumaksono@yahoo.com"

COPY install.sh install.sh RUN chmod +x install.sh && sync && ./install.sh && rm install.sh

VOLUME /gen WORKDIR /gen CMD bash`

`apt-get update && apt-get install -y build-essential cmake curl openjdk-8-jdk libgtk2.0-dev pkg-config libv4l-dev libavcodec-dev libavformat-dev libswscale-dev python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev ant git

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

export ANT_HOME=/usr/bin/ant

mkdir opencv_bin && cd opencv_bin mkdir modules

git pull https://github.com/opencv/opencv.git git pull https://github.com/opencv/opencv_contrib.git

cp -R opencv_contrib/modules/aruco modules

mkdir build && cd build

cmake -DOPENCV_EXTRA_MODULES_PATH=/opencv_bin/modules /opencv_bin/opencv -D BUILD_SHARED_LIBS=OFF .. make -j8

mkdir /opencv-java-bin

mkdir /gen`

could it probably be a memory problem? I'm mostly working at 90% usage but when I tried to close some processes, at 70% it's still not working I tried to search and there's one mention that I might be running out of space, so I tried docker prune, but it's not working either

[alyhasansakr]

@theodorelu27 create the docker image without the CMD part, attach to it and execute each command one by one to make sure you are not doing anything wrong, also to check the space and memory.

[theodorelu27]

@alyhasansakr I've manually built the image, apt-get update took a pretty long time and currently I'm running the make without any option so building took some time, but I got it to work However, I still can't get the apt-get to work during Docker build although the very same command is working when ran from the container

I've also made some test code to create, read from image, detect, and draw detected markers

Should I include them in a pull request before proceeding? Also since for now we changed the goal to Aruco marker detection, would the connection between client and server still be within the scope of this issue?

[alyhasansakr]

@theodorelu27 So it works if you do it manually but not if it is within dockerfile, that is strange. I want to get something out of the way first, why do you use CMD instead of RUN for build steps? CMD should only be used to run the final application (the application that we develop). If nothing helps, then we talk next Saturday.

We still need a client and a server, and the client has to send the image to the server where the server detects the codes and send the coordinates back to the client, so nothing change here.

Make a pull-request for what you already have.

[theodorelu27]

@alyhasansakr CMD is a carry over from the sample code that I used as a base for the Dockerfile, so I can change it.

Fow now, I'd proceed with the client-server modules while trying to make sure it can be built from the Dockerfile