Doesn't work in newest versions of unity. Bounding boxes are broken.

[derekmtran]

I downloaded the source and opened it in Unity 2018.3.6. I downloaded the TFSharp 0.3 plugin and imported it in. I tried it on both iOS and Android. The bounding boxes are very out of place.

On iOS, the bounding box is just shown as a small dot in the middle of the screen. (iPhone XS Max)

On Android, The bounding box is larger than the screen size (so much so that the label is completely off screen) most of the time. It never shows a correct bounding box.

I'm not sure if this is helpful, but the camera views only take up a fraction of the screen.

The same happens on Editor Mode. (For editor mode, I comment out the check in the code for back-facing camera so that I can use my webcam) The detection just shows a dot in the middle of the camera view.

On Android, I get a lot of errors about duplicate files loaded into ML-Agents/Plugins/Android. I have to deleted those duplicates to get it to run. I don't think this is the problem though.

Have any of the dependencies changed? I cloned it multiple times and tried various versions of TFSharp and different models. I also triple checked the instructions in the documentation on settings, like ENABLE_TENSORFLOW flag in Player Settings.

iOS:

Android:

[Syn-McJ]

Hi @derekmtran, Thanks for reporting, I'll check newest Unity version as soon as I can.

[derekmtran]

Thank you! Let me know if there is anything I can provide on my end to help investigate. I’m hoping to do a mini proof of concept project in the next day or two with object detection with Tensorflow in unity and this is the closest thing I’ve found to getting me started. Appreciate you putting up the code and helping everyone out!

[JolofNar]

I have had a similar problem. Fixed the bounding box issue by setting backgroundSize = new Vector2(Screen.width, Screen.height); I'm still not 100% on how to fix the screen being tiny in the middle, but am currently multiplying the scale for background by ratio.

Hope this helps

[derekmtran]

Where did you add that line? Which platform did you test on?

[JolofNar]

In the script PhoneCamera.cs in the start method where it sets backroundSize, I changed the the original value for that. I recall it was because it was using the camera height and width originally, and that limits it to the screen size. platform was android Samsung S8.

[derekmtran]

Interesting. I'm using Samsung S8+ on Android. Now I'm just getting bounding boxes that are approximately the same size as the phone screen (pretty much a fullscreen box) and it just says "Person" on just about everything. Did you have to change anything else about the project? I tried both 0.3 and 0.4 plugins. Any chance you can put up your unity project somewhere for me to try and run locally?

[JolofNar]

I'm awfully new on here so not sure how to. I can copy and paste the PhoneCamera.cs its the only thing i have changed. I have moved a little bit of stuff out of update as I was only interested in portrait and tried to resize the little view box, by multilplying the scale by the ratio set for aspect ratio(that is imperfect, but better than it was) . I tried a few other things as well there, but none got it to quite the right size, and you're the first person I have found having the same issue as me. I have also had pretty poor performance, but it does seem to find things like laptops, tvs and of course people. It also seems to think mobile phones are remotes. I saw some people suggesting quantizers(?) to improve performance, but I'm pretty new to ML.

[JolofNar]

Here is my Start Method, its the only thing I have changed (pretty much) Other than commenting out the update method entirely.

private void Start() {

    LoadWorker();

    defaultBackground = background.texture;
    WebCamDevice[] devices = WebCamTexture.devices;

    if(devices.Length == 0)
    {
        this.uiText.text = "No camera detected";
        camAvailable = false;

        return;
    }

    for(int i = 0; i < devices.Length; i++)
    {
        if(!devices[i].isFrontFacing)
        {
            this.backCamera = new WebCamTexture(devices[i].name, Screen.width, Screen.height);
        }
    }

    if(backCamera == null)
    {
        this.uiText.text = "Unable to find back camera";

        return;
    }

    this.backCamera.Play();
    this.background.texture = this.backCamera;

    //this.backgroundSize = new Vector2(this.backCamera.width, this.backCamera.height);
    backgroundSize = new Vector2(Screen.width, Screen.height);
    camAvailable = true;

    float ratio = (float) backCamera.width / (float) backCamera.height;
    fitter.aspectRatio = ratio;

    float scaleY = backCamera.videoVerticallyMirrored ? -ratio : ratio;
    background.rectTransform.localScale = new Vector3(ratio, scaleY, 1f);

    int orient = -backCamera.videoRotationAngle;
    background.rectTransform.localEulerAngles = new Vector3(0, 0, orient);

    Debug.Log($"PressedNow ratio: {ratio}");
    string func = mode == Mode.Classify ? nameof(TFClassify) : nameof(TFDetect);
    InvokeRepeating(func, 1f, 1.0f);
}

[derekmtran]

The best way is probably to upload it to your own GitHub repo, but it sounds like you didn't change much. If you are still willing to upload the entire project somewhere, though, I can take a look at your build settings and what not and see if I have anything majorly off. It sounds like your tracking is much better than what mine is doing out of the box. Hm... which plugin are you using? 0.3 or 0.4? Did you also have to change anything when you initially launched the project in Unity? After I imported and tried to run, I got the following errors (duplicate errors), so deleted the files highlighted on the right to get it to build. Did you have to do this as well?

[JolofNar]

As far as I can see I have all of those still in there. I am pretty certain i built 0.4 as I wanted to look at making my own model. I feel like I may have had a problem at first importing things as I hadn't changed the player settings for .Net 4, and I think I reimported again after doing that to make sure I had everything

[derekmtran]

What version of Unity? And what OS?

[JolofNar]

2018.2.12f1 on windows

[derekmtran]

I wonder if running on Windows has anything to do with it (I'm running on Mac)... I'm going to keep digging around and see what I can find. Thanks, so far!

[Syn-McJ]

Hi @derekmtran, I checked the new Unity version on Windows and had this error as well. Deleting duplicated assembles seems to be the way to go and it isn't the problem.

The problem is that background texture doesn't stretch to the whole screen anymore. Something must have been changed in the new Unity and I'm not quite sure how to fix this. If you're in a hurry you might have better luck asking how to stream camera feed correctly on the Unity forum or stackoverflow.

I'm gonna try to fix this myself next week, but given that I'm no Unity expert it will most likely take me a while. If you find a solution quicker, be kind to share it here.

Drawing bounding boxes is just a matter of right scaling. If you manage to fix camera issue and bounding boxes are still wrong, you can simply compare what coordinates does the model produce and how should they be scaled on the screen to create a correct rectangle.

[JolofNar]

Further to all of this, I have found using ARCore and getting the texture2d through there gives much better results. I've not used python so I can't compare it to that version of tensorflow, but would be really interested in comparison.