Handling MPO images in separate_detections_into_folders

[aweaver1fandm]

We ran into an issue where separate_detections_into_folders.py was crashing when run with the --render_boxes flag. The error was essentially that you could not use quality=keep when original image is not JPEG. All the images we were processing had .JPG as the extension but when we did some digging we found that pil_image.format was MPO and not JPEG.

In our case the trail cameras were capturing stereoscopic images and saving them as .JPG. According to the Pillow documentation, it both opens and saves such images by using the first image in the sequence, which explains why the detector works fine. However, the quality flag currently being used to save images (starting at line 386 in separate_detections_into_folders.py) only applies to true JPEG images.

We propose the following changes to that section of code:

       # Write output with EXIF metadata and quality='keep'
        # but only if the image is format JPEG.  Other image formats such as MPO
        # will fail if quality flag is used
        if exif is not None:
            if pil_image.format == "JPEG":
                pil_image.save(target_path, exif=exif, quality='keep')
            else:
                pil_image.save(target_path, exif=exif)
        else:
            if pil_image.format == "JPEG":
                pil_image.save(target_path, quality='keep')
            else:
                pil_image.save(target_path)

With this change in place, the code ran fine and appropriately produced the bounding boxes in the MPO images. Since we are not intimately familiar with the full extent of code in the project, similar changes may need to be made elsewhere.

[agentmorris]

[edited after reading the OP's comment more carefully re: PIL's handling of MPO]

Good catch, we'll definitely fix this. I'd like to be able to verify the fix, though, and I don't have an MPO camera trap image to test with. Would you mind sending links to a couple of images here (maybe one or two with animals, one or two without), or sending to info@lila.science?

Also is it accurate that you're OK with PIL's behavior re: only using the first image? If you're just using these MegaDetector results to sort images, then you'll go back to the original files for stereo analysis, this doesn't matter, but otherwise it seems like you went to a lot of trouble to capture stereo images and you're losing that information.

More generally, just out of curiosity, I'm super-interested in why one captures stereo camera trap images. If you don't mind, it would also be great if you could share a bit more about your use case!

Thanks!

[aweaver1fandm]

Some sample images are attached. I have attached both the original and the result we got including one image that had nothing in it.

As far as specifically MPO images, we don't specifically need to keep all images that are part of the stereoscopic image. We're not expecting, nor do we need multi-image format to be properly preserved.

As for why we're capturing stereo images in the first place, the impression I get from the actual researcher doing this work, is it's completely accidental. They happen to be using some cameras which capture stereo images but get downloaded as JPEG file. The files open just fine when you click them so the researcher never realized there was a difference. We only found the difference because the bounding box code failed. We run megadetector on Linux and even when we did something like: file myfile.jpg, on the command-line it said the file is JPEG. It wasn't until we printed out pil_image.format in the python code that we saw it was in actually in MPO format.

[agentmorris]

All clear, thanks. I'm a little suspicious that the format is properly preserved when rendered on GitHub; can you send links to the original images, or attach to email to info@lila.science? Thanks!

[agentmorris]

The OP shared images offline, I was able to reproduce and fix, using exactly what the OP suggested. I verified that everything still works fine for .jpg images. Thanks for finding this!

(Also, what is that animal? A raccoon? A possum? An undead fox?)

[aweaver1fandm]

Dan,

I think it's a possum

On Fri, Mar 31, 2023 at 9:01 PM Dan Morris @.***> wrote:

The OP shared images offline, I was able to reproduce and fix, using exactly what the OP suggested. I verified that everything still works fine for .jpg images. Thanks for finding this!

(Also, what is that animal? A raccoon? A possum? An undead fox?)

— Reply to this email directly, view it on GitHub https://github.com/microsoft/CameraTraps/issues/336#issuecomment-1492768810, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANJXBC7QAAH53TZYVFJFGMDW655ABANCNFSM6AAAAAAWMAPRTM . You are receiving this because you authored the thread.Message ID: @.***>