search

Shared-Reality-Lab / IMAGE-server

IMAGE project server components
Other
2 stars 7 forks source link

Problematic graphic type: collages #101

Closed jeffbl closed 1 year ago

jeffbl commented 3 years ago

Am testing out the extension, and went to wikipedia to get some random images. Searched for "Montreal" and got this graphic:

https://upload.wikimedia.org/wikipedia/commons/9/9f/Montreal_Montage_2020.jpg

This is a montage of six separate photos around Montreal, but the preprocessors seem to try and interpret it as an image of one scene, finding a random set of things that to my ears bears almost zero correspondence to the actual content. I thought I was just selecting one of the subimages since I was using mouse right-click.

I don't know how common these are, so I don't know how high priority this should be, but a few potential options to consider:

  1. detect these and just say we can't handle them
  2. try and break it down into the component images (e.g., this one has white borders separating each individual image, but this isn't likely true of all montages) and process each separately
  3. have some browser-based UI to subselect individual images out of the montage (this sounds like it would be fairly complicated)

Assigning to members of UX and ML teams since this may require some design, as well as implementation. Also something to watch for in user feedback to see how common this case is.

Cybernide commented 2 years ago

I think the first step is to make a montage a detectable type, right? Can our classifier handle these? - I think how we handle montages follows from there.

Sabrina-Knappe commented 2 years ago

I would say breaking down the montage into a set of images and IDing those would already be much better than what people currently have access to

Sabrina-Knappe commented 2 years ago

So basically if we can recognize these, then I would go to option 2.

Cybernide commented 2 years ago

Is there any way we could use heuristics to determine whether an image is a montage? As I said, unless we can actually detect montages, I think pretty much all other points are moot.

Cybernide commented 2 years ago

@jeffbl Suggest we move this to backlog - without a clear way of even detecting montages, any solutions we have for handling them are pretty much moot. Seems like it's probably going to be some work to investigate their detection as is.

jeffbl commented 2 years ago

Agree, not critical, especially now. We should reconsider for final milestone, in context with any other similar issues we may want to detect.

Cybernide commented 2 years ago

Assigning to @gp1702 We need to figure out how to detect montages before we can deal with/analyze them

jeffbl commented 2 years ago

Assigning to @gp1702 since that's what I think @Cybernide intended here?

jeffbl commented 1 year ago

@AntonioShen Moving to Nov14 sprint, but if you aren't going to be working on it, I'll move to backlog instead - just let me know.

jeffbl commented 1 year ago

Moving to backlog.

jeffbl commented 1 year ago

Now that we have a collage detector (#569), I've logged #646 to integrate it into the photo handler, so am closing this.