search

WordPress / openverse

Openverse is a search engine for openly-licensed media. This monorepo includes all application code.
https://openverse.org
MIT License
237 stars 187 forks source link

Incorporate Rekognition data into the catalog #431

Open obulat opened 1 year ago

obulat commented 1 year ago

Summary

Rekognition data in the form of object labels was collected for roughly 100m records in the Openverse catalog.

These labels should be sanitized for suitability in the Openverse project and applied to records in the Openverse Catalog as tags.

Description

Some exploratory work was done to assess the quality of these labels. The team generally felt positive about them, given we would blanket remove a subset of them (e.g. ones that assume a gender). We will need to do a broader analysis to determine if there are more labels we would want to exclude, and then incorporate them into the existing tags for each record in the catalog. The automated tags include a confidence score associated with the tag value, and we should also incorporate those values into the overall document score for relevant searches.

Best guess at list of implementation plans:

Documents

Issues

Milestone

Incorporate Rekognition Data

zackkrida commented 1 year ago

Early Testing

Back in April I ran a simple script to do some basic analysis of the Rekognition labels. I mostly wanted to test the speed of reading all of the data.

Here's the script I used: https://gist.github.com/zackkrida/cb125155e87aa1c296887e5c27ea33ff

Infra setup

The script was run on a manually-provisioned EC2 instance. The instance was configured with permissions to access our S3 bucket. I also used an instance with Enhanced Networking support so the script would theoretically stream the rekognigtion data as fast as possible.

Unfortunately I only loosely recall how long it took, and am struggling to find my notes. I believe it was around 4-5 hrs. I do remember being happy with the speed.

General recommendations

For this project I would strongly recommend we download the full list of Rekognition labels from this page: https://docs.aws.amazon.com/rekognition/latest/dg/labels.html and filter out anything related to gender prediction.

As far as the approach we take to importing the rekognition data, we could probably use a script much like the one I wrote to stream the rekognition data and then perform sql updates in batches, adding the new tags to the existing array with a provider value of "Rekognition". We may also want to store the confidence of each tag in the Catalog DB. This would give us more flexibility in the future. We could fine tune tags in Elasticsearch, for example, and only choose to show those with a certain confidence level.

AetherUnbound commented 5 months ago

The project proposal has recently been merged, and issues for the 3 implementation plans have been created (linked above). I plan on starting the API-related IP soon.

openverse-bot commented 4 months ago

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

AetherUnbound commented 4 months ago

No change since the previous update - IPs still need to be drafted.

openverse-bot commented 4 months ago

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

AetherUnbound commented 4 months ago

The IP for the API-side of things has been merged (#4189) and can be seen here. The only issues necessary for this work has been created and will be worked on in the next week or so: #4273.

@fcoveram has also established mock-ups for how the machine-generated tags will be displayed in the frontend in #4192. This was a necessary prerequisite for the frontend IP, #4039, which @obulat will be working on.

Work can also begin on the final IP, #4040, which will be a more subjective dive into the tags themselves and what policy Openverse will take for machine-generated labels.

openverse-bot commented 3 months ago

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

AetherUnbound commented 3 months ago

Provider information is now present in the API responses under the unstable__provider key with the deployment of #4280.

The frontend IP is also currently in a revision stage and expected to be accepted soon: #4302

openverse-bot commented 3 months ago

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

AetherUnbound commented 3 months ago

The final IP for the incorporation of the tags into the catalog has been opened, and raised a number of good discussion points: #4417. We have a few things to iron out there as it relates to the data normalization project (#430) and the removal of the ingestion server (#3925).

openverse-bot commented 2 months ago

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

AetherUnbound commented 2 months ago

We've taken a pause on the Rekognition work since the discussion on it has prompted questions on the above. We've got some good clarification by way of #4465 and #4524, but for now I'm going to move this project to On Hold in the short-term while we resolve those discussions!

openverse-bot commented 1 month ago

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

AetherUnbound commented 1 month ago

This project was recently moved into In Progress (from On Hold) now that the project lead (myself) is back from AFK. Based on the merging of #4417, I've added a number of issues to the project milestone:

Work on many of these can begin immediately! Particularly #4642, which I may start on this week.

openverse-bot commented 1 month ago

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

AetherUnbound commented 1 month ago

Discussion around the label set to use (#4643) is ongoing, and the DAG work (#4645) will be picked back up this week.

sarayourfriend commented 1 month ago

@AetherUnbound when working on https://github.com/WordPress/openverse/issues/4643, it occurred to me that the bounding box information in the tags (and potentially the categories) might also be useful in the future. I just checked the project proposal and the ingestion IP, but I didn't see any clear determination about whether the bucket would be kept around after this work. Mostly just wanted to make sure that it would be, and that we aren't treating ingestion of the tags in the current mode to be the definitive end-all-be-all of that dataset's usefulness to us. Just asking for clarification that we won't delete that bucket, basically.

AetherUnbound commented 1 month ago

Oh definitely not - my intention was to keep the source data in perpetuity regardless! That was implicit in the lack of mentioning what would happen to the bucket, but it can be made explicit in the final IP if you would like me to make it so.

sarayourfriend commented 1 month ago

Maybe worth adding into the project proposal as a clarification about the outcomes of the project, but so long as it's recorded somewhere, I'm happy about that! We could also add an issue to add it to Terraform and move it to infrequent access, as we discussed in the issue related to #3810, to save money on storage of it for the foreseeable future, as well as documenting its existence and our long-term intentions with keeping it around :slightly_smiling_face: That issue wouldn't be part of the project's "shipped" status, though, to clarify.

AetherUnbound commented 1 month ago

Great points! I'll go ahead and make that issue and the adjustment to the project proposal.

openverse-bot commented 2 weeks ago

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

AetherUnbound commented 1 week ago

The reviewed Rekognition label list has been added to our documentation, and we now have an issue for filtering the tags during the data refresh which can be worked on: https://github.com/WordPress/openverse/issues/4813

The add_rekognition_labels DAG is complete as well 🎉 I intend to run it as soon as I can.