Rekognition `GetContentModeration` response differs from documentation

[ifellinaholeonce]

Describe the bug

I have recently started using the Rekognition API for video content moderation. I noticed that the responses I have been getting do not match the documentation in a couple of ways - the labels themselves use different words than the docs, and a number of fields on the object are nil.

I was able to find older documentation using Wayback Machine here that seems to better match the current behaviour. The labels properly align and the nil fields are not documented.

Is it possible the gem updated the struct for the responses but is not hitting the right version of the API or something? I haven't noticed anywhere that specifying an API version was necessary. Am I possibly simply missing a configuration setting?

Expected Behavior

struct Aws::Rekognition::Types::ContentModerationDetection should properly match documented behaviour found here and here.

Current Behavior

struct Aws::Rekognition::Types::ContentModerationDetection is missing some fields and uses undocumented labels. Notice in the below snippet in the moderation_labels array that each element is missing values for taxonomy_level, start_timestamp_milis, end_timestamp_milis and duration_milis. Also notice that some of the names of labels are "Tobacco", for example, which do not match any of the name in the docs. I would expect it to be either "Drugs & Tobacco" or "Drugs & Tobacco Paraphernalia & Use" in this case.

#<struct Aws::Rekognition::Types::GetContentModerationResponse
 job_status="SUCCEEDED",
 status_message=nil,
 video_metadata=#<struct Aws::Rekognition::Types::VideoMetadata codec="h264", duration_millis=13760, format="QuickTime / MOV", frame_rate=25.0, frame_height=1920, frame_width=1080, color_range="LIMITED">,
 moderation_labels=
  [#<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.2705247402191162, name="Smoking", parent_name="Tobacco", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.2705247402191162, name="Tobacco", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=1000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.0228523015975952, name="Smoking", parent_name="Tobacco", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=1000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.0228523015975952, name="Tobacco", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=1520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.377088189125061, name="Explicit Nudity", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=1520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.1258432865142822, name="Graphic Female Nudity", parent_name="Explicit Nudity", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=1520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.3413856029510498, name="Sexual Activity", parent_name="Explicit Nudity", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=1520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.4593212604522705, name="Smoking", parent_name="Tobacco", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=1520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.4593212604522705, name="Tobacco", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=3.6541855335235596, name="Explicit Nudity", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.030139207839966, name="Graphic Female Nudity", parent_name="Explicit Nudity", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=3.6497228145599365, name="Sexual Activity", parent_name="Explicit Nudity", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=5.710151672363281, name="Smoking", parent_name="Tobacco", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=5.710151672363281, name="Tobacco", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.7081072330474854, name="Explicit Nudity", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.7058758735656738, name="Sexual Activity", parent_name="Explicit Nudity", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.3264174461364746, name="Smoking", parent_name="Tobacco", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=2520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.3264174461364746, name="Tobacco", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=3000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.4072818756103516, name="Explicit Nudity", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=3000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.879955291748047, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=3000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.4072818756103516, name="Sexual Activity", parent_name="Explicit Nudity", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=3000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.1348352432250977, name="Smoking", parent_name="Tobacco", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=3000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.1348352432250977, name="Tobacco", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=3000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.879955291748047, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=3520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=4.893579006195068, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=3520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=4.893579006195068, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=4000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.2859199047088623, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=4000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.2859199047088623, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=4520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.868089437484741, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=4520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=2.868089437484741, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=5000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.615448236465454, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=5000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.615448236465454, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=5520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=3.0176239013671875, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=5520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=3.0176239013671875, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=6000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=3.4282007217407227, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=6000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=3.4282007217407227, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=6520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.4649137258529663, name="Smoking", parent_name="Tobacco", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=6520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.4649137258529663, name="Tobacco", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=10520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.5620267391204834, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=10520, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.5620267391204834, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=11000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.3054022789001465, name="Graphic Violence Or Gore", parent_name="Violence", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>,
   #<struct Aws::Rekognition::Types::ContentModerationDetection timestamp=11000, moderation_label=#<struct Aws::Rekognition::Types::ModerationLabel confidence=1.3054022789001465, name="Violence", parent_name="", taxonomy_level=nil>, start_timestamp_millis=nil, end_timestamp_millis=nil, duration_millis=nil, content_types=nil>],
 next_token=nil,
 moderation_model_version="6.1",
 job_id="6399...e",
 video=#<struct Aws::Rekognition::Types::Video s3_object=#<struct Aws::Rekognition::Types::S3Object bucket="ifellinaholeonce-storage", name="7230860-hd_1080_1920_25fps.mp4", version=nil>>,
 job_tag=nil,
 get_request_metadata=#<struct Aws::Rekognition::Types::GetContentModerationRequestMetadata sort_by="TIMESTAMP", aggregate_by="TIMESTAMPS">>

Reproduction Steps

Upload a video to S3, start content moderation and then get content moderation..

client = Aws::Rekognition::Client.new(
    credentials: Aws::Credentials.new(
    aws_access_key_id,
    aws_access_key
  ),
  region: aws_region,
)

content_key = "your-s3-item-content-key"

resp = client.start_content_moderation(
  video: {
    s3_object: {
      bucket:,
      name: content_key
    }
  },
  min_confidence: 1 # set this really low so that we can easily get some moderation flags returned
)

job_id = resp.job_id

# wait a minute for processing to complete

result = client.get_content_moderation(job_id:)

Possible Solution

No response

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-rekognition

Environment details (Version of Ruby, OS environment)

ruby 3.2.3p157

[mullermp]

Thanks for opening up an issue. What version of aws-sdk-rekognition are you using? The latest version is 1.94.0.

[mullermp]

I don't use rekognition, but it looks like to use the V7 label categories (the Drugs & Tobacco label), you need to opt in to some private version?

https://docs.aws.amazon.com/rekognition/latest/dg/moderation.html#moderation-labels-update-sdk

[ifellinaholeonce]

Sorry I should have specified the gem version. It is indeed the newest

    aws-sdk-rekognition (1.94.0)
      aws-sdk-core (~> 3, >= 3.191.0)
      aws-sigv4 (~> 1.1)

That is interesting @mullermp about the private version. The response does state moderation_model_version="6.1", so maybe this is the missing piece.

[ifellinaholeonce]

@mullermp I think this is the information I was missing (I swear this wasn't in the docs yesterday 😂). Looks like I just happened to show up during a transition time. I don't think there is anything more to do here so I am going to close this. Thanks for the help.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

[mullermp]

No worries. I'm glad that was the answer. @RanVaknin from the support team informed me that on May 24, that new version will become default.

[mullermp]

Let me know about those nil members. I'm fairly confident (99.99% sure) that we are parsing the service responses correctly - we have very extensive tests on this per protocol. It's likely that the version of the model from the service simply just does not return any values for those.