PatchOperation$AddOperation instance creation fails on path values with selection filters

[KoichiSenada]

Describe the bug PatchOperation.AddOperation instance creation fails on path values with selection filters. https://github.com/pingidentity/scim2/commit/4b3d63c7c22ec044f5da4bb14848ac1366a3fa13#r35642158

com.unboundid.scim2.common.exceptions.BadRequestException: path field for add operations must not include any value selection filters
    at com.unboundid.scim2.common.exceptions.BadRequestException.invalidPath(BadRequestException.java:197)
    at com.unboundid.scim2.common.messages.PatchOperation$AddOperation.<init>(PatchOperation.java:101)

To Reproduce Send a SCIM2 protocol https://tools.ietf.org/html/rfc7644#section-3.5.2 compliant PATCH request.

{
  "schemas" : [ "urn:ietf:params:scim:api:messages:2.0:PatchOp" ],
  "Operations" : [ {
    "op" : "add",
    "path" : "addresses[type eq \"work\"].country",
    "value" : "Australia"
  }, {
    "op" : "add",
    "path" : "addresses[type eq \"work\"].locality",
    "value" : "Australia"
  }, {
    "op" : "add",
    "path" : "addresses[type eq \"work\"].formatted",
    "value" : "Australia"
  } ]
}

Expected behavior It is expected that the PatchOperation$AddOperation instance would be created. https://tools.ietf.org/html/rfc7644#section-3.5.2

   Valid examples of "path" are as follows:

       "path":"members"

       "path":"name.familyName"

       "path":"addresses[type eq \"work\"]"

       "path":"members[value eq
              \"2819c223-7f76-453a-919d-413861904646\"]"

       "path":"members[value eq
              \"2819c223-7f76-453a-919d-413861904646\"].displayName"

[Stummi]

We seem to encounter the same issue when trying to integrate AD.

Ironically, its just the AddOperation Constructor which plainly refuses any path with a value selection filter. Ther underlaying tools can work with the filter without issues:

val body = """
    {
        "addresses": [
        {
            "type": "work",
            "locality": "old"
        }
        ]
    }
""".trimIndent()

val om = ObjectMapper()
val node: ObjectNode = om.readValue(body, ObjectNode::class.java)
val newAddr: JsonNode = om.readValue("\"new\"", JsonNode::class.java)
JsonUtils.addValue(Path.fromString("addresses[type eq \"work\"].locality"), node, newAddr)

println(node)

so the solution would just be removing the check in the AddOperation constructor

[davidmarne-wf]

@braveulysses any chance at getting fix in here, or at least an answer as to why the json creator for AddOperation has this clause?

[braveulysses]

@lance-purple-unboundid, can you please look into this issue?

[dlevesque-primalogik]

Any update on this? We also have this issue with AzureAD. Has anyone found a workaround?

[dahuber-github]

I resolved this by editing the PatchOperation class directly. I made the following method changes:

@JsonCreator
private AddOperation(
    @JsonProperty(value = "path") final Path path,
    @JsonProperty(value = "value", required = true) final JsonNode value)
    throws ScimException
{
  super(path);
  if(value == null || value.isNull() ||
       ((value.isArray() || value.isObject()) && value.size() == 0))
   {
     throw BadRequestException.invalidSyntax(
         "value field must not be null or an empty container");
   }
  if(path == null && !value.isObject())
  {
    throw BadRequestException.invalidSyntax(
        "value field must be a JSON object containing the attributes to " +
            "add");
  }
  this.value = value;
}

@Override
public void apply(final ObjectNode node) throws ScimException
{      
    if(getPath() != null)
    {
      for (Path.Element element : getPath())
      {
        if(element.getValueFilter() != null)
        {
            JsonNode fieldNode = node.path(element.getAttribute());

            ArrayNode arrayNode = null;
            if (fieldNode.isMissingNode())
                arrayNode = node.withArray(element.getAttribute());
            else
                arrayNode = (ArrayNode) fieldNode;

            String strType = element.getValueFilter().getAttributePath().toString();
            String strValue = element.getValueFilter().getComparisonValue().asText();

            ObjectNode newNode = null;
            for (Iterator<JsonNode> iterator = arrayNode.elements(); iterator.hasNext();)
            {
                JsonNode childNode = iterator.next();
                if ((childNode.get(strType) != null) && (childNode.get(strType).asText().equals(strValue)))
                {
                    newNode = (ObjectNode) childNode;
                    break;
                }
            }

            if (newNode == null)
                newNode = arrayNode.addObject();

            newNode.put(strType, strValue);             
        }
      }
    }

  JsonUtils.addValue(getPath() == null ? Path.root() : getPath(), node, value);
  addMissingSchemaUrns(node);
}

I've tested these changes against Microsoft Azure and it seems to work. I'm sure there is a better way to do this. I provide this in hopes it helps the development team.

[davidmarne-wf]

@braveulysses @lance-purple-unboundid any updates?

[MartinHaeusler]

I'm running into this issue as well. My identity provider (Azure Active Directory) sends me update operations like this in PATCH requests:

{
    "op": "Add",
    "path": "phoneNumbers[type eq \"work\"].value",
    "value": "+123"
}

This is valid according to the SCIM2 specification. Trouble is: as soon as the patch request contains one of these, the entire request is rejected because of the exception thrown in the @JsonCreator constructor of AddOperation. So I can't even ignore these updates.

Is there any fix in sight? The ticket is pretty old...

[kqarryzada]

A fix for this issue has not been targeted, but I can try to take a look soon.

[MartinHaeusler]

@kqarryzada I am currently investigating the issue; if it works out I may have a Pull Request for you in the next couple of days. My main problem with it is that the SCIM2 standard permits a lot of requests which are complete nonsense.

For example:

{
    "op": "Add",
    "path": "phoneNumbers[type ne \"work\"].value",
    "value": "+123"
}

This is totally valid according to the RFC, as it permits "any" path filter. But semantically, this request makes no sense, because we should add a phoneNumber with a type not equal to "work". So which value should we use for the type field? All we know it should be not equal to "work". "foo" would be valid in that regard. So basically, the only filters that make sense are eq and and. Furthermore, filters may not occur in the last step of the path, because it specifies the field which should receive the value. No mention of that in the RFC either (unless I overlooked it?). I'll make sure that the "happy paths" described in the RFC are working, and meaningful exceptions are thrown for everything else.

[MartinHaeusler]

@kqarryzada I've created a Pull Request with my fix. Unfortunately I had to replicate some of the code from JsonUtil, because the Add operation really is... a bit special, and I didn't want to pollute the logic of the general JsonUtil with the specifics of Add. All existing tests pass, and I've added new ones based on the requests we're receiving from Azure Active Directory.

The Pull Request notice says that you're currently not accepting third-party code contributions. Maybe you could take a look regardless in this case? The issue is open for a long time, and this would be a working fix.

[kqarryzada]

@MartinHaeusler, I've taken an initial look at this. I agree that the only filter types that would make sense are eq and and, since other filter types are ambiguous. However, as you've noted, and filters make the logic quite messy because we must traverse the filter "subtree" and validate that all non-component subfilters are of the equality type. Then, we must evaluate the desired updates.

I think that as an initial step, it makes sense to evaluate this issue from the perspective of the desired use case, which is a single eq filter. You are also correct that we do not accept code contributions, but the PR you've listed is very helpful for understanding the use cases you're trying to support.

[MartinHaeusler]

@kqarryzada thanks for looking into the topic; I'm always happy to help. For now, my team uses a fork of this repository which has my PR applied, but we would welcome an opportunity to switch back to the official releases once this issue has been resolved.

[lenyt]

@kqarryzada any update on this issue? Note that this software happily takes patch operation with a "replace" that has a similar path specification so there is inconsistency there.

[lenyt]

Although I realize that some filters that apply for "replace" don't make sense for "add" and possibly vice versa. For example, "ne" filter does make sense for replace while it doesn't really make sense for add.

[kqarryzada]

@lenyt, I am still actively working on this issue.

As you point out, we support value selection filters that are used in remove and replace operations, because that behavior is well-defined by the specification in RFC 7644. When a client specifies a filter for these operation types, it performs the modification, but only to values in the array that match the filter. For example, for a remove operation with a path of emails[type eq "work"], the user resource will be updated to remove some emails, but only those with a "type": "work" field. Thus, the type eq "work" filter is actually used to perform filtering on the emails attribute of the user resource. This was the intended use of value selection filters.

Add operations are different because the filter provided in the path is not really used to perform filtering. When an add operation is created with a path of emails[type eq "work"].value, it is actually requesting that the "type": "work" field is also added to the user resource in addition to the value field that is specified. This behavior is not actually defined by RFC 7644, but it is not forbidden, either. This form is popular enough to warrant updating the SCIM SDK to account for this.

Hopefully this background helps explain why we have existing filter support for other operation types, but the behavior for add was not permitted before.

[kqarryzada]

This behavior should now be supported in the 3.0.0 release, which is available now. If anyone runs into issues with this, please open a new issue.