Server

[leodido]

Needs a way to specify configs affecting the service command, but not in the service provider config.

Eg.,

• [x] Whether to enable the self resource or not
• [x] Defaults values (min, max) for count parameter (refs #55)

We also must implement one (or more) way to provision configuration. Better if using a layer capable of abstracting the type of configuration source.

Eg.,

• [x] YAML file

Please list here other things, editing this message.

cc @alelb, @leogr, @samechelon.

[leodido]

This message is about choices.

Choices we have to do.

• [ ] How to handle the authenticated subject alias (eg., /Me endpoint) - Section 3.11 @ RFC7644
  • [ ] Permanent redirect (308) to /User
  • [ ] Process directly with Location header pointing to the aliased resource (ie., User)
  • [ ] Both way but configurable
• [x] How to handle the /ServiceProviderConfigs endpoint: ServiceProviderConfig is not represented as a resource, has not an identifier, a "get all" method does not make much sense
  • [x] Leave as is currently
  • [x] Use the StaticResourceService
  • [x] Create a NotIdentifiableResourceService
• [x] How to behave when storage server is not reachable
  • [x] @alelb can you confirm this is ok now?
• [x] How to enforce meta.resourceType to be present also when attributes (inclusions) is specified (refs #55)
  • [x] Make possible that parent and children have different values for same characteristic
  • [x] Make filtering paths working also on leafs
• [x] How to handle ambiguous attributes
  • [x] Only match against the base schema
• [x] What to do when user specifies an unknown attribute or excluded attribute
  • [x] Panicking (current implementation)
  • [x] Ignoring it

Please list here other choices, editing this message.

cc @alelb, @leogr, @samechelon.

[leodido]

The subattributes meta.resourceType is enforced to be always returned. Thus it is now an element of the minimal set of attributes returned.

This has required also some bug fixing, in particular to the Contexts() function that now correctly executes filtering also on subattributes which parent has already been filtered out.

cc @leogr

[leogr]

@leodido As per RFC, service provider SHALL ignore extraneous attributes for interoperability purpose because they may exists in other schemas implemented by 3rd-party but not by the current service provider, thus I suggest to just filter out those attributes which cannot be contextualised (ie. Projection should not return them).

Also, I think we have to do not check for Undefined() paths here, because contextualised path cannot be undefined (if that was possible, it would mean that schema was invalid and it's not a Projection responsibility ensuring that).

Finally, I think also that this kind of API should never panic.

[leodido]

I finally fix'd (almost) everything we need to fix to make the GET /<resource>/:id to work.

Anyway there are already some gotchas we need to discuss.

A. Inclusions

1. Existing attribute(s)

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891?attributes=displayname,name.givenname

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "resourceType": "User"
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
    "urn:ietf:params:scim:schemas:core:2.0:User": {
        "displayName": "Tiffany Fork",
        "name": {
            "givenName": "Tiffany"
        }
    }
}

Notice the values of meta.location, meta.created and so on (ie., zero values since not grabbed from storage) ...

Imho we should fix this ... cc @leogr, @alelb

2. All unknown attributes

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891?attributes=unk,ciao,pippo

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "location": "https://example.com/v2/Users/2819c223-7f76-453a-919d-ab1234567890",
        "resourceType": "User",
        "created": "2010-01-23T04:56:22Z",
        "lastModified": "2011-05-13T04:42:34Z",
        "version": "W/\"a330bc54f0671c9\""
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
    "urn:ietf:params:scim:schemas:core:2.0:User": {
        "active": true,
        "addresses": [
            {
                "...": "...",
            },
            {
                "...": "...",
            }
        ],
        "displayName": "Tiffany Fork",
        "...": "..."
}

Notice the strange (null) value of lastModified ... cc @alelb

3. No inclusions

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891

JSON as (A.2), (B.2).

4. Existing extension attribute(s)

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891?attributes=urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:costCenter,urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:manager.$ref,displayname

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "resourceType": "User"
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
    "urn:ietf:params:scim:schemas:core:2.0:User": {
        "displayName": "Tiffany Fork"
    },
    "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
        "costCenter": "4130",
        "manager": {
            "$ref": "../Users/26118915-6090-4610-87e4-49d8ca9f808d"
        }
    }
}

5. Existing reference attribute(s)

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891?attributes=displayName,groups.$ref

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "resourceType": "User"
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
    "urn:ietf:params:scim:schemas:core:2.0:User": {
        "displayName": "Tiffany Fork",
        "groups": [
            {
                "$ref": "https://example.com/v2/Groups/fc348aa8-3835-40eb-a20b-c726e15c55b5"
            }
        ]
    }
}

B. Exclusions

1. Existing attribute(s)

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891?excludedAttributes=displayName,locale,nickName,profileurl,title,timezone,username,usertype,preferredlanguage,active,x509certificates.value,photos.value,photos.type,phonenumbers.type,phonenumbers.value,ims.type,ims.value,name.familyname,name.formatted,name.honorificsuffix,name.honorificprefix,emails.type,emails.value,groups.value,groups.display,addresses.type,addresses.streetaddress,addresses.type,addresses.region,addresses.country,addresses.formatted,addresses.postalcode,addresses.locality

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "location": "https://example.com/v2/Users/2819c223-7f76-453a-919d-ab1234567890",
        "resourceType": "User",
        "created": "2010-01-23T04:56:22Z",
        "lastModified": "2011-05-13T04:42:34Z",
        "version": "W/\"a330bc54f0671c9\""
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
    "urn:ietf:params:scim:schemas:core:2.0:User": {
        "addresses": [
            {
                "primary": true
            },
            {}
        ],
        "emails": [
            {
                "primary": true
            },
            {}
        ],
        "groups": [
            {
                "$ref": "https://example.com/v2/Groups/fc348aa8-3835-40eb-a20b-c726e15c55b5"
            }
        ],
        "ims": [
            {}
        ],
        "name": {
            "givenName": "Tiffany",
            "middleName": "Geraldine"
        },
        "phoneNumbers": [
            {},
            {}
        ],
        "photos": [
            {},
            {}
        ],
        "x509Certificates": [
            {}
        ]
    },
    "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
        "costCenter": "4130",
        "department": "Tour Operations",
        "division": "Theme Park",
        "employeeNumber": "701984",
        "manager": {
            "$ref": "../Users/26118915-6090-4610-87e4-49d8ca9f808d",
            "displayName": "John Smith",
            "value": "26118915-6090-4610-87e4-49d8ca9f808d"
        },
        "organization": "Universal Studios"
    }
}

Notice the complex attributes with their zero values (eg. [], {}) ...

Similar to the case A.1.

We could think to a layer responsible to eliminate zero values from responses. cc @leogr, @alelb

And also pay attention to do not eliminate attributes for which the nil value is possible.

2. All unknown exclusions

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891?excludeAttributes=pippo,pluto

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "location": "https://example.com/v2/Users/2819c223-7f76-453a-919d-ab1234567890",
        "resourceType": "User",
        "created": "2010-01-23T04:56:22Z",
        "lastModified": "2011-05-13T04:42:34Z",
        "version": "W/\"a330bc54f0671c9\""
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
    "urn:ietf:params:scim:schemas:core:2.0:User": {
        "active": true,
        "...": "..."
    },
    "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
        "costCenter": "4130",
        "...": "..."
    }
}

3. No exclusions

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891

JSON as (A.2), (B.2).

4. Existing extension attribute(s)

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891?excludedAttributes=urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:costcenter,urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:department

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "location": "https://example.com/v2/Users/2819c223-7f76-453a-919d-ab1234567890",
        "resourceType": "User",
        "created": "2010-01-23T04:56:22Z",
        "lastModified": "2011-05-13T04:42:34Z",
        "version": "W/\"a330bc54f0671c9\""
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
   "urn:ietf:params:scim:schemas:core:2.0:User": {
        "active": true,
        "...": "..."
    },
    "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
        "division": "Theme Park",
        "employeeNumber": "701984",
        "manager": {
            "$ref": "../Users/26118915-6090-4610-87e4-49d8ca9f808d",
            "displayName": "John Smith",
            "value": "26118915-6090-4610-87e4-49d8ca9f808d"
        },
        "organization": "Universal Studios"
    }
}

5. Existing reference attribute(s)

GET localhost:8787/v2/Users/2819c223-7f76-453a-919d-ab1234567891?excludedAttributes=urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:manager.value,urn:ietf:params:scim:schemas:extension:enterprise:2.0:User:manager.$ref,groups.$ref

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "location": "https://example.com/v2/Users/2819c223-7f76-453a-919d-ab1234567890",
        "resourceType": "User",
        "created": "2010-01-23T04:56:22Z",
        "lastModified": "2011-05-13T04:42:34Z",
        "version": "W/\"a330bc54f0671c9\""
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
   "urn:ietf:params:scim:schemas:core:2.0:User": {
        "active": true,
        "...": "...",
        "groups": [
            {
                "display": "Employees",
                "value": "fc348aa8-3835-40eb-a20b-c726e15c55b5"
            }
        ],
        "...": "..."
    },
    "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User": {
        "costCenter": "4130",
        "department": "Tour Operations",
        "division": "Theme Park",
        "employeeNumber": "701984",
        "manager": {
            "displayName": "John Smith"
        },
        "organization": "Universal Studios"
    }
}

C. Both

TBD

[leogr]

For inclusion With the only exception of meta.resourceType that must be always present in our implementation, there're two things you have to check:

• there's a typo (lastMod instead of lastModification) here
• and also note:

  // Meta ...
  type Meta struct {
  Location     string     `json:"location" validate:"uri,required"`
  ResourceType string     `json:"resourceType" validate:"required"`
  Created      *time.Time `json:"created" validate:"required"`
  LastModified *time.Time `json:"lastModified" validate:"required"`
  Version      string     `json:"version,omitempty"`
  }

  almost all are required and haven't omitempty, but should they be so? Please, check the RFC.

That said I didn't get why externalId is returned, even it has omitempty

[leogr]

For exclusion About complex attributes with their zero values (eg. [], {}), I think it's an issue with mongo projection. I'm not sure, but it may happen when you project a sub-attribute that's not present within the document. Anywau, something like that:

        "phoneNumbers": [
            {},
            {}
        ],

it's not nil nor empty, so cannot be a JSON marshal responsibility to clean up them, event they have omitempty.

[leodido]

About your first comment you were right.

I fixed the typo and setup the other fields of meta as not required and omitempty.

About your last comment investigating I found the following.

Assume a query with excludedAttributes=emails.type,emails.value.

The object returned from the mongo adapter will contain:

emails:[map[primary:true] map[]]

Notice the second field (ie., the empty map).

And then I found that resource.Resource MarshalJSON impl. does not implement omitempty currently.

[leogr]

@leodido note that “externalId" is actually missing from toResource here. Also, it should be better if the "omitempty" check was applied to toDoc too (just to avoid saving the empty field, not mandatory).

[leogr]

About

emails:[map[primary:true] map[]]

IMHO, the storage should not return that, however I'm not sure if we can fix that within the storage layer.

[leogr]

@leodido I noticed that response examples above are wrong. For instance the following:

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "resourceType": "User"
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
    "urn:ietf:params:scim:schemas:core:2.0:User": {
        "displayName": "Tiffany Fork",
        "name": {
            "givenName": "Tiffany"
        }
    }
}

should be so:

{
    "id": "2819c223-7f76-453a-919d-ab1234567891",
    "meta": {
        "resourceType": "User"
    },
    "schemas": [
        "urn:ietf:params:scim:schemas:core:2.0:User",
        "urn:ietf:params:scim:schemas:extension:enterprise:2.0:User"
    ],
    "displayName": "Tiffany Fork",
    "name": {
         "givenName": "Tiffany"
     }
}

Because only extensions' attributes must be enclosed within base schema's attributes must not be enclosed per namespace within a complex, base schema attributes must not.

See https://tools.ietf.org/html/rfc7644#section-3.4.1

Marshalling/Unmarshalling should be implemented according with those rules.

What's wrong?

[leodido]

@leogr you're right.

Probably the cause is this line.

[leogr]

Hey you, by https://github.com/fabbricadigitale/scimd/pull/46/commits/166b561e2e564149be87de9467e3acf3217d38f0 I've fixed empty {} issue. Double check please, and let me know if it works properly.

[alelb]

@leodido , @leogr Take a look to my last commit please. I moved ValidateRequired in required package to avoid import cycle with .../api/attr package. We can move all functions to perform validation in a shared package. What do you think about this? Any other proposals? Thanks!

[leodido]

@alelb regarding validations, as discussed yersterday.

We could profit from gin validation capabilities for the common attributes struct.

Rather for the data map (complex) of the resource we could write some line of code that will perform structural validation based on the attribute characteristics of the attribute (ie., keys) contained in the data map.

[leodido]

Hey @marcomenoni welcome!

Thanks a lot for the contribution. :trophy:

Only one main observation.

We should move away the logic (password hashing via bcrypt) implemented by your listeners into a dedicated package. So to split the adapter from its listeners: the idea is that they have to be plug-n-play (imagine another way of hashing password implemented via another listener handler) and also storage agnostic.

So i propose to move this part of the code in another package. I propose to call it listeners, within storage/listeners folder.

This way it could be plugged one day also on other kind of storage without duplicating code for example. And the code is well separated.

[marcomenoni]

ok, I am going to move the listeners in a dedicated package

[leodido]

JSON file needs to be embedded within the binary (as I done last week) but this results in a lot of extra space and computation.

Furthermore loading from paths (substituted with virtual ones pointing to the embedded representation of JSON files, at runtime) over complicates the configuration system.

[leodido]

We are almost there guys ...

cc @fabbricadigitale/openinnovation