search

simpleidserver / SimpleIdServer

OpenID, OAuth 2.0, SCIM2.0, UMA2.0, FAPI, CIBA & OPENBANKING Framework for ASP.NET Core
https://simpleidserver.com/
Apache License 2.0
693 stars 91 forks source link

[SCIM] FlatAttributes in MongoDB #571

Closed gabrielemilan closed 10 months ago

gabrielemilan commented 11 months ago

Hello,

I notice that in SCIM using mongoDB, FlatAttributes array grows a lot every time I add a user to one group. At least 3 kb per user.

Since the limitation is 16mb, I think can be an issue if a group has thousands of users, isn't it?

Regards. Gabriele Milan

simpleidserver commented 11 months ago

Hello,

Indeed it can be an issue. This ticket is closed related to this one #570. I'm going to work on it ASAP.

KR,

SID

simpleidserver commented 10 months ago

With the latest changes, the FlatAttributes have been removed from the SCIMRepresentation collection. They have been replaced by a list of references to the SCIMRepresentationAttribute collection

I have written a small documentation explaining how to migrate from version 4.0.3 to 4.0.4. You can find the documentation at the following link: https://github.com/simpleidserver/SimpleIdServer/blob/release/4.0.4/website/docs/migrations/403to404.md

gabrielemilan commented 10 months ago

I tried adding 3k users inside a single group and the PATCH takes a lot of time:

image

the query impacted seems this one:

db.scimRepresentationAttributes.find({ "_id": { "$in": [ "0017c4ba-e62c-4e11-b043-a4ab9693b3f5", "0018c359-b172-48d2-8aa5-15543281ae2c", "002c36ce-11bb-40a9-b641-a9d3c0374874", "00332286-9787-426f-a3af-e9394a92cde7", "00332f91-691e-47c0-8c43-8352da7befa2", "0035843a-f4c5-43e2-b514-f5a992a03977", "005736b5-0d49-489a-be34-ab3da9ef980b", "006c6ec4-97bb-4145-a3fe-a9a544f5dc89", "007657d7-9474-44c4-ac59-ae7791f14290", "007ba1f8-ed64-4a61-afee-b7c9bc3042ef", "007cd5f7-33a0-44ca-be5c-4bfd454adaf6", "00807ee8-12a4-48e8-98c9-a129fda1bca0", "00812cd4-233f-4bc7-8bde-547582c095aa", "00842ef3-d50e-4ba5-bbe3-cc82e8ce87db", "00894d41-f23f-4daa-97e1-0510233f0e9d", "008d2a23-cc5d-4ad2-8951-2b9973dd811c", "0097d69c-41a1-4f3e-a046-86a1ed67fa24", "0099aef0-5ccd-4cc3-a162-7e60a5563935", "00a14adb-1a41-4d6c-b07d-67781bac93d1", "00a2da5b-2637-4a7f-8ef1-94a20fb96d90", "00af775f-fb47-407e-958f-8f17e755cd03", ......

gabrielemilan commented 10 months ago

I set this options and now the response is 3 sec.

IsNoContentReturned = true

Meanwhile I checked the code and the most expensive part is when the system gets all references from the flat attributes collection. Is it necessary even if the operation is ADD, is it possible maybe to add a condition to bypass the gets of all data when the operation is ADD, I think is not needed, isn't it?

Thanks a lot.

simpleidserver commented 10 months ago

I have made some modifications in the release/4.0.4 branch. Instead of retrieving the latest version of the representation in the database, the patch operations are now applied to the existing SCIMRepresentation, and the resulting data is returned to the client.

With this change, there is no longer a need to set the IsNoContentResult property, as it has been removed from the Options :).

Kind regards,

SID"

gabrielemilan commented 10 months ago

Ok seems fine now, can I have the ETA of version 4.0.4?

Thanks :-)

simpleidserver commented 10 months ago

If everything goes well, we will publish the release 4.0.4-rc1 this evening :)

gabrielemilan commented 10 months ago

Hello, I did other tests with 10k users and the performances are now acceptable. To me you can deliver the new version :-)