search

cfelder / ldap3-orm

ldap3-orm, object-relational mapping for ldap3
GNU Lesser General Public License v3.0
11 stars 0 forks source link

Speeding performance on EntryType instance and Connection add #4

Open TcaManager opened 5 years ago

TcaManager commented 5 years ago

Hi, It's not a real issue, we use your Framework to make validation of entries before exporting an ldif. With a code that looks like this

ldif_connection = Connection(server=None,client_strategy='LDIF')
user_entry_type = EntryType('...') #Put whatever data here, dn is a template using uid attribute
for user_data in users_data: # users_data is a list of dict containing more than 10 attributes
    user = user_entry_type(user_data) # It takes about 0.001 for each user
    ldif_connection.add( # This too takes about 0.001 for each user
        user.entry_dn,
        user.object_classes,
        user.entry_attributes_as_dict
    )

We found that each of the two operations in the loop cost 0.001 s. Are there some parameters we could use to speed up one of those operations? Is it possible to make it much faster with more CPU or RAM? Can multprocessing help here? Thanks.

TcaManager commented 5 years ago

For example, if I profile a script inserting 2 users, I get the following data with the Spyder Code Profiler, image

cfelder commented 5 years ago

Thanks for reaching out @TcaManager. Indeed there is some overhead on instantiation of dynamic classes. I'll have a look at this but it will need some time to profile the code myself to pin point the problem. If you are using SMP multiprocessing can increase performance by chunking your entries to several processes. Nevertheless this just mitigates the real issue.