[Feature] add "memory" option to ldap:import command

[MostafaNobaghi]

Environment:

• LDAP Server Type: ActiveDirectory
• LdapRecord-Laravel Major Version: v3
• PHP Version: 8.1

I tested the connection with php artisan ldap:test and get successful message and data table.

When i run php artisan ldap:import get no output. and log file only contains log about binding:

[2023-07-04 00:56:47] local.INFO: LDAP (ldap://192.168.0.5:389) - Operation: Binding - Username: aparat
[2023-07-04 00:56:47] local.INFO: LDAP (ldap://192.168.0.5:389) - Operation: Bound - Username: aparat

it look like search event doesn't trigger.

and when i test fetching data by $users = \\App\\Ldap\\User::query()->where('sn', 'smith')->get(); it return users

this is my .env

LDAP_LOGGING=true
LDAP_HOST=192.168.0.5
LDAP_PORT=389
LDAP_BASE_DN="dc=APARATINSURANCE,dc=com"

LDAP_SSL=false
LDAP_TLS=false
LDAP_USERNAME=aparat
LDAP_PASSWORD="password"
LDAP_MAIL_ATTRIBUTE=userprincipalname
LDAP_ATTRIBUTE=mail
LDAP_USERNAME_ATTRIBUTE=samaccountname
LDAP_SYNC_FILTER=(&(samaccountname=*))
LDAP_EMPLOYEE_NUMBER=samaccountname

and this is inside my config/auth.php:

'providers' => [

        ...,

        'ldap' => [
            'driver' => 'ldap',
            'model' => LdapRecord\\Models\\ActiveDirectory\\User::class,
            'database' => [
                'model' => App\\Models\\User::class,
                'sync_passwords' => false,
                'sync_existing' => [
                    'mobile' => config('services.ldap.mobile_attribute', 'mobile'),
                ],
                'sync_attributes' => [
                    'email' => config('services.ldap.mail_attribute', 'mail'),
                    'last_name' => 'sn',
                ],
            ],
        ],

    ],

     !------- UPDATE --------!

When i remove LDAP_BASE_DN or put an invalid value in it. the import commant return "there were no users found to import" and the log is:

[2023-07-04 21:20:28] production.INFO: LDAP (ldap://172.16.0.5:389) - Operation: Binding - Username: aparat
[2023-07-04 21:20:28] production.INFO: LDAP (ldap://172.16.0.5:389) - Operation: Bound - Username: aparat
[2023-07-04 21:20:28] production.INFO: LDAP (ldap://172.16.0.5:389) - Operation: Paginate - Base DN: DC=APARATINSURANCE,DC=com - Filter: (&(objectclass=\74\6f\70)(objectclass=\70\65\72\73\6f\6e)
(objectclass=\6f\72\67\61\6e\69\7a\61\74\69\6f\6e\61\6c\70\65\72\73\6f\6e)(objectclass=\75\73\65\72)(!(objectclass=\63\6f\6d\70\75\74\65\72))) - Selected: (objectguid,*) - Time Elapsed: 175.47

[stevebauman]

Hi @MostafaNobaghi,

Are you sure your configured BASE_DN is correct? Can you perform a query on your configured model to ensure that you're able to retrieve users from your domain?

https://ldaprecord.com/docs/core/v3/models/#retrieving-models

[MostafaNobaghi]

Hi @MostafaNobaghi,

Are you sure your configured BASE_DN is correct? Can you perform a query on your configured model to ensure that you're able to retrieve users from your domain?

https://ldaprecord.com/docs/core/v3/models/#retrieving-models

I queried with $users = \App\Ldap\User::query()->where('sn', 'askari')->first(); and get the user data When i remove LDAP_BASE_DN or put an invalid value in it. the import commant return "there were no users found to import" and the log is:

[2023-07-04 21:20:28] production.INFO: LDAP (ldap://172.16.0.5:389) - Operation: Binding - Username: aparat
[2023-07-04 21:20:28] production.INFO: LDAP (ldap://172.16.0.5:389) - Operation: Bound - Username: aparat
[2023-07-04 21:20:28] production.INFO: LDAP (ldap://172.16.0.5:389) - Operation: Paginate - Base DN: DC=APARATINSURANCE,DC=com - Filter: (&(objectclass=\74\6f\70)(objectclass=\70\65\72\73\6f\6e)
(objectclass=\6f\72\67\61\6e\69\7a\61\74\69\6f\6e\61\6c\70\65\72\73\6f\6e)(objectclass=\75\73\65\72)(!(objectclass=\63\6f\6d\70\75\74\65\72))) - Selected: (objectguid,*) - Time Elapsed: 175.47

[stevebauman]

The LDAP model you're querying with and your configured model in your auth.php is different. Use the one you're querying with and see if that works 👍

[MostafaNobaghi]

Thanks for your help @stevebauman

I used LdapRecord\Models\ActiveDirectory\User model which is used in auth.php for query a user:

$user = \LdapRecord\Models\ActiveDirectory\User::query()->where('sn', 'askari')->first(); 

and works fine. it return

 LdapRecord\Models\ActiveDirectory\User {#7863
    +exists: true,
    +wasRecentlyCreated: false,
    +wasRecentlyRenamed: false,
  }

geting guid:

  $abbasi->getFirstAttribute('objectguid') //= b"\x05p·Å /ŒA§Y퉿Ü\x1F+" 

I also used the \LdapRecord\Models\ActiveDirectory\Entry and \App\Ldap\User (it extended from LdapRecord\Models\Model) in auth.php and result was the same, no output and no search operation in log file on import command.

i installed this project on 2 other servers and it works fine there. (its deployed with docker and enviroments are the same). and all three LDAP Servers are Active Directory

[stevebauman]

@MostafaNobaghi

I used LdapRecord\Models\ActiveDirectory\User model which is used in auth.php for query a user:

Oh, you said you used App\Ldap\User in the prior comment:

I queried with $users = \App\Ldap\User::query()->where('sn', 'askari')->first(); and get the user data

---

i installed this project on 2 other servers and it works fine there. (its deployed with docker and enviroments are the same). and all three LDAP Servers are Active Directory

That's really strange that you're able to query for results but none are being returned during the import. This seems like a configuration issue, which is going to be difficult for me to provide help with. You will have to debug this further with your environment. I would try by going into the vendor/directorytree/ldaprecord-laravel/src directory and dumping the $loaded variable here:

https://github.com/DirectoryTree/LdapRecord-Laravel/blob/98e8c4e5f38a006f09f7f86ce290228881e043b1/src/Commands/ImportLdapUsers.php#L115

Try changing your configuration and re-running the import to see if any results are returned there in the dump 👍

[MostafaNobaghi]

Apologize for confusion.

Oh, you said you used App\Ldap\User in the prior comment:

yes, I used App\Ldap\User at first try, and after your response: The LDAP model you're querying with and your configured model in your auth.php is different. Use the one you're querying with and see if that works 👍

I tried querying data using LdapRecord\Models\ActiveDirectory\User (which is in auth.php) again and it worked too.

thanks again for your time and kind help and responses. i will try more to find out the problem. however it would be hard because the ldap server is not accessible directly and i must debug it in production server.

I really appreciate it if You remember something can help me as a clue, share it with me. I will inform You when I make it work.

[MostafaNobaghi]

@stevebauman Hello I did a little digging. First i Upgrade the pakage from version 2 to 3 (3.0.5) and it didn't solve the issue then i continued debugging. I put a logger before and after $loaded : https://github.com/DirectoryTree/LdapRecord-Laravel/blob/98e8c4e5f38a006f09f7f86ce290228881e043b1/src/Commands/ImportLdapUsers.php#L115

Like this:

logger('before loading')
$loaded = $this->importer->loadObjectsFromRepository($this->argument('user'));
logger('after loading')

and run the import command and check the log file and the second log didn't logged

then i tracked this and to here:

https://github.com/DirectoryTree/LdapRecord-Laravel/blob/98e8c4e5f38a006f09f7f86ce290228881e043b1/src/Import/LdapUserImporter.php#L99-L101

and again tried logging like:

      logger('before PAGINATION');
      $objects = $this->objects = $query->paginate();
      logger('after PAGINATION');
      return $objects;

and again it didn't log second logger.

then i replaced the $query->paginate(); with $query->limit(10)->get(); and then run the command again and it load 10 users and imported it as normal and as i expected. and output the import result correctly.

I did this further and it lead me to this line, the codes after this line doesn't execute (it is inLdapRecord repo):

https://github.com/DirectoryTree/LdapRecord/blob/20b7fa492240650e138b6336caf592815ca1a8bc/src/Query/Builder.php#L376

$pages = $this->getCachedResponse($query, $callback);

i couldn't figure it out what is it in that cause the problem. there is no exception and error Do You have any idea?

UPDATE

When i change https://github.com/DirectoryTree/LdapRecord-Laravel/blob/98e8c4e5f38a006f09f7f86ce290228881e043b1/src/Import/LdapUserImporter.php#L100 to this:

   return $this->objects = $query->where('sn', 'Askari')->paginate(); 

it works and load users with name "Askari" (15 users) and import them.

[MostafaNobaghi]

After finding out issue is related to pagination and big number of objects, I inspected command options and tried "chunk" and users imported with chunk option. I'll use chunk as a work around for now. but still curious what is the problem with pagination.

[stevebauman]

@MostafaNobaghi Chunking uses the same pagination mechanism under the hood. Are you sure you're only retrieving 15 users? It sounds like you may be hitting an out-of-memory issue, which would explain chunk working, and pagination not.

[MostafaNobaghi]

@stevebauman it is 15 users with sn filter for test. with 15 users it works. but without filter, number is more than 2000. how does it work when not using chunk? does it get all users at once and paginate them? or does it get all pages at once?

[stevebauman]

Yea you're likely running out of memory. With the paginate method it collects all users into memory. With chunk, it iterates through 100 records (iirc) each, but doesn't keep all of them in memory. If it works with chunk, then keep using chunk, as that's why the feature exists.

[MostafaNobaghi]

thank you. chunk doesn't handle delete-missing do you think it is Ok I add a memory option to the command options?

[stevebauman]

Yeah thats intentional. See this comment for resoning:

https://github.com/DirectoryTree/LdapRecord-Laravel/issues/411#issuecomment-1575004174

Can you expand on what you mean by memory command options?

[MostafaNobaghi]

Yes. It is logical chunk and delete-missing can't work together. that's why i suggested memmory option in command. for example we ran this command: php artisan ldap:import --memory="2048" then in ImportLdapUsers something like:

public function handle(LdapUserImporter $importer, Repository $config): int
    {
        if ($this->option('memory')) {
            ini_set('memory_limit', "{$this->option('memory')}M");
        }
        .
        .
        .
    }

[stevebauman]

@MostafaNobaghi Yeah I think that's a great idea 👍

I'm away from my computer until tomorrow, but I can have that implemented before Monday morning (unless you'd like to submit a pull request of course).

[MostafaNobaghi]

Thank you. yes i'd like to do that and it is not urgent for me.