User import and purge is very slow

[bdharrington7]

I'm wondering if I'm the only one seeing this, as user import and purging is incredibly slow (about 1 user / 4-5 seconds). I only have 12 users in my test db, but I can't imagine this being bearable for a large forum.

• nodebb-plugin-import latest (at 9e9d56251f0f2035037a9293862e97b663e03a10)
• importer is nodebb-plugin-import-vanilla @ master (but these queries haven't been changed in a while)
• MongoDB v3.0.7

[akhoury]

hmm .. really? that's new - i will take a look. Which NodeBB version you're importing against ?

[bdharrington7]

Running on the v1.0.0 tag for NodeBB

On Mon, Apr 18, 2016 at 6:49 AM Aziz Khoury notifications@github.com wrote:

hmm .. really? that's new - i will take a look. Which NodeBB version you're importing against ?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/akhoury/nodebb-plugin-import/issues/186#issuecomment-211388263

[breakingrobot]

ping @akhoury 1 users per 2/4 seconds while topics are really fast on IP.Board too

[trjohnst]

Seeing similar results with nodeBB version 1.7.5, and nodebb-plugin-import version 0.4.7 (installed by nodebb admin panel). Purge seems to completely freeze and return to idle on larger sets. After I manually restricted the start/limit sizes down to 100 users with my plugin, nodebb-plugin-importer-phpnuke-phpbb, purge started working again but user import/purge is very slow compared to other types.

[akhoury]

Yea, i'm sorry, purging data didn't get much love, I usually just backup, drop the whole db and restore.

# node setup, install import-plugin, set the config, save them
# create a mongo dump folder 
mongodump 

#  run the import

# if for some reason, I want to do it all over again, I run mongo restore with drop flag, which replaces the db with the one I dumped, usually in a folder called "dump"
mongorestore --drop 

# restart nodebb, restart the import process

As for users import, yes, it is slow, mostly because it's the most expensive to create in the database, nodebb makes a lot of calls to create a single user, and then I have to make some more calls to adjust some of the data, unfortunately, until we can integrate the import process in core, I can't make it much faster, unless someone wants to take a stab at it, be my guest.

[trjohnst]

Thanks for the response @akhoury. I will definitely use this trick to expedite my development.

Is there anyone putting time into integrating? It was my understanding that this plugin was also a bit out of date (the readme indicates needing to use nodebb at v1.7.5 to run correctly)

[akhoury]

Well, from time to time, i bring it up to date, but this has nothing to do with the integration, the integration would be a decision to the nodebb team.

As for the nodebb 1.7.5 version, it really doesn't matter, you can easily upgrade after import, i don't keep it up to date all the time because it's hard to keep up, I wait for major releases or important changes first.

[akhoury]

How did however, started an "augmentation" work, where i augment all the core modules to do what I need them to do. https://github.com/akhoury/nodebb-plugin-import/tree/master/server/augmented