search

chilts / mongodb-queue

Message queues which uses MongoDB.
209 stars 91 forks source link

fix #11 #12

Closed longlostnick closed 8 years ago

longlostnick commented 8 years ago

Think I have the indexes figured out now.

Indexes:

{ deleted: 1, visible: 1 } { ack: 1 }

Queries:

{ deleted: ..., visible: ... } { ack: ..., visible: ..., deleted: ... }

I'm going to test this out on my end some more and report back in a few days or so to confirm this all works fine, but wanted to get your eyes on it.

longlostnick commented 8 years ago

Hmm, i'll take a look at that failing test too.

longlostnick commented 8 years ago

Not sure what's going on here. They are intermittently passing locally. Any ideas?

chilts commented 8 years ago

Wow, awesome, thanks. :)

Not sure about that failed test. I wonder if it has to do with MongoDB v3.

chilts commented 8 years ago

Hmm, I have v3.0.6 installed locally and I don't get the same problem during the tests.

longlostnick commented 8 years ago

@chilts FYI we're still running into some scaling issues. When there gets to be a large amount of unprocessed items (in our case 50k+) it seems like the findOneAndUpdate still doesn't perform very well. The only think I can think of is coming up with is actually improving how it's querying, but not sure of a better way off the top of my head. Honestly, at the end of the day this is probably just pushing the limits of using mongodb-queue for my use case. I will still use it for a bunch of other small queues, but I'm gonna switch this one over to SQS. Just thought I'd followup with an update. Thanks!

chilts commented 8 years ago

Hi @ifightcrime, that's still awesome to know. Yeah, I think it is pushing the limits. The project is intended to allow for getting an easy-to-use queue up to speed quickly, especially if MongoDB is already being used.

I suspect no mater what we do, queries all looking to query and update the first unprocessed item may well keep clashing with each other hence it's a fundamental limitation being all on the same machine.

Anyway, thanks for looking into this for me.

If you don't mind answering one more question: if you're still hitting issues (whatever they are) do you think I should keep your new indexes in the code, or revert back to the older ones? ie. which one did you think performed best (even if both hit scaling issues)?

Thanks again.

longlostnick commented 8 years ago

@chilts indexes now are definitely better.

chilts commented 8 years ago

Lovely, thanks. ;) You're a star and your help is very much appreciated!