Slow ingestion on cloud

[that1guy]

Last we discussed, we didn't have know what might be causing this. Things to try:

• Get logs that expose raw URLs the sync agent it fetching.
• Use anchor given in previous response for upcoming polling job.

[brozeph]

Marking as resolved for now - several modifications were made to increase performance:

• changed an async.map call to async.mapLimit to limit concurrent writes to MongoDB and ElasticSearch in a single update
• modified posting-sync-agent configuration to limit to 5 postings per bulk round
• re-indexed MongoDB to remove extraneous indexes that would slow inserts, updates and removals
• modified posting-sync-agent so that removals occur in series prior to updates and inserts so that there is reduced potential for concurrent requests creating a lock

[that1guy]

Thanks Josh! Killer job man! I'm juggling applications to angel investors, pitches, and drumming up beta users, so development may be a little slower on my end but I'll try to work off hours to keep things moving forward with UI.