Ingest at state level

[that1guy]

Ingesting at this metro-level: USA-SFO on my local machine has been slow. Thus I can rarely paginate past page two of search results and now that I'm beginning to integration socket.io feed updates that will be slow too... you get the idea.

This reminded me of this issue that I think we only discussed over email? Can find GH issue so I'm creating this.

posting-sync-agent config file (US-CA)

{
  "logging": {
    "level": "trace"
  },
  "models": {
    "postings": {
      "schedule": {
        "interval": "every 1 mins"
      },
      "polling" : {
        "location": {
          "state": "USA-CA"
        }
      }
    }
  },
  "services": {
    "3taps": {
      "apikey": "f2862071ede0bbd93bee4f091c522a9e"
    },
    "postings": {
      "url": "http://localhost:4043/v1/postings"
    }
  }
}

Node ingest agent logs

Starting child process with 'node server'
Watching directory '/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent' for changes.
22:24:17.517Z TRACE posting-sync-agent: logger created
22:24:17.523Z TRACE posting-sync-agent: initializing service proxies
22:24:17.524Z TRACE posting-sync-agent: initializing business layer modules
22:24:17.527Z TRACE posting-sync-agent: registering routes for /v1/status
22:24:17.529Z TRACE posting-sync-agent: registering routes for /v1/version
22:24:17.529Z TRACE posting-sync-agent: beginning polling on interval: every 1 mins
22:24:17.538Z  INFO posting-sync-agent: starting standard server on 0.0.0.0:8881
22:25:00.728Z TRACE posting-sync-agent: anchor for Fri Apr 10 2015 15:25:00 GMT-0700 (PDT) retrieved in 720 milliseconds
22:25:01.585Z TRACE posting-sync-agent: poll with anchor 2018960712 completed in 855 milliseconds
22:25:01.586Z TRACE posting-sync-agent: no new postings retrieved
22:26:03.622Z TRACE posting-sync-agent: anchor for Fri Apr 10 2015 15:26:03 GMT-0700 (PDT) retrieved in 3621 milliseconds
22:26:03.625Z ERROR posting-sync-agent: No anchor found
    Error: No anchor found
        at checkResponseForError (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/3taps/lib/index.js:28:10)
        at Request._callback (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/3taps/lib/index.js:74:17)
        at Request.self.callback (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/request/request.js:344:22)
        at Request.emit (events.js:98:17)
        at Request.<anonymous> (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/request/request.js:1239:14)
        at Request.emit (events.js:117:20)
        at IncomingMessage.<anonymous> (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/request/request.js:1187:12)
        at IncomingMessage.emit (events.js:117:20)
        at _stream_readable.js:943:16
        at process._tickCallback (node.js:419:13)
22:27:02.162Z TRACE posting-sync-agent: anchor for Fri Apr 10 2015 15:27:02 GMT-0700 (PDT) retrieved in 2158 milliseconds
22:27:02.668Z TRACE posting-sync-agent: poll with anchor 2018964227 completed in 506 milliseconds
22:27:02.669Z TRACE posting-sync-agent: no new postings retrieved
c22:28:05.254Z TRACE posting-sync-agent: anchor for Fri Apr 10 2015 15:28:05 GMT-0700 (PDT) retrieved in 5251 milliseconds
22:28:05.256Z ERROR posting-sync-agent: No anchor found
    Error: No anchor found
        at checkResponseForError (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/3taps/lib/index.js:28:10)
        at Request._callback (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/3taps/lib/index.js:74:17)
        at Request.self.callback (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/request/request.js:344:22)
        at Request.emit (events.js:98:17)
        at Request.<anonymous> (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/request/request.js:1239:14)
        at Request.emit (events.js:117:20)
        at IncomingMessage.<anonymous> (/Users/braddavis/WebstormProjects/htsApp/api/posting-sync-agent/node_modules/request/request.js:1187:12)
        at IncomingMessage.emit (events.js:117:20)
        at _stream_readable.js:943:16
        at process._tickCallback (node.js:419:13)

All a similar but different note, the urge to deploy to cloud, run multiple ingest agents, populate mongo with millions of items is growing... Late next week or early week after I look forward to working together to deploy to cloud in a fashion that can be updated quickly an redeployed.

[brozeph]

I think we have this one answered now... yes?

[that1guy]

Think so! Didn't test on my end yet, but will..

On Tuesday, April 21, 2015, Joshua Thomas notifications@github.com wrote:

I think we have this one answered now... yes?

— Reply to this email directly or view it on GitHub https://github.com/HashtagSell/posting-sync-agent/issues/4#issuecomment-94870334 .

HashtagSell.com p: 415.294.1981 e: brad.davis@hashtagsell.com

[that1guy]

Closing.. fixed. thank you sir!!