No new postings retreived

[that1guy]

On my local machine sync agent never seems to bring any postings down.

config file

{
  "logging": {
    "level": "trace"
  },
  "models": {
    "postings": {
      "schedule": {
        "interval": "every 1 mins"
      },
      "polling" : {
        "location": {
          "metro": "USA-SFO"
        }
      }
    }
  },
  "services": {
    "3taps": {
      "apikey": "f2862071ede0bbd93bee4f091c522a9e"
    },
    "postings": {
      "url": "http://localhost:4043/v1/postings"
    }
  }
}

Node logs showing no postings found

19:49:00.573Z TRACE posting-sync-agent: logger created
19:49:00.576Z TRACE posting-sync-agent: initializing service proxies
19:49:00.577Z TRACE posting-sync-agent: initializing business layer modules
19:49:00.583Z TRACE posting-sync-agent: registering routes for /v1/status
19:49:00.588Z TRACE posting-sync-agent: registering routes for /v1/version
19:49:00.592Z TRACE posting-sync-agent: beginning polling in USA-SFO on an interval of every 1 mins
19:49:00.622Z  INFO posting-sync-agent: starting standard server on 0.0.0.0:8880
19:50:00.677Z TRACE posting-sync-agent: anchor for Fri May 08 2015 12:50:00 GMT-0700 (PDT) retrieved in 654 milliseconds
19:50:02.011Z TRACE posting-sync-agent: poll with anchor 2122663490 completed in 1331 milliseconds
19:50:02.012Z TRACE posting-sync-agent: no new postings retrieved
c19:51:00.475Z TRACE posting-sync-agent: anchor for Fri May 08 2015 12:51:00 GMT-0700 (PDT) retrieved in 472 milliseconds
19:51:01.537Z TRACE posting-sync-agent: poll with anchor 2122667573 completed in 1060 milliseconds
19:51:01.538Z TRACE posting-sync-agent: no new postings retrieved
19:52:00.478Z TRACE posting-sync-agent: anchor for Fri May 08 2015 12:52:00 GMT-0700 (PDT) retrieved in 478 milliseconds
19:52:01.822Z TRACE posting-sync-agent: poll with anchor 2122673062 completed in 1344 milliseconds
19:52:01.823Z TRACE posting-sync-agent: no new postings retrieved

However if I manually look up the anchors content via URL I see 3Taps is giving results.

http://polling.3taps.com/poll?auth_token=f2862071ede0bbd93bee4f091c522a9e&anchor=2122663490&location.metro=USA-SFO

Excerpt of 3Taps response from manual URL

{"success":true,"anchor":2122670333,"postings":[{"id":2122663499,"source":"E_BAY","category":"SELE","external_id":"281686599181","external_url":"http://www.ebay.com/itm/SNES-Official-Super-Nintendo-AC-Power-Supply-Adapter-SNS-002-NEW-RARE-SEALED-/281686599181","heading":"SNES Official Super Nintendo AC Power Supply Adapter SNS-002 NEW RARE *SEALED***","timestamp":1431114540,"annotations":{"price":"25.0","BIN price":"35.5","address":"Santa Cruz,CA,USA","listingtype":"AuctionWithBIN","zipcode":"95065"},"deleted":false,"location":{"country":"USA","state":"USA-CA","metro":"USA-SFO","region":"USA-SFO-SAT","county":"USA-CA-SNR","city":"USA-SFO-SCR","zipcode":"USA-95065","formatted_address":"95065,Santa Cruz,CA,USA","lat":"37.034585","long":"-121.981339","accuracy":8,"geolocation_status":3}},{"id":2122663503,"source":"E_BAY","category":"SELE","external_id":"261881073030","external_url":"http://www.ebay.com/itm/Nintendo-3DS-Xl-Latest-Model-2015-First-Bidder-Gets-Free-Shipping-/261881073030?pt=LH_DefaultDomain_0","heading":"Nintendo 3DS Xl!!! Latest Model!!! (2015) First Bidder Gets Free Shipping!!!!!!!","timestamp":1431114540,"annotations":{"price":"179.0","BIN price":"250.0","address":"Santa Rosa,CA,USA","listingtype":"AuctionWithBIN","zipcode":"95403"},"deleted":false,"location":{"country":"USA","state":"USA-CA","metro":"USA-SFO","region":"USA-SFO-NOR","county":"USA-CA-SON","city":"USA-SFO-STR","zipcode":"USA-95403","formatted_address":"95403,Santa Rosa,CA,USA","lat":"38.502811","long":"-122.753536","accuracy":8,"geolocation_status":3}},{"id":2122663509,"source":"E_BAY","category":"SELE","external_id":"281686596936","external_url":"http://www.ebay.com/itm/Atari-2600-Miner-2049er-Tigervision-Game-Cartridge-1983-NICE-RARE-NR-/281686596936?pt=LH_DefaultDomain_0","heading":"Atari 2600 Miner 2049er Tigervision Game Cartridge 1983 NICE RARE NR","timestamp":1431114540,"annotations":{"price":"5.0","BIN price":"25.0","address":"Morgan Hill,CA,USA","listingtype":"AuctionWithBIN","zipcode":"95037"},"deleted":false,"location":{"country":"USA","state":"USA-CA","metro":"USA-SFO","region":"USA-SFO-SOU","county":"USA-CA-STL","city":"USA-SFO-MOG","zipcode":"USA-95037","formatted_address":"95037,Morgan Hill,CA,USA","lat":"37.168085","long":"-121.627038","accuracy":8,"geolocation_status":3}},
.
.
.
.

[brozeph]

Now have this running properly in the staging environment in a container...

NOTE: The URL below will only work if you add your IP to the security group "Staging App" in AWS.

http://52.11.249.169:8884/v1/status

{
  "anchors": {
    "averageRequestTime": 2954.6111111111113,
    "totalErrorCount": 35,
    "totalRequestCount": 36,
    "totalRequestTime": 106366
  },
  "polling": {
    "averageRequestTime": 2104,
    "totalErrorCount": 0,
    "totalRequestCount": 1,
    "totalRequestTime": 2104
  },
  "postings": {
    "averagePostingCount": 15,
    "averageSaveTime": 4088,
    "totalErrorCount": 0,
    "totalPostingCount": 15,
    "totalSaveCount": 1,
    "totalSaveTime": 4088
  },
  "schedule": {
    "interval": "every 30 secs",
    "totalRunCount": 36
  }
}

[brozeph]

Also, turns out that this had to do with how the hashtagsell/docker-build script staging-posting-sync-ca.sh was configured - it used US-CA instead of USA-CA.

[brozeph]

Assigning back to @that1guy to verify locally - this issue pertains to the agent not working in a local environment. Please be sure to git pull as well to have the latest code and if the issue is reproduced, please post the configuration used.

[that1guy]

Things are now running great on my local machine. So well in fact that I'm concerned something may not be running correctly in cloud. Here is my config file on my local machine.

local.json on local machine.

{
  "logging": {
    "level": "trace"
  },
  "models": {
    "postings": {
      "schedule": {
        "interval": "every 30 secs"
      },
      "polling" : {
        "location": {
          "state": "USA-CA"
        }
      }
    }
  },
  "services": {
    "3taps": {
      "apikey": "f2862071ede0bbd93bee4f091c522a9e"
    },
    "postings": {
      "url": "http://localhost:4043/v1/postings"
    }
  }
}

Status details from my local machine after running for 10 min.

{
  "anchors": {
    "averageRequestTime": 2305.9583333333335,
    "totalErrorCount": 51,
    "totalRequestCount": 72,
    "totalRequestTime": 166029
  },
  "polling": {
    "averageRequestTime": 1561.3333333333333,
    "totalErrorCount": 0,
    "totalRequestCount": 24,
    "totalRequestTime": 37472
  },
  "postings": {
    "averagePostingCount": 5.476190476190476,
    "averageSaveTime": 3002.285714285714,
    "totalErrorCount": 8,
    "totalPostingCount": 115,
    "totalSaveCount": 21,
    "totalSaveTime": 63048
  },
  "schedule": {
    "interval": "every 30 secs",
    "totalRunCount": 74
  }
}

Why are the status details @ http://52.11.249.169:8884/v1/status indicating we've only had total save count of 3?

Status of sync agent running on AWS as of 1:32pm. pst.

{
  "anchors": {
    "averageRequestTime": 3100.6744585768306,
    "totalErrorCount": 5815,
    "totalRequestCount": 5818,
    "totalRequestTime": 18039724
  },
  "polling": {
    "averageRequestTime": 4201,
    "totalErrorCount": 0,
    "totalRequestCount": 3,
    "totalRequestTime": 12603
  },
  "postings": {
    "averagePostingCount": 5,
    "averageSaveTime": 8053.333333333333,
    "totalErrorCount": 2,
    "totalPostingCount": 15,
    "totalSaveCount": 3,
    "totalSaveTime": 24160
  },
  "schedule": {
    "interval": "every 30 secs",
    "totalRunCount": 5818
  }
}

[that1guy]

FYI between 1:32p and 1:45p I've ingested approx 5000 items on my local machine.. just to put things in perspective.

[that1guy]

Not relevant to today's issues.