I get error: This webhook is not valid, you should add the token to your API and try again later.

[pdichone]

Hi I was able to go through the youtube tutorial on setting up the webhook with node.js. All works fine, however when I go back to the gridscan dashboard, I get the "This webhook is not valid, you should add the token to your API and try again later." error every time.

Any ideas why this is happening?

Thank you.

[via-main]

Hi pdichone, Can you tell me a little bit more about your current setup? The following are some things to check to make sure your severs are ready to accept our webhooks: -Are you currently attempting to use this webhook with our example application or with a live site? -When the webhook is being sent out, is the application that is taking this request accessible via the web?(Either a web application behind a URL or using port forwarding on a local application) -When the request is received by your server, is it responding to the GET request from the webhook using the correct secret/validator and accepting the POST request afterward if it responded to the GET correctly?

[pdichone]

Hi jschrader00,

Thanks for the quick response. I really appreciate it. To answer your questions.

1. Yes, I am trying to mimic what the examples show. As I mentioned earlier, I am able to run on my local machine (terminal and localhost) no problem, but not able to validate.
   Maybe the reason why this is happening is because, when I go to the dashboard to view "Sample Wifi Gridscan Data" and "...Bluetooh Gridscan Data" all I see is the loading icon saying " scanning grids", and it never shows any sample data.

2. Yes, I am able to access all of that in my local host, just like in the youtube video tutorial. Although, I want to be able to set it all up on a live site ( will need some help with that as well).

3.Yes - GET request seems to work fine Express server listening on port 8080 "GET /gridscan 200 2.841 ms - 44 GET /css/style.css 404 2.132 ms - 314 GET /gridscan 304 0.646 ms - - GET /gridscan 304 0.206 ms - Express server listening on port 8080 GET /gridscan 200 2.841 ms - 44 GET /css/style.css 404 2.132 ms - 314 GET /gridscan 304 0.646 ms - - GET /gridscan 304 0.206 ms - -

[via-main]

pdichone, No problem at all, we're happy to help!

Looking at your webhook records, I'm seeing that you're using the URL 'http://localhost:8080/gridscan'. Since this webhook is sent from our server to your application, when using the example application you'll need to enter your external IP address instead of 'localhost' and will need to make sure that port 8080 is forwarded to the machine running the example application on.

Please let me know if this works for you or if you have any further issues.

[pdichone]

I have just tried that, and updated the webhook Posturl to point to my IP address and getting nowhere at this point. Sorry for the noob issues - this is my first time setting this up.

[pdichone]

I now have it pointing to my-ip:8080/gridscan and now I can't access it on my browser

[via-main]

You have the right IP address in there now, and you said you had port 8080 forwarded to the machine you're running this on? If port 8080 doesn't work, you can modify the application to use port 80 and try using the URL 'http://24.16.145.242/gridscan' to validate with. You can modify the file 'www' in the apis/nodejsexpress/bin folder on line 4 to use 80 instead of 8080. Once you've updated the application's port, you'll need to update your port forwarding settings on your router to then forward port 80 to the machine running the application.

As to your second post: What URL are you using to access the locally running application via your browser?

[pdichone]

I am using the 'http://24.16.145.242/gridscan' to access the locally running application, and I am getting a 404 - Not found.

I feel like I am missing a lot of things :(

Modified the www file too.

[via-main]

Ah, that explains why you were not gaining access to your locally running application. Since your machine is the one running the application, you will access it using locahost and whichever port it is currently on. I'm sorry, just have to confirm, is your port forwarding setup in the way I mentioned before? Without the port forwarding your locally running application cannot be accessed via the web and therefore we cannot POST data to it.

[pdichone]

Since, the ultimate goal is actually to have all of this running on an actual, remote server like AWS, is there a tutorial that would walk me through that process?

Essentially what we want to be able to do is the following:

Use the webhook to post the updating data to our server, so that I can then do a GET and take that information and consume it on an app I am going to be building.

[pdichone]

To answer your most recent question: -Are you referring to the usage of the IP address?

[via-main]

If the example application we provided were running on a remote server such as on AWS and could receive requests via a URL that is accessible to the web, then you would be able to see the data coming in via the console on that remote server.

The reason I mentioned the port forwarding setup is that the machine you are running the application on is not setup to receive requests over the internet like a remote server would, therefore your router would need to be aware that when something sends a request to your IP address and port it needs to be routed to the machine running the application. When you have a web server setup and with a URL, the port forwarding is not necessary. Once you have your remote sever setup, you can run our application on that server (start by using port 80). You should be able to validate your webhook and start receiving data at that point.

[pdichone]

I see. We have AWS web server and I am going to go that route since that's the ultimate goal anyway. So, you are saying that to set it all up with AWS I would follow the same instructions? Is that correct?

[via-main]

If you have a running web server in AWS, you should be able to follow those instructions on the remote server and have the application running. By the end of tomorrow we're going to be releasing a guide to walk through the process of setting everything up in AWS with the example application and it will include detailed information regarding the setup of everything necessary to run our example application. Hopefully this new guide will help resolve any issues if you run into any while running this application on AWS.

[via-main]

Good afternoon pdichone,

Our new guide to create a simple setup and receive GridScan data using AWS and our example application has been released, if you would like to use that as a quickstart.

[pdichone]

Wow! Thank you jschrader00 will get into it as soon as possible. This is great! Really appreciate it!

[via-main]

No problem at all! Let me know if you have any questions as you're working through it.

[pdichone]

So everything is working on AWS side of things, however when I add an actual webhook in the dashboard, it won't show me. It showed the added webhook at first, and then I hit "validate" button and gave the same error I have been getting all along, and next the webhook disappeared altogether from the dashboard. I even created a new one, and still nothing. Any ideas?

[via-main]

Looks like your webhooks are not using the Secret specified in the guide, I modified one of the two webhooks you have setup to use the 'Example-Secret' value specified in the guide, clicked the validate button and it validated successfully. Can you try this with your other webhook to confirm that this works on your end? Also, I am seeing both records, are you currently filtering the webhooks table?

[pdichone]

@jschrader00 On another note, I am able to see the log ( downloaded file) from the AWS app, yet the webhook is gone from BeaconGrid dashboard...

[via-main]

I see what the issue was in terms of the visibility on the dashboard, this issue has now been resolved and you should be able to see all of your webhooks again

[pdichone]

Perfect. I can now see the validated webhook.

So, my next question is the following: is there a way I can get then the direct link where I can use to do a GET and then parse all of the webhook json info?

In the quickstart tutorial you mentioned extending the Listening server code to spit out the JSON object - var gridScanObj = JSON.parse(req.body);

So I dug the app.js file for the express node.js project that we added to AWS, and I am trying to figure out where exactly I add the pice of code in order for me to then be able to get the JSON link I get perform a GET and parse the beacons information, if that makes any sense?

Please accept my apologies for asking too many questions.

[pdichone]

@jschrader00 Also, I was under the impression that if I went to to root folder ../gridscan I would get the json tree pertaining to the beacons. Now all I get is

{
token: "token-code-here"
}

[via-main]

Line 59 of the gridscan.js file in the routes folder is where the JSON formatted data is printed out. I believe that in this case, the request body that is sent to the running application from BeaconGrid is already turned into an object when reading the req.body object field rather than being a string that needs parsed due to how JavaScript is handling the headers of the received POST request. However if it is not the case, it should be able to be parsed via the JavaScript code included in the guide.

[pdichone]

I have updated the gridscan.js file:

 console.log('-----GRIDSCAN-POSTED-MESSAGE----');
  console.log(req.body);
  var gridScanObj = JSON.parse(req.body);
  console.log('---------------------------------');
  res.json({
    info: gridScanObj
    //token: secrets.validator
  });

So that instead of showing the token secret, it spits out the json object. In order to update this file, will have to create a new AWS app, or is there a way to reupload or update files?

[via-main]

I've now added a section of this example application's guide: 'Updating the example application in AWS' that should explain how to update an AWS application's source code. Let me know if you have any further questions.

[pdichone]

Thank you. I did upload a new version and all looks good but then when I go to the url to check, I get the 502 Bad Gateway - error.

[via-main]

When you zipped up the files to upload them, did you zip it up such that the files were at the first level of the directory or the second? First level directory being: -secrets.json -run.js -package.json -app.js -(Folder)node_modules -(Folder)routes -(Folder)views

Second level directory being: -(Folder)nodejs-express -secrets.json -run.js -package.json -app.js -(Folder)node_modules -(Folder)routes -(Folder)views

If when you unzip and go into the unzipped folder, the directory looks like the second one, then that would be the issue since the zipped code needs to be at the top directory.

If when you unzip and go into the unzipped folder it looks like the first then it sounds like your code may have bugs in it. If this is the case, what code did you change that might be causing the problem?

[pdichone]

Ok, so I was using the old zip folder which didn't have the run.js file. I downloaded the new one ( provided in the new tutorial) and changed the gridscan.js file from:

router.get('/', function (req, res) {
  res.json({
    token: secrets.validator
  });

to:

router.get('/', function (req, res) {
  res.json({
    token: secrets.secret
  });

and zipped up the folder and uploaded. Even with that minor change, I get a 502 Bad Gateway error. I did this to test, just to make sure I wasn't changing anything major that could cause a bug.

Also, the order of the directory is like the first order.

Not sure what's going on.

[via-main]

I've just followed the steps and downloaded the most current version of the repo, zipped up the code in the nodejs-express folder and re-uploaded to an example environment and the application ran as expected.

I then modified the code in the way you specified that you have, then zipped and reuploaded and it also ran as expected (though since the field being returned in that body is now different, the URL will work but the webhooks will fail to post since it modifies the handshake logic between BeaconGird and the running application).

I'm not sure what is occurring with your setup in this case. Are you clicking the URL on the AWS Elastic Beanstalk application dashboard to test it out and getting the 502?

[pdichone]

Yeah I am not sure either why this is just not working for me anymore. I went ahead and deleted all of the re-uploads versions and downloaded a fresh zip file from the repo, and now deploying. Will then reupload a modified zipped file and see if this continues. If so, I will just start it all from scratch - deleting the grid webhook, and start all over from scratch. "I'm not sure what is occurring with your setup in this case. Are you clicking the URL on the AWS Elastic Beanstalk application dashboard to test it out and getting the 502?" - Yes.

So for the return of the json object I need, do I put the JSON.parse(req.body) inside the get like this:

  router.get('/', function (req, res) {
  res.json({
    token: secrets.secret,
    myJson: req.body
  });

since req.body already is in JSON format?

Will let you know the results in a few minutes. Thanks for your help. I really appreciate it.

[pdichone]

So I downloaded the error log that occurs when I update and deploy a new code version:

npm ERR! node v6.10.0
npm ERR! npm  v3.10.10
npm ERR! path /var/app/current/package.json
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall open

npm ERR! enoent ENOENT: no such file or directory, open '/var/app/current/package.json'
npm ERR! enoent ENOENT: no such file or directory, open '/var/app/current/package.json'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! Please include the following file with any support request:
npm ERR!     /var/app/current/npm-debug.log
npm ERR! Linux 4.4.51-40.60.amzn1.x86_64
npm ERR! argv "/opt/elasticbeanstalk/node-install/node-v6.10.0-linux-x64/bin/node" "/opt/elasticbeanstalk/node-install/node-v6.10.0-linux-x64/bin/npm" "start"
npm ERR! node v6.10.0
npm ERR! npm  v3.10.10
npm ERR! path /var/app/current/package.json
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall open

npm ERR! enoent ENOENT: no such file or directory, open '/var/app/current/package.json'
npm ERR! enoent ENOENT: no such file or directory, open '/var/app/current/package.json'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent ....

It looks like it can't find the package.json file, which I looked and the uploaded zip file contained all the necessary files.

[pdichone]

Alright, figure out the issue - it was the directory issue you mentioned earlier: I needed to actually zip the contents of the project folder, not the project.

I made a few changes inside of router.get as follows:

 router.get('/', function (req, res) {
  res.json({
    token: secrets.secret,
    mObject: req.body
  });

And I am getting an empty object:

{
token: "Example-Secret",
mObject: { }
}

This, according to what you mentioned earlier, is due to the handshake modifications. How do I re-establish that handshake so I can actually get the beacons data being posted?

[via-main]

The fundamental way that our webhooks work is that when trying to send data out via webhook, BeaconGrid servers first send a GET request to the URL specified in the webhook. This GET request asks for the validator to verify the identity of the URL we are sending data to. If the server behind the URL that the GET request was sent to responds with the validator (as this guide's code does), then BeaconGrid servers send out a POST request with the data the webhook is delivering within the body. So, changing the code in the GET request handler in our example application in any way such that the validator is not in the response to the GET will break the handshake logic and thus you will not receive the POST request that normally follows.

[pdichone]

I see. So how do I get the POST request then? How do I re-establish the handshake? Do I have to perhaps create a new router with a different path? like:

   router.get('/newrouter', function (req, res) {
       res.json({ 
           mObject: req.body
         });

  });

Or do I have to create a new webhook in the dashboard (in the Gridscan) and re-do everthing?

[via-main]

Your code for the GET request handler in the code should be:

router.get('/', function (req, res) {
  res.json({
    token: secrets.validator
  });
});

If you want to do anything with the POST request with the payload of data, then you need to modify the POST request handler to do something other than just console logging the results

router.post('/', function (req, res) {
  console.log('-----GRIDSCAN-POSTED-MESSAGE----');
  console.log(req.body);
  console.log('---------------------------------');
  res.json({
    token: secrets.validator
  });
});

So instead of the console.log(req.body); you can write code to save it to a database, forward it somewhere else, etc..

[pdichone]

@jschrader00 Thanks for your insights. I have been looking for a way to do just that ( I am new to NodeJs) and so I am not really understanding how to re-direct/forward or pass that JSON payload to a different directory or url so I can then see the JSON data and able to consume it as a client. It may be trivial, but I haven't been able to find a solution for that. Do you mind to give me some pointers or direct me to the right direction on that?

Any help, pointers would be highly appreciated.

Thank you

[via-main]

@pdichone Certainly, I can help with that. Where is it you eventually want the data to end up? Rather than passing it on, perhaps we can come up with a way to get the data directly to your datasource.

[pdichone]

@jschrader00 Thank you. Well, all I really need is to be able to make a GET request on client side ( will be building apps that will be consuming each beacons data) and then be able to use it - just like an API. Since the beacons data is updating periodically, I really don't think it would be a good idea to even store that data in a database; as long as we are able to redirect/send the JSON payload to say:

http://mybeacon-env.us-west-2.elasticbeanstalk.com/beacons-data and then I can just point to that url to fetch:

[ { timeDetected: '2017-05-19T18:32:54.063640Z',
    nearbyBlemac: '00:1E:C0:21:CC:1C',
    blePrivate: '0',
    bleUUID: 'Brcst:0201040303AAFE1716AAFE10FD0262677269642E696F2F76696B3341544A38',
    rssi: -62,
    deviceMac: 'C47F5101511F',
    humanName: 'HE#3',
    timestamp: 1495218781545 },
  { timeDetected: '2017-05-19T18:32:54.063640Z',
    nearbyBlemac: 'EE:29:3F:5C:CB:43',
    blePrivate: '1',
    bleUUID: 'FE9A',
    rssi: -57,
    deviceMac: 'C47F5101511F',
    humanName: 'HE#3',
    timestamp: 1495218781545 },
  { timeDetected: '2017-05-19T18:32:54.063640Z',
    nearbyBlemac: '0C:F3:EE:0C:15:AA',
    blePrivate: '0',

That would be sufficient for now. Does this make sense?

[via-main]

It sounds to me like you're looking to have this information be coming from an API rather than a Webhook. It sounds like you'd like to have the mobile application be able to query data from our servers directly rather than using a webhook for the realtime sensor information being sent out to where you want the data. Am I understanding that correctly? If that is the case we're more than happy to extend our existing BeaconGrid API to allow the querying of sensor data. Can you tell me a bit about what parameters you would like to query on when searching for this data?

[pdichone]

Right! Correct! Yes!.
Mostly will be querying the entire payload of what the beacons are pickingup and posting. The idea behind this is so that, in the app, I am able to show clients realtime data from the beacons that are active.

So, say if I have 10 beacons, I would like to be able to pass parameters that will allow me to choose say "beacon1" or "beacon2" and then get all the realtime data for those beacons.

That's all. Simple as that.

[pdichone]

So for today ( I am in time crunch ), if I could just get the payload url where I can parse the data, it would be a huge win :)

[via-main]

That sounds good to us. At the moment we do not have this feature available with our external API. We'll start developing the addition to our BeaconGrid API immediately and you can expect to see this feature by the end of next Tuesday or Wednesday at the latest. When you mention that you want to see realtime data, at what rate do you think you'll want to be querying on that data for the mobile app?

[pdichone]

I don't think I'll need it needs to be at a higher rate - it only needs to be refreshed regularly so that at least it our partners are able to see that indeed the data is realtime.

Would mind, for today, to get me at least a url where I can consume the JSON data, as specified earlier? I just need to be able to have a little demo today to show. Once the external API is done, I will use it then.

[pdichone]

I would really appreciate it.

[via-main]

Certainly, am I correct in assuming you are looking for the Javascript code by which you can forward the data on to a URL?

[pdichone]

Yes, so that I can do a GET and in the client side to fetch that data. All of this is inside the post request in the webhook

  console.log('-----GRIDSCAN-POSTED-MESSAGE----');
  console.log(req.body);
    // code here to push the req.body json object to another url so I can consume it :)
  console.log('---------------------------------');
  res.json({
    token: secrets.validator
  });
});

[via-main]

So we can get this done in a timely fashion since you mentioned you were on a time crunch, would you like to hop on a phone call/Google Hangout/ platform of you choice? I can help walk you through the process and be better able to answer your questions in realtime, if this works for you.

[pdichone]

Yes absolutely, my gmail email is pdichone@gmail.com. We can do google hangout.
Thank you!

[via-main]

Excellent, I'll send out the invitiation

[via-main]

I've sent out the invitation and am ready whenever you are. I sent it from jschrader@beacongrid.com