Masternodes / Network

[spy0012]

Masternodes and network do not show on the pages, the cron runs but no data parsed

I know this WIP just trying to iron out some issues before our coin launch, we really like the modifications you have done.

Also the icons are really close to the data in the blocks see above image were maternodes is null the icon is right next to null no space same as supply and diff ect in the blocks along the top.

[uaktags]

Icon issue was noticed yesterday, it will be an easy fix. Masternode, i'll work on something for ya, but MNs are proving to be a coin specific (no standard in the crypto-community) on what RPC is available. I may have to break the sync script down to a coin-specific thing (similar to the idea i'm having with coin_commands)

[spy0012]

Mine is based off Pivx so that must be the reason for it not picking up the MN data, because some semi modified explorers you have to choose what type of coin your using.

Appreciate you looking into this

[uaktags]

Yea, Vulc is what I'm using to test and too based off of Pivx (most MNs i've seen are) but I've seen things diverge greatly from there in terms of MN specific commands. Some keep all of PIVX and add only a few of their own, and some remove/replace some of PIVX. Basically what we need is:

getmasternodecount
getmasternodescores ( blocks )
getmasternodewinners ( blocks "filter" )

getmasternodecount is the one we use for the totals, but like with Vulc, you only get the same Online,Stable and Enabled, so there's no need for the second parameter:

{
    "total" : 1417,
    "stable" : 1417,
    "obfcompat" : 1417,
    "enabled" : 1417,
    "inqueue" : 1387,
    "ipv4" : 903,
    "ipv6" : 259,
    "onion" : 105
}

[spy0012]

Initially i got an error which was strange about spaces and tabbing, but i recopied and it works great

[uaktags]

Can you confirm (i think you mentioned it before) if the following works:

node scripts/masternodes.js

[uaktags]

Being PIVX-variant, you should be fine, but just want to confirm where we stand.

[spy0012]

When i run that i get a lot of errors

  name: 'MongoError',
  index: 0,
  code: 11000,
  errmsg: 'E11000 duplicate key error collection: explorerdb.masternodes index: addr_1 dup key: { : null }',
  [Symbol(mongoErrorContextSymbol)]: {} }
Masternode does not exists in DB ADD TXID:undefined OUTIDX:undefined
{ MongoError: E11000 duplicate key error collection: explorerdb.masternodes index: addr_1 dup key: { : null }
    at Function.create (/var/www/vhosts/explorer/node_modules/mongodb-core/lib/error.js:43:12)
    at toError (/var/www/vhosts/explorer/node_modules/mongodb/lib/utils.js:149:22)
    at coll.s.topology.insert (/var/www/vhosts/explorer/node_modules/mongodb/lib/operations/collection_ops.js:844:39)
    at /var/www/vhosts/explorer/node_modules/mongodb-core/lib/connection/pool.js:532:18
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)
  driver: true,
  name: 'MongoError',
  index: 0,
  code: 11000,
  errmsg: 'E11000 duplicate key error collection: explorerdb.masternodes index: addr_1 dup key: { : null }',
  [Symbol(mongoErrorContextSymbol)]: {} }

[uaktags]

Nevermind, I see it in your message. Try using 3 backticks to format the message better.

Okay, so we're not getting any TXID or OUTIDX.

On your wallet, go to the console and do a listmasternodes command

pivxd listmasternodes

for instance.

You can also go to https://127.0.0.1:3001/api/listmasternodes change the url to your explorer's url

[spy0012]

in explorer i get - This method is restricted. in wallet i get


[
  {
    "rank": 1,
    "network": "ipv6",
    "txhash": "0b728da5fc086580783e96680a3d11965209698e2e9d74ee0aa2485e71d092e8",
    "outidx": 1,
    "status": "ENABLED",
    "addr": "MA2EZySdNuSKo19bQPVDwDYe2tRUa29fpP",
    "version": 70011,
    "lastseen": 1556596991,
    "activetime": 1070160,
    "lastpaid": 0
  },
  {
    "rank": 2,
    "network": "ipv6",
    "txhash": "7fe121d1e29b92480c6aaf03e90e5a95fe9cdfcdad7ab9213b7d8c9e8bf11852",
    "outidx": 0,
    "status": "ENABLED",
    "addr": "MUErk3VswTcEti4CLsXbFExkJFkg5hYxqK",
    "version": 70011,
    "lastseen": 1556597144,
    "activetime": 1091155,
    "lastpaid": 0
  },
  {
    "rank": 3,
    "network": "ipv4",
    "txhash": "877340b5b1c8cc0b418daf2e90e07673d340fe69f8c4173b8c006823d8cb4770",
    "outidx": 1,
    "status": "ENABLED",
    "addr": "MB6XZQE1JwGiAiDnxbprpUF5FVtK764Bgn",
    "version": 70011,
    "lastseen": 1556596928,
    "activetime": 1090872,
    "lastpaid": 1556597483
  },
]

On the network issue /api/getpeerinfo works in explorer yet it hasn't inputted them into DB and no errors running peers.js

[uaktags]

as a quick test, update your settings.json file:

  "commands_needed": [
    "base.json", "vulc.json"
  ],

Basically this is what I was talking about before where some commands aren't found across the community so I don't hardcode them. Instead, I want to make it more modular and have base commands and then extended/overwritten commands move up. Until then, this is just a basic system of "list the commands that are allowed, don't allow any others".

[spy0012]

masternodes total shows at the top, ran masternodes.js and now populates, just network left running peers.js doesn't throw any errors though

masternodes.js output

the options [createIndexes] is not supported
(node:3838) DeprecationWarning: collection.ensureIndex is deprecated. Use createIndexes instead.
Masternode already exists in DB Update TXID:0b728da5fc086580783e96680a3d11965209698e2e9d74ee0aa2485e71d092e8 OUTIDX:1
Masternode already exists in DB Update TXID:1900a0df8e1fbf003e6d1c7d813b7c913745c42995e238082aabef0e8d41179c OUTIDX:1
Masternode already exists in DB Update TXID:4233952f1dddc66526306af4e5590ea8928caf478bcb23b74c175af9661c624a OUTIDX:1
Masternode already exists in DB Update TXID:316fe4b4c98b37d092912eb67aaabd72cf36104d6082f2c70841d043671ef2e2 OUTIDX:1
Masternode already exists in DB Update TXID:7fe121d1e29b92480c6aaf03e90e5a95fe9cdfcdad7ab9213b7d8c9e8bf11852 OUTIDX:0
Masternode already exists in DB Update TXID:877340b5b1c8cc0b418daf2e90e07673d340fe69f8c4173b8c006823d8cb4770 OUTIDX:1
Masternode already exists in DB Update TXID:174469ef35552f3c40c37fe748dc087fc1d6907786eea67961ee3891ade9edec OUTIDX:0

[uaktags]

Great, mns are working. Network, I confirm too, has an issue. it appears that the ipstack portion is failing. I may opt to have that be a selector in the future. Allow people the option to either use Maxmind's Free GeoIP which runs locally in ram (needs a good size amount though), use IPStack, or go without. For now, I disabled the ipstack portion until I figure out why its failing, you'll lose the geo information.

[uaktags]

Oh yea. It's probably not working because we're all using my old API key that I left in the template as an example. Just checked IPStack's logs, over 120% utilization for the month :-P.

But yea, so IPStack (Free) is good for small testing/dev with less than 10,000 reqs a month (which means 10,000 ip addresses a month). Maxmind (free) would be better for a free option, but may not be entirely accurate (we're just after the country though).

IPStack subscription of 9.99 is great for 50k lookups a month or less. Anything more, I'd recommend with Maxmind for 25$/month and you host it locally.

I'll start piecing together a solution where you can set what you want in the front-end.

[spy0012]

I use my own ipstack key in explorer and I know I haven’t used up the quota

[uaktags]

Hmm well that was worth the thought. I tested the script and it refused to hit the call back for the ipstack request, so there's something off with that either way. After the last commit, I was at least able to get the db to update which allows the /ext/ API to work for network.

[spy0012]

Modified peers.js and it errors at https://github.com/uaktags/explorer/blob/fc161d10790ce808185d9264a189c3723bec6240/scripts/peers.js#L48

full output

 /var/www/vhosts/explorer/scripts/peers.js:48
      lib.syncLoop(body.length, function (loop) {
                        ^
TypeError: Cannot read property 'length' of undefined
    at Request._callback (/var/www/vhosts/explorer/scripts/peers.                                                                                                                                                       js:48:25)
    at self.callback (/var/www/vhosts/explorer/node_modules/reque                                                                                                                                                       st/request.js:185:22)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at Request.onRequestError (/var/www/vhosts/explorer/node_modu                                                                                                                                                       les/request/request.js:881:8)
    at emitOne (events.js:116:13)
    at ClientRequest.emit (events.js:211:7)
    at Socket.socketErrorListener (_http_client.js:387:9)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Modified the line above to show

lib.syncLoop(body[0].length, function (loop) {

Instead of 

lib.syncLoop(body.length, function (loop) {

and it displays the network ip's and details, also shows country not sure if that is because they were in the DB already

Output of peers.js after the change

(node:26122) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
(node:26122) DeprecationWarning: collection.ensureIndex is deprecated. Use createIndexes instead.

Not sure about the error as i am more mysql than mongo

[uaktags]

Hmm, body should definitely not be an array. Not sure why body is coming up with there being multiple of them (body[0] and body[1]). Line 48 should NOT be showing country unless your wallet/blockchain for whatever reason provides that.

The main logic of this loop is that it's grabbing the json from your wallet (getpeerinfo), and that info does not yet touch the db. So it grabs that on line 43, 45-47 not yet implemented (but great for testing if you want to comment out 45 and 47 leaving 46 there to purge the DB of peers), then starts a normal foreach loop. 49-60 are just formatting purposes for ipv4 vs ipv6. Line 61 we check the data similar to "select * from peers where address = address". If we get something 62-82 is in play, other wise go to 83 and create our peer because it doesn't exist in the db.

So only 83 and 72 are the lines that activate our IPStack check, and its only after that IPStack check that a new db insert is executed.

So yea...im not sure why we're getting that result haha.

[spy0012]

Okay so i emptied peers from DB, and then run peers.js and i get the same error as above it did complain about body[0] so removed it but still the same error lib.syncLoop(body.length, function (loop) {

Not loading peers

[uaktags]

Alright, I'll start working on a more commented peers.js to figure out what is going on, but I'd recommend stepping through the process manually to try and see what it looks like. Going to /api/getpeerinfo, the very first characters you see should be 0: id: or [{ id depending on if your browser shows Raw JSON or parsed/prettified json. Then checking what body is set to, like by doing a console.log(body) on line 46-47. It should match exactly the same as the raw JSON from /api/getpeerinfo [ { id.

If not, we need to know what your wallet is doing with the command getpeerinfo.

[uaktags]

@spy0012 , okay, in there I added:

"use_IPStack": true

as a default to settings.js, you can either leave this out of your settings.json, or you can add it to the peers section as shown in the template, but since it's TRUE by default, you're fine as is for now. So this re-enables the IPStack.

Body.length issue I believe has to do with there being an error in the call, so I added a check for request errors in peers.js. It'll fail out and console.log the error message.

Also, for the peers section, there's a setting for "purge_on_run": false this is great for these testing purposes, to clear your db on each run to go fresh. I've also uncommented the console logs for

console.log('Delete the db version:', peer[i].version.split(":")[1]); //remove
                  } else if(cmp(peer[i].version.split(":")[1], semver) == 0){
                    console.log('Do nothing, they\'re the same');

Once you can make it this far, we should be able to see what everything is doing.

[spy0012]

To answer your previous question /api/getpeerinfo in browser gives

[{"id":1,"addr":"94.177.251.97:9022","addrlocal":"213.136.86.60:58190","services":"0000000000000005","lastsend":1556727915,"lastrecv":1556727915,"bytessent":1243046,"bytesrecv":1910378,

Plus more of it as there are 13 peers

[uaktags]

Looks good so far

[spy0012]

So i did the commits and got the following from peers.js

{ Error: getaddrinfo EAI_AGAIN https:80
    at Object._errnoException (util.js:1022:11)
    at errnoException (dns.js:55:15)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:92:26)
  code: 'EAI_AGAIN',
  errno: 'EAI_AGAIN',
  syscall: 'getaddrinfo',
  hostname: 'https',
  host: 'https',
  port: 80 }

it hangs there, but i noticed that https:80 is incorrect as i am using 8080 as per the settings port in settings, i don't know were that port 80 is coming from. Also port 443 for SSL using var forceSSL = require('express-force-ssl');

[uaktags]

The worrisome part of your error message is the fact that node / request is saying that the host and hostname is https. Your error is a dns error, not a iquidus-explorer error. Basically it's saying "I don't know the IP for the host named https". Without seeing your settings.json, my guess is you've changed

"address": "127.0.0.1:3001",

to

"address": "https://example.com",

request is failing because of https://github.com/uaktags/explorer/blob/b159b0a142e169f8e7b780ee0b96ea9a76853613/scripts/peers.js#L57

So essentially, it's trying to do a request to http://https://example.com:8080/api/getpeerinfo. So remove the https designation from your settings, I'll also remove the :3001 from the settings.json.template as we shouldn't have it there since we already have "port" as a setting. We may need a new issue to track SSL support and remove any of these hardcoded requests like line 57 showed.

I may just remove the requests from this all together and go with using raw bitcoin-core rather than bitcoin-node-api for these scripts. There's no reason I can think of that we need to do a request to express, which then translates that to bitcoin-node-api and then eventually to bitcoin-core

[spy0012]

I don't have a port on my address i did have https://myurl.com now removed https:// and port separate as you said 8080

peers.js ran with no errors but no update on the page

Sorry to be a pain

[uaktags]

Okay, what outputs did we get from the script though? I'm less concerned with what shows on the network page and more concerned with seeing the console.log information

[spy0012]

When i runnode scripts/peers.js I don't get any output in terminal at it just runs

root@vmi254042:/var/www/vhosts/explorer# node scripts/peers.js
root@vmi254042:/var/www/vhosts/explorer#

Can't seem to find a log for the console either

[uaktags]

There's not going to be a log, it all gets outputted to console.

This almost can't be as something should be catching a console.log somewhere. Either Line57 fails and it console.log's an error, or continues and hits line 90,92, and 95.

On line 57 do this

} else {
console.log('calling getpeerinfo next');
request({uri: 'http://'+settings.address +

and then on line 60 do this

}else{
console.log('Response %s', response);
console.log('Body %s', body);
var livepeers = [];

Post output (but truncate the peers if it shows up).

[spy0012]

calling getpeerinfo next
Response [object Object]
Body [object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[uaktags]

Okay, that looks correct, so we're getting the peers (that's the objects, you can see them if you remove the "Body %s" part. Lets then do this:

livepeers[i] = address;
          console.log(address);
          db.find_peers(address, function(peer) {

            if (peer.length) {
              console.log("peer exists");
                for(i=0; i<peer.length; i++){

} else {
              console.log("peer doesnt exist, create");
              create_peers(address, body[i].version, version);  

I'm expecting to see:

127.0.0.1
peer exists
Delete the DB version

or

127.0.0.1
peer doesnt exist, create

Outputted.

[spy0012]

calling getpeerinfo next
Response [object Object]
94.177.251.97
peer doesnt exist, create

But if i run it again i get the same left response % in took body % out as you can probably tell

[uaktags]

This is what I'm expecting to see, this means that it's going through that far, the script is outputting and making it to the create_peers. Have you done anything else to see how far it makes it?

If not, then the question is, is your "settings.peers.use_IPStack" set to true? if so, then you can do this:

if(settings.peers.use_IPStack){
console.log('call out to ipstack');
    request({uri: 'http://api.ipstack.com/' + address + '?access_key=' + settings.peers.apikey, json: true}, function (error, response, geo) {
console.log('adding ip to database');
      db.create_peer({

This is literally the part I had to comment out because IPStack was failing to return for us, so we'll find out real quickly now if ipstack is working or not. If you don't get "adding ip to database" then the problem is with IPStack and/or your API Key. That's why fc161d1 removed IPStack and b159b0a put the selector back in there for IPStack.

[spy0012]

Can't get anything to run now

/var/www/vhosts/explorer/scripts/peers.js:50
mongoose.connect(dbString, { useCreateIndex: true,
^^^^^^^^

SyntaxError: Unexpected identifier
    at createScript (vm.js:80:10)
    at Object.runInThisContext (vm.js:139:10)
    at Module._compile (module.js:616:28)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)
    at startup (bootstrap_node.js:188:16)
    at bootstrap_node.js:609:3

[uaktags]

npm update while in explorer, undo anything that you may have changed on your system, and make sure you're running node scripts/peers.js from /var/www/vhosts/explorer. Not much I can really do for you on this though.

[spy0012]

I went back to the remove IPStack commit and got the following output

calling getpeerinfo next
Response [object Object]
94.177.251.97
peer doesnt exist, create
185.35.64.141
peer doesnt exist, create
94.177.242.241
added 94.177.251.97 to the peers database
peer doesnt exist, create
2a02:c207:2021:4471::1

This added ip's to the network page with country being - geo.country_name One of the ipv6 is strange - 2001:19f0:5401:258e:dd6c:c537:f764:7082 very long lol

added IPStack back in as you suggested and here is the output

calling getpeerinfo next
Response [object Object]
94.177.251.97
peer exists
Do nothing, they're the same
185.35.64.141
peer exists
Do nothing, they're the same

Still says geo.country_name as country

[uaktags]

It says geo.country_name because of https://github.com/uaktags/explorer/blob/4c9d83e36fe9d4bb2092ea46f284116d781d71b8/scripts/peers.js#L42, because there is no value to provide it since there is no IPStack being used. 2001:19f0:5401:258e:dd6c:c537:f764:7082 is a valid IPv6 address.

I'm getting really confused by what you did/didn't do.

1. You're on the latest commit for peers.js and it shows https://github.com/uaktags/explorer/blob/master/scripts/peers.js exactly as written
2. You've set peers.use_IPStack to false (because you're getting my hardcoded line, so you must have)
3. Your Network page is populated?

If yes to all, then the network page is working exactly as it should be, there's a problem with either your API key or your connection/response from IPStack (hence previous mentions), and this issue is basically closed. Next step would be for me to remove the Country from the Network page as per use_IPStack boolean.

If any of those 3 questions are no, then we have a problem and I can't help until they're all yes.

[spy0012]

My Peers.js matches yours except for the console.log entries which gives me

5.231.205.119
peer doesnt exist, create
call out to ipstack
207.180.224.32
peer doesnt exist, create
call out to ipstack

And nothing is added to the page, i deleted peers from db so they would be added fresh, but seems it gets stuck at IPStack, even though i have double checked my api key and i have only used

API Usage: | 0% (8 / 10,000) so i don't know why it fails there

[uaktags]

It could be your dns, it could be some firewall issue you're having from the server to http://api.ipstack.com, but it all targets that area. Set the setting to false until you've worked out that issue. You're being stopped at the request to IPStack which is why I have that setting in place.

[spy0012]

I tend to agree as, i have commented out

//request({uri: 'http://api.ipstack.com/' + address + '?access_key
and
//});

and the DB populates

[uaktags]

Don't comment that out, use the setting.

[spy0012]

Okay peers.js matches yours exactly, set use IPStack to false and nothing, no writing to DB

I think my easiest option is to either set country to Private instead of geo.country_name or just remove that portion of the table.

Thanks for your help on this.

[uaktags]

Wait, no that doesn't make any sense though. You were just getting results for "geo.country_name" which means it was already hitting

db.create_peer({
      address: address,
      protocol: protocol,
      version: version,
      //todo
      //semver: semver,
      country: "geo.country_name"
    }, function(){
      console.log('added %s to the peers database', address)
    });

which means that "settings.peers.use_IPStack" was already set to false. If you had done what you said here https://github.com/uaktags/explorer/issues/10#issuecomment-488809976 then you would have gotten an error that geo.country_name was undefined, you wouldn't have gotten the hardcoded "geo.country_name". If settings.peers.use_IPStack is set to false, then it HAS to be updating the database and it will be hardcoding the geo.country_name.....if you have peers.js matching the current repo version.

Something about what you're writing and what the repo has, isn't matching up at all right now. You're saying you're going to set the Country to Private, but that doesn't make any sense either, because you're also saying it's not writing to the DB when set to false.

is your settings.json setup correct with that variable?

 "peers": {
    "apikey": "myAPIKeyIsHere",
    "use_IPStack": false,
    "purge_on_run": false
  },

If it's setup like that, then it's writing to the DB unless you have some other error in place which is what all my edits previously were for.

[spy0012]

Okay let me get this right, here is what i have. settings.js is same as yours settings.json is set to

"apikey": "myownkeyhere",
"use_IPStack": false,
"purge_on_run": false

I have run peers and i get

So i must have missed something before

[uaktags]

Great, okay, so now you can either A) change the hardcoding of line42 to "Private" or wait until later, I'll add a check in the template so you don't have that hardcode in there.

[spy0012]

Will wait appreciate everything you have done and your patience Very strange, i have just used a old peers.js from one of the many explorers i have tried and this is what i get

(node:21321) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
(node:21321) DeprecationWarning: collection.ensureIndex is deprecated. Use createIndexes instead.

how ever my network page gives

https://pastebin.com/rzh3nVKN

[spy0012]

Thanks, just got the 3 columns, gonna look at modifying the peers.js that works for me, pastebin link, and see if i can add your modifications enable and disable of IPStack etc.

[uaktags]

Then I'd recommend forking the github repo, because if you then do any more pulls later, you'll overwrite what you have done custom. Just FYI.