Add KxoId support

[ghost]

Edit: KxoId offers Unique Ids. The registration requests happen within the ZeroNet network by using PeerMessage to broadcast registration requests. This allows people to still register even if their government has blocked the server's ip - as long as there is some path of peers connecting from their client to the server. Additionally, IP addresses are never leaked, and can be hidden with Tor.

[danimesq]

I don't think Nofish will implement it, since ZeroID is the default and working 24/7.

[ghost]

Mine'll work 24/7 after today - currently setting up a ZeroNet client for this.

[danimesq]

Is you planning to add more trusted nodes? I suggest @d14na, since he own a proxy.

[ghost]

Yes, I am, but first I need to make sure the code works correctly to allow for multiple trusted nodes/clients.

[danimesq]

ZeroNet could have a democratic system to not need only one private key?

[d14na]

Is KxoId an alternative to ZeroID?

My understanding is the ZeroID requires a centralized server; to handle the private keys (or certificates)?? I'm not sure if that's right.

From the little bit I've just read, KxoId would require a group of Trusted Peers?? Certainly an improvement. Is there any way to make it completely decentralized? I know that its possible with blockchain tech; but maybe that doesn't work with p2p, when everyone is seeding the data.

[ghost]

KxoId is one of the many alternatives to ZeroId. Others include KaffieId, CryptoId, PeakId, and PolarId.

I think blockchain may be possible with p2p like this.

Yes, ZeroId is centralized, along with PeakId and PolarId. KaffieId (and CryptoId I believe) are not, but the private key is public, so there's no way to ensure username/id uniqueness. They are essentially done locally. KxoId takes the best of both I guess. It uses the PeerMessage plugin to ensure requests to a central server are over the ZeroNet network (which means it works for users of tor as well I believe). But then I am going to add a group of trusted peers to make it less centralized.

[d14na]

It uses the PeerMessage plugin to ensure requests to a central server are over ZeroNet

oh, got it! i haven't started digging into PeerMessage yet. I've been waiting for the completion of Postal Service over Swarm to handle the communications for my app, but I'd consider PM as an option/alternative or even a replacement (no idea when PSS will be ready).

I think blockchain could be possible with p2p like this.

the issue is almost ALL things blockchain require crypto. not sure if that's what this community wants. but I'd like to make it an option and see

[d14na]

@krixano Would you mind confirming a concern I have about ZeroID. After I created d14na, a private key was displayed, which I saved and am now using to switch between accounts by saving it to master_seed in the users.json file. Its working fine.

However, is it safe to assume that the ZeroID server has this private key stored? Or does it just have the public key generated from it (Bitcoin address)? I'm considering giving the options for users to use the same master_seed (for their ZeroID) to manage their accounts (incl crypto balances) in Zeronet Explorer, but I won't do that if everyones private key is stored on a central server somewhere.

How does KxoId work? Will you (and the Trusted Peers) have any access to private keys? Thanks!

[ghost]

It does not have the private key. The public key is sent, it verifies the id, then creates a certificate with the public key and the user id. This certificate is sent back to you so that you can store it in your files for each zite. This certificate is used to verify your userid and data from other clients.

The private key never leaves your computer, even for KxoId, KaffieId, etc.

[d14na]

The private key never leaves your computer

ok great!

This certificate is sent back to you

I'm assuming that's the cert_sign?

On the topic if users.json, I see records for each zite with at least auth_address and auth_privatekey; even zeroid.bit has these. what's the purpose?

I hate to keep bothering you, but I just can't find this information anywhere. Thanks again!

[ghost]

auth_address is your identity address / public key. the private key is stored in users.json where no zite can access it (unless you own the zite, then you use the zite's private key in there to sign your zite when you click the sign button, but this is handled by ZeroNet's interface). This private key you use to create a signature for your own content.json, where the content.json contains hashes to all of your files (including data.json) so that those files can be verified also.

[d14na]

This private key you use to create a signature for your own content.json

I get that you wouldn't want to use the same key on every zite (similar to LastPass or any password manager), but can all those auth_privatekeys be derived from the master_seed? If not, then you definitely MUST save users.json (or risk losing access to all of your individual zite data) which is fairly inconvenient (if you're subscribed to many 100+ zites), compared to storing a single private (master) key.

BTW, I really like your Portal http://127.0.0.1:43110/1JBXrjCabLEWXmKJ2pJ4XhxA4rwYAEazKw

[ghost]

Ok, actually. I'm slightly wrong. The auth_privatekey's in the users.json file for all of the zites is just the private key for the zite's public key. Each zite gets a default public key. But they aren't used if you are logged in. If you are logged in, then the public and private key of your id is used. When you sign up with an Id provider, then the public and private key of that id becomes the default public and private key for the zite.

can all those auth_privatekeys be derived from the master_seed?

I'm not 100% sure, but I think maybe???

Btw, I think (iirc) most of these questions should be answered in my tutorials on DevCenter, especially the Users and Databases tutorial.

[wandrien]

@d14na @krixano

When you get to a site first time, a new private key is derived from your master seed:

auth_privatekey = derive(masted_seed, zite_address)
auth_address = bitcoin_address(auth_privatekey)

It is your default auth data, which is selected by default in the account selection UI under the name "No certificate".

If the site is an ID provider, it can sign your (local to that specific site) "no certificate" auth_address, so the address gets associated with the username@idprovider.

[wandrien]

You just added kxoid.bit to the lists of allowed signers?

Now with that change users with the same name are possible, so ZeroTalk must display the full qualified name user@provider instead of just a nickname.

[ghost]

@geekless Right, that's what I just said.

[d14na]

If you are logged in, then the public and private key of your id is used

@krixano ok, that seems reasonable

most of these questions should be answered in my tutorials on DevCenter

I'm gonna go through that entirely today.

When you get to a site first time, a new private key is derived from your master seed

@geekless appreciate that snippet. it confirms what my experiments have shown, that everything seems to come back to the master_seed. eg in users.json, I can (1) delete the entire object, then (2) set the main object's key (which is a public key) to "1", but as long as (3) the master_seed is set correctly, EVERYTHING works! automagically gets rebuilt upon visiting each zite.

Minimum users.json

{
  "1": { 
    "master_seed": "<32-byte random seed>"
  }
}

[d14na]

so ZeroTalk must display the full qualified name user@provider instead of just a nickname

@geekless @krixano

given that ZeroID is currently centralized (and I'm assuming has no plans to change that), it makes sense to support multiple providers. however, that doesn't seem as it would improve the user experience that currently allows for just the nickname.

tl;dr why not list/rank the "@" mentions based on the popularity of that nickname on the zite?

a thought i just had. when the user types the "@" mention, it could display a dropdown list which would not only filter by characters entered, but also sorted by ranking. the ranking could be determined by that user's (interactions, upvotes, etc) of the name on the zite's db. so user@zeroid.bit may rank higher than user@kxoid.bit for ME, but maybe the opposite for YOU. It seems fair, and would make it much easier to select the user's most likely target.

also, in terms of display. we wouldn't want to always have to show the fully-qualified name of a provider. perhaps an identicon-like symbol/graphic could be used to clearly identify which provider that name belongs to.

unfortunately, when it comes to having multiple users with the same nickname using the same provider, I just don't see how that works anywhere. I'd need to see an example of its usage.

EDIT: It would also be nice, if a user where able to associate multiple providers to the same account on a zite.

[ghost]

On KaffieId (and I believe CryptoId), people aren't restricted to the username they use because the private key is public so they sign their certificates locally themselves. The username is never used internally to verify users, only the certificate and keys. The usernames are only for people to easily identify others. Therefore, there could be two different id's with the same username, but not the same keys.

[d14na]

The username is never used internally to verify users

sure, but then how does that work with "@" mentions?

[ghost]

Usernames can still be used by the website, they just don't verify users belong to an id provider. In websites, they can get the username and the public address of an id. The public address is unique, the username isn't necessarily. That's why urls for ZeroTalk, ZeroMe, etc. use your private address to reference your posts - to ensure the url is unique.

Additionally, if two people have the same username from the same id provider and someone mentions that userid, then they both get the mention - even if they aren't the same id (they have separate auth_addresses). Basically, the user id's are just for easy reference for the users of the zite.

[d14na]

well, correct me if I'm wrong, but if I have superman@kaffieid.bit, and then 6 months later, some wannabe super decides to also be superman@kaffieid.bit, then wouldn't their mentions start showing up on my feed? it seems like it could open up to trolling. and I know how the internet loves to troll.

[ghost]

I've edited my post above

[d14na]

then they both get the mention

yeah, i think that needs to be avoided

[ghost]

Right, which is why ZeroId is centralized, and also why I'm creating this Id, so that this is prevented, but that the requests for registration still happen over/through ZeroNet.

The other option would be to completely get rid of id providers and only use the keys, but they are hard to remember, which is what they were supposed to help in the first place (so you don't have to remember a long public key).

Then there's identicons, which is actually helps this situation since they can be based on the public key - and I'll be adding this to all of my zites (using jdenticon library - KxoNetwork already uses this).

[d14na]

kxoid's approach works for me.

[ghost]

@d14na You should really look at my "Users and Databases in ZeroNet" tutorial. I'm rereading through it and I think it better explains stuff than what I did here. You might want to read at least the "The Basics" tutorial first though.

[d14na]

You should really look at my "Users and Databases in ZeroNet" tutorial

yeah, i'm reading everything on the zite now. I really like the whole structure. I'm curious as to how Questions and your ZeroExchange work together. Do you favor one over the other. Anyway to get them to share the same database?

[ghost]

The questions are different on ZeroExchange from Dev Center. I favor ZeroExchange because it was made after Dev Center and is much better (although there are small bugs that I still need to work on). So I would use ZeroExchange.

I could use the CORS permission and ZeroNet plugin to be able to show ZeroExchange questions on Dev Center, but Dev Center has basically been abandoned right now.

[d14na]

http://127.0.0.1:43110/DevCenter.bit/?/tutorials/the_basics

@krixano this NEEDS to be much more accessible. I feel like everything is starting to make sense now. I really had no idea how to use the sidebar, or that all that information even existed. eg. It used to be a pain in the ass to open the local directory of a zite, now its a simple click. Tracking data usage, provider identities, etc, etc, its all right there and I had no idea how to use it before.

Perhaps a link from the sidebar to a dedicated zite with this information would go a long way in helping n00bs.

[d14na]

So I would use ZeroExchange

Perfect. I'll make the move over AFTER I've absorbed all I can from DevCenter.

Also, I just saw this yesterday http://127.0.0.1:43110/142jqssVAj2iRxMACJg2dzipB5oicZYz5w/. Is this a good place to discuss development topics. I understand the old Dev.net has been abandoned?

[ghost]

I think some of the top developers have that zite. I, as well as gitcenter, zerolstn, and I believe glightstar, have it. Although, there hasn't been much activity on the zite for the past couple of months.

[d14na]

@krixano finished The Basics, so far so good, one question (it may be answered later, but I didn't want to forget)

When a visitor downloads the file, they first decrypt the digital signature with the zite's public key (the zite's address).

I'm assuming its this section

 "signers_sign": "GyyJkYW7ScPWeaKnF43EZnRpAqeOzisEQSLv8Uf+HDBYKkkQ8wXUo/RWFknu7FX0vw8Qxvjm8sQaIY/hjNFFsXs=",
 "signs": {
  "1CoDiNGYdEQX3PmP32K3pbZnrHJ2nWxXun": "HLdFFBAfMCx4lKVzxIxsonCptO0V/kXTW/sIEyP2hiGkfBSAduz9ejPyzBnZg/VVDIga5PGL2nvgAror3IwWTqg="
 },

This has bugged me from early on. If you could enlighten me as to "how" one decrypts the signature HLdFFBAfMCx4lKVzxIxsonCptO0V/kXTW/sIEyP2hiGkfBSAduz9ejPyzBnZg/VVDIga5PGL2nvgAror3IwWTqg= with the public key 1CoDiNGYdEQX3PmP32K3pbZnrHJ2nWxXun and what is the expected result? The sig looks like base64, but what is the decryption algo? Also, is signers_sign involved in the process as well? Thanks!

Starting "Tips and Tricks" now...

[d14na]

@krixano all wrapped up with Dev Center; what AMAZING detail. even created 2 plugins from Lola's tutorial, one to manage users.json's master_seed and another to view/parse log/* files.

Anyway, I just had one question regarding databases.

Not only is there a waiting period of 30 seconds between content publishes, but there is also a waiting period for all of the changes to be spread across all peers of the network.

Why 30 seconds? Is that simply for spam/ddos protection or is there a more (p2p) technical reason?

---

Also, went back thru the "official" docs, and discovered the encryption used in the signing process is ECIES. I'm not an expert at the cryptography stuff, but I believe ECIES needs both public and private keys. So more than likely (since there is only the public key and the signature available), the private key used to sign is the public key of the zite and the public key that I need to decrypt has to be derived?? @geekless Does that sound reasonable (just a guess), or if you happen to have a snippet you can share, that would be great, but I'll start experimenting with that later this afternoon.

Thanks again for all your help. Gotta make a quick appearance at this bbq, then I'm gonna start on some hard-core Zeronet coding.

Cheers!

[ghost]

I'm guessing the 30 seconds is there because file updates can happen quite a lot - every single post you make on a zite - and every update uses the hard drive. All of the dynamic stuff of a zite uses files, then locally, the information from these files are aggregated into the local-only database.

I don't know anything about encryption methods. I just know a little of a basic overview of public and private key encryption (sign with private key, you can decrypt with public. Sign with public, you can decrypt with private). That's why I don't go too into depth on any of that in the tutorials.

[wandrien]

@d14na:

The public key is generated from the private key by the encryption algorithm. (It's pure math, I don't know much about the details.)

To be more specific, the site address is not just the public key, but a hash sum of it (encoded in the Bitcoin address format). So:

public_key = public_from_private(private_key)
site_address = bitcoin_address_encoding(hash_sum(public_key))

When the site author publishes the site, the sign is generated and saved along the content:

sign = sign_for_content(private_key, content)

The sign is a number that, being applied to the same content, produces the public key. When a host verifies the site integrity, it does:

public_key = public_key_from_sign(sign, content)
site_address = bitcoin_address_encoding(hash_sum(public_key))

It compares the generated site_address with the actual site address. If they are equal, the sign is correct.

---

The real implementation is more complicated. It adds one more level of indirection: the site private key is used to sign the list of allowed signers, and then the signers' private keys can be used to sign the site instead of the master private key.

But I saw a bug reported on that matter: http://127.0.0.1:43110/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t/repo/issues/view/?1P4w4Rvh4vS9r6G5xTiSh8qEW87EKZtnJB/0@1Md6YJNZX5yrdSaL9mmrKZNMAfULoXzMX2 , so I'm not sure if it actually works as expected.

[wandrien]

@d14na:

The real code:

Signing the signers list:

        if inner_path == "content.json" and privatekey_address == self.site.address:
            # If signing using the root key, then sign the valid signers
            signers_data = "%s:%s" % (new_content["signs_required"], ",".join(valid_signers))
            new_content["signers_sign"] = CryptBitcoin.sign(str(signers_data), privatekey)

Verifying the signers list:

                    valid_signers = self.getValidSigners(inner_path, new_content)
                    signs_required = self.getSignsRequired(inner_path, new_content)

                    if inner_path == "content.json" and len(valid_signers) > 1:  # Check signers_sign on root content.json
                        signers_data = "%s:%s" % (signs_required, ",".join(valid_signers))
                        if not CryptBitcoin.verify(signers_data, self.site.address, new_content["signers_sign"]):
                            raise VerifyError("Invalid signers_sign!")

Verifying the content:

                    valid_signs = 0
                    for address in valid_signers:
                        if address in signs:
                            valid_signs += CryptBitcoin.verify(sign_content, address, signs[address])
                        if valid_signs >= signs_required:
                            break  # Break if we has enough signs
                    if valid_signs < signs_required:
                        raise VerifyError("Valid signs: %s/%s" % (valid_signs, signs_required))
                    else:
                        return self.verifyContent(inner_path, new_content)

[d14na]

@geekless its been a long day, lol

I found a javascript/node library that handles the signing and verification https://www.npmjs.com/package/bitcoinjs-message. More importantly, it WORKS in React Native. Unfortunately, the same can't be said for bitcoinjs-lib. I've spent most of the evening trying to hack it, but I think I'm just gonna pin that for another day (or week).

signers_data = "%s:%s" % (new_content["signs_required"], ",".join(valid_signers))

ok, yes! I finally figured that out. Basically it comes down to 1:<bitcoin-address/public-key>. And bitcoinjs-message decodes that perfectly, so I know that library is working.

I'm not sure how important this step is. I've ONLY ever seen one signer "signs_required": 1 and its always been the public key of the zite. Perhaps this will change in the future though.

sign = sign_for_content(private_key, content)

This is what I'm working on now. I'm trying to figure out what content is. Once I have content, then I'll be able to validate that content.json's contents are VALID, most importantly the files: {}. I've already got the individual file's SHA512 hashes validating correctly.

Once I can validate the signature of content.json itself, then I'm confident Zeronet Explorer will be safe to use for STATIC zites. However, writing/updating data will require signing /users/.../content.json and that still needs bitcoinjs-lib, but maybe I can find a lighter RN-friendly substitute.

If I can get this snippet working in RN, I think I'm good:

const bitcoin = require('bitcoinjs-lib')

const keyPair = bitcoin.ECPair.fromWIF(<PRIVATE-KEY-HERE>)
const { address } = bitcoin.payments.p2pkh({ pubkey: keyPair.publicKey })

const pk = keyPair.__d
const signature = bitcoinMessage.sign(<CONTENT-TO-BE-SIGNED>, pk, keyPair.compressed)

Thank you very much for confirming this whole process, as it keeps me moving forward without having to second guess everything. Security is VERY important to me and I want to make sure its done right. Afterwards, making everything nice and pretty is fun!

[d14na]

@geekless maybe i'm dreaming, but I think I got it working

/* Retrieve the signature. */
const signsSignature = contentJson.signs[address]

/* Delete signs (as we can't verify ourselves in the signature). */
delete contentJson.signs

/* Convert the JSON to a string (BUT mimick Python `json.dumps` spacing). */
contentJson = JSON.stringify(contentJson).replace(/":/g, '": ').replace(/,"/g, ', "')

/* Verify the Bitcoin signature. */
const isValid = bitcoinMessage.verify(contentJson, address, signsSignature)

JSON.stringify(contentJson).replace(/":/g, '": ').replace(/,"/g, ', "')

So this was the problem. Javascript removes all whitepace for the stringify command, but Python's json.dumps doesn't. To remove whitespace in Python, you have to use json.dumps(mylist, separators=(',',':')) as shown here. In my case, I needed to add the whitespace BACK IN for it to match the Python signatures used in ZeroNet.

I've dealt with similar when porting libraries in the past, but this felt lucky lol (that 2nd replace). Anyway, deciding now if I wanna stay up and try to push out an Explorer update later today. (yawn)

Thanks again for the help!

[ghost]

@shortcutme @HelloZeroNet

[purplesyringa]

@HelloZeroNet It's an important PR, please merge it or at least say why you don't.

[HelloZeroNet]

Let's try it :)

[purplesyringa]

@HelloZeroNet ZeroTalk site hasn't been updated completely: KxoID is still missing from data/users/content.json.

[HelloZeroNet]

Thanks for reporting, fixed!

[ghost]

Thanks so much for accepting the PR. One thing that should be considered is someone brought up the point that ZeroTalk currently only shows the part before the @domain unless you hover over the username - and technically two people can have the same name, one from zeroid and one from kxoid.

[HelloZeroNet]

Yeah, i just changed and it should display the non-zeroid providers