Base Addresses and used Wallets

Long time no see! I like your project a lot, unfortunately I'm not able to contribute to it yet. I was able though, to complete what I wanted to achieve.

1) I was wondering if the query delegationsAndRewards was part of the original design or you implemented on the fly when I reported it? My app relies on this functionality entirely, please don't remove it.

2) My app scans the blockchain using Ogmios, checking every address in every Trx and serializing it in order to obtain the base address, if the base address matches my reference base address (the wallet of interest) it saves that address in a stack. Why all this? The idea is to know how much ADA a wallet possess using as only input a BaseAddress (56 chars hex). Can this be a feasible feature for Ogmios?

I wanted to offer you both @KtorZ and @rhyslbw a beer, but I don't know if you drink, or if you don't. So in the doubt as a token of my gratitude I'm offering you a pizza instead, I sent it to addr1q8alec26en95yzvzwpxfukglsw4nx9wrx99p3m8k2drwppduzkt66uw9t5kspx5jwjecx80dz4g33htknafhdhkvzd5sc9rzynù

Thanks guys, yours, Yuca

Ps: Do you mind, or give me authorization to speak about Ogmios in a Youtube video, and to mention both of you in the video?

Hello @tapiocapool :)

I'm not able to contribute to it yet. I was able though, to complete what I wanted to achieve.

That's the best form of contribution actually! Using the project, and giving feedback on your experience. Thank you for doing it.

1. I was wondering if the query delegationsAndRewards was part of the original design or you implemented on the fly when I reported it? My app relies on this functionality entirely, please don't remove it.

Ogmios' interface is directly derived from the node-to-client interface from cardano-node. So it is the node that defines which queries are available for use. The thing with delegationAndRewards was truly an oversight from my side :sweat_smile: ... Ogmios intend to support all the queries available on the node and I somehow skipped on that one. Implementing it was fast because the machinery was already in place for all other queries. So It was mostly just plumbing and wiring. Worry not, It won't be removed.

2. My app scans the blockchain using Ogmios, checking every address in every Trx and serializing it in order to obtain the base address, if the base address matches my reference base address (the wallet of interest) it saves that address in a stack. Why all this? The idea is to know how much ADA a wallet possess using as only input a BaseAddress (56 chars hex). Can this be a feasible feature for Ogmios?

I am not quite sure to understand your question. At first you say "My app scans ... using Ogmios", which I read as something you've done and works; but then ask whether it is feasible :thinking: ? Or is your question more like:

Is there a way to know how much ADA is assigned to an address, without having to do a full scan of the blockchain?

If that's the question, then the answer is: yes! You can use the utxo query for that, which can take a list of addresses as filter.

e.g.:

{ "query": 
    { "utxo":
        [ "addr1qe32e2zpu4favd0jemfkwkn2v8wx9w7caj8467jy4a3ps4qa5m2hm"
        ]
    }  
}

will yield all the UTxO currently available at the address. If your application only needs to track the current UTxO, then that would be sufficient. A full blockchain scan is only necessary if you need to have access to the transaction history. The node doesn't keep track of past transactions in-memory, only the resulting UTxO so querying transactions is not possible.

I wanted to offer you both @KtorZ and @rhyslbw a beer, but I don't know if you drink, or if you don't. So in the doubt as a token of my gratitude I'm offering you a pizza instead, I sent it to addr1q8alec26en95yzvzwpxfukglsw4nx9wrx99p3m8k2drwppduzkt66uw9t5kspx5jwjecx80dz4g33htknafhdhkvzd5sc9rzyn

Thanks! It's been received! I do not drink beer indeed (more of the Whiskey type), but pizza is always good! I'll make sure to share that with Rhys at some point, once we meet again after the COVID apocalypse. I'd be happy to mention you on the sponsor page if you don't mind :)

Ps: Do you mind, or give me authorization to speak about Ogmios in a Youtube video, and to mention both of you in the video?

You have my blessing :pray: I'll be really glad if you speak about the project and get more people involved. Much appreciated.

Sorry for the delay, work is keeping me very busy lately. Thanks!! Yeah I wanted to document how I achieved to develop my app without having to use (or misuse) a lot of disk space. Not to diminish the CardanoDbSync project but I felt it was too bloated for what I wanted to achieve. The ledger is +/- 8GB and the Postgres was 17GB, and then I had to build the queries too. Ogmios is slim and very fast. Perfect for what I wanted. I was thinking on making a video where I talk about your project and how I used it to gather the info for my app, and how I found it interesting. I have other ideas where I might use Ogmios as well.

As I said on previous messages, I'm a member of a group of small pool operators (now 57 +/-) with very little or no delegation at all, we delegate in turn to one pool so one of us in turn can mint blocks, periodically we rotate the delegation. But all check-ups, in order to know who has delegated to the designated pool and how much is done by hand. I developed a way to help out the group administrator to keep all participants on check.

This is the link to the group's landing page https://www.f2lb.org/

Below is the link to my app. Is the draft of (how I called it) The Accuser (it was meant to be inserted as a plugin on any webpage). Is a draft, there's a lot to be done yet. http://f2lb.tapioca.link/show-case.html Don't be fooled by its simplicity, behind that page there is There's a full Cardano node + Ogmios + two NodeJS workers. I called them: George (yes, as the monkey) and The Taxman. All report to an Apache Solr core for data persistence and querying.

What I was asking first was if is it feasible for Ogmios, given a stake address check the overall amount of ADA controlled without having to scan the blockchain for addresses that match that base address.

For example, given the stake address 30686ace397847509a7a7439f7747be278954b5634328e880d833381, in that wallet there are xxxxxxADA, just like you can see in the button ViewAllAddresses here:

https://cardanoscan.io/stakekey/30686ace397847509a7a7439f7747be278954b5634328e880d833381

What I do, when someone gets added to the participation queue (that person provides a stake address, it's mandatory for participation), then I get Ogmios to scan every and each txout since the beginning of Shelley, serialize that txout address, if it matches with my reference (30686ace397847509a7a7439f7747be278954b5634328e880d833381) that means that that address belongs to that person so I store it for later use. When I gather all single addresses (addr_format) I do a QueryUtxo [ all addresses here ] in order to get the balance, just like in the View All Addresses button. I just wanted to know if there was an easier way to do this. Or maybe this is the right way.

Thanks again!! Have a good one!

Ps: Sure, you can mention me in the Supporters page if you wish!

Hey @tapiocapool , thanks for all the explanation. That's actually a quite cool project! I am glad you found Ogmios a good fit for it, that's indeed typically the kind of use-cases I thought Ogmios would be useful.

What I was asking first was if is it feasible for Ogmios, given a stake address check the overall amount of ADA controlled without having to scan the blockchain for addresses that match that base address.

Unfortunately there's no such query in the node's interface. For this you will have to scan the blockchain AND, to maintain the UTxO state yourself like a wallet would do (past transaction may be spent!).

I just wanted to know if there was an easier way to do this. Or maybe this is the right way.

Arguably, there could be an easier way but at the moment, this is the right way of doing it.

That's actually a quite cool project! I am glad you found Ogmios a good fit for it, that's indeed typically the kind of use-cases I thought Ogmios would be useful.

Thank you! Yes, I didn't see fair that in order to consult simple information to the ledger I would have to use a lot of disk space, and disk space on AWS is not cheap for example. For what I wanted to achieve this was perfect. No overhead. A docker image, easy setup, and all the abstraction made for me. Talking to a node, directly. Wonderful!

I plan to use Ogmios for another project, this one will be way different as the information displayed will have other purposes, I'll tap to Ogmios' real-time low cost capabilities and use the real-time information it can gather from the blockchain. I'll keep you guys informed.

Unfortunately there's no such query in the node's interface. For this you will have to scan the blockchain AND, to maintain the UTxO state yourself like a wallet would do (past transaction may be spent!). Arguably, there could be an easier way but at the moment, this is the right way of doing it.

No problemo at all! I just wanted to know from the veterans (Yes both of you are gurus to me), if I was just doing a dumb stuff or not.

My strategy was this, George (as I call the scraper based on Ogmios) scans the blockchain and using a list of base addresses stops (so to speak, he saves and keep going) when one of the serialized txout addresses in a block matches, saves that address into a stack, each participant has his own stack, so I know what addresses belong to what player. Periodically, a cronjob calls the Taxman (a NodeJS worker that uses the same instance of Ogmios), to query the uxto of the addresses extracted by George, sums it all up and saves it into what I call The Oracle (a Solr Instance).

Ogmios is fast and lightweight, I like that too! I'm very happy with I could achieve using it!

Thanks for answering my questions! Again, have a good day!

Okay, so you use the chain-sync protocol to figure out all the addresses, and then use the state-query protocol to get their corresponding UTxO directly from the ledger. That works too, and it's actually pretty neat because you don't need to track UTxO yourself and can rely on the ledger to do it correctly for you! Nice.

Small note about the chain-sync (I realize I have under-documented that in the user-manual): Ogmios can do pipelining (that is, process multiple requests in flights while the responses are being constructed) which speeds up a lot the overall chain-sync. In a local setup, I typically do chain-sync with ~1000 in flight messages (easily achieved by sending a burst of 1000 messages when opening the connection, and then, by simply re-sending a new message for every processed message). For example:

const client = new WebSocket("ws://localhost:1337");

client.ogmios = function wsp(methodname, args) {
    client.send(JSON.stringify({
        type: "jsonwsp/request",
        version: "1.0",
        servicename: "ogmios",
        methodname,
        args
    }));
}

client.burst = function burst(N) {
  for(let n = 0; n < N; n += 1) {
    this.ogmios("RequestNext");
  }
}

client.once('open', () => { 
  client.burst(1000); // On 'open', bootstrap the sync and put many messages in flight.
});

client.on('message', function(msg) {
  // do something
  client.ogmios("RequestNext"); // Then, simply request a new message for any processed message. 
});

Hello @KtorZ ! I've noticed by trial and error that Ogmios had the capacity of not locking its websocket operations. I mean that while one process was scanning the blockchain in order to find addresses that had a specific base address (Hash16), I could spin another process for querying the uxto or delegation. But you gave me a wonderful idea that I didn't though before. The only problem I see is that I would need to know in advance some starting points.

For example: Before I start fetching blocks I would need to know a slot number and its hash of 1 specific point. My app start at { points: [{ slot: 2901397, hash: 'e46f103a7d385243145b75135aa44691bb6a7cdddaeb15d24bcfb7ea0aa58081'}] }

From here to the first of Shelley is a long way up, I could do something like: Process 1 -> Go and Start from Byron! Process 2 -> Go and start from Shelley! Process 3 -> Go and start from Allegra! So on and so forth.

This way would definitely speed things a lot. But I would always need to do a FindIntersect before fetching blocks and in order to launch a swarm of process that help me speed up, so I would need to map the blockchain, not during sync but after I get to the tip and Ogmios starts to see the blocks coming real time.

That map could serve to spin the workers with blockchain starting points so I can re-index quickly whenever a new player comes along.

I'm curious to read about your thoughts on this.

@tapiocapool Best to make a new issue since it's a new topic. Also it would be great to have you evaluate the new TypeScript client as there are some generic patterns we can adopt to handle performance improvements internally. For example a full blockchain scan with pipelining brings a full sync down to around 15 minutes currently, and this is something we can abstract in the client https://github.com/KtorZ/cardano-ogmios/issues/30

CardanoSolutions / ogmios

Base Addresses and used Wallets #27