Database allowing storing big binary data for Wei counts + supporting machinary for computations in Wei

2 Jan, 2022 - this card is half done by @bertllll with regards to changing data type to a blob object

This is an interim card to MASQ-Project/Node#382

payment_suggested_after_sec payment_grace_before_ban_sec permanent_debt_allowed_gwei balance_to_decrease_from_gwei balance_decreases_for_sec unban_when_balance_below_gwei (accountant/mod.rs)

routing_byte_rate routing_service_rate exit_byte_rate exit_service_rate (neighborhood.rs)

Lately (Bert): in order to support calculations above blob objects we need to adjust a lot of old code.

Since I started this task I've been trying to unify some queries which in my opinion should be more reused over the place of Accountant, reducing the level of inconsistency. I'm going to intensively gain from repeated use of these new core tools that enable inserting and updating values in the DB; with that I should be able to manage, on-body tailored, all kinds of queries of this sort, using the same code structure by which I can avoid writing and testing a similar functionality multiple times.

Also DB changes, with different types to store in, and a DB migration will be needed.

Another related thing lies in the design of financials command. Go there and change all that needs to be changed regarding a broader range of decimals. Use large data types in place of the old ones. Check the code for the UI as well as the backend in Accountant.

@dnwiebe @bertllll

So in researching and theorizing on estimations, I have come to a set of rate values which we can use as defaults for testing and potentially GEMINI alpha/beta releases.

What I need to find out as well, is how the scan timing values operate and interact, before we attempt at changing those values

For now here are the set of RatePack Values to be considered to hardcode @bertllll : (I know they seem like very small values, but I've done calculations basing the use across 200MB for a typical HTTP/S content)

routing_byte_rate = 3.5333e-9 routing_service_rate = 3.5333e-7 exit_byte_rate = 1.7667e-8 exit_service_rate = 1.7667e-6

It appears from a scan through the code that the units of RatePack are in gigawei, not wei.

This is a problem: one gigawei is not a small enough amount to charge for a byte. Part of this card will be applying conversions so that we can speak gigawei on the blockchain, but wei in the code. There may be a requirement for data types bigger than u64...and some of those data types may need to be signed. Probably the widest data type will need to be in the database, unless we're not concerned about losing track of amounts smaller than a gigawei. SQLite's widest numeric type is i64, which means storing a wider number will mean either converting to and from strings, or storing the number in a BLOB column. We could also divide it into pieces and store it in several 64-bit numeric columns, but I like a single BLOB column better.

It appears from a scan through the code that the units of RatePack are in gigawei, not wei.

This is a problem: one gigawei is not a small enough amount to charge for a byte. Part of this card will be applying conversions so that we can speak gigawei on the blockchain, but wei in the code. There may be a requirement for data types bigger than u64...and some of those data types may need to be signed. Probably the widest data type will need to be in the database, unless we're not concerned about losing track of amounts smaller than a gigawei. SQLite's widest numeric type is i64, which means storing a wider number will mean either converting to and from strings, or storing the number in a BLOB column. We could also divide it into pieces and store it in several 64-bit numeric columns, but I like a single BLOB column better.

Hi @dnwiebe - yes we did expect this and I had conducted a number of calculations to come up with a realistic default value instead of the current ones in the RatePack (which are 100,1000,101,1001 etc)

I converted and used all correct units (MASQ token in gwei) and came up with some values I think would be reasonable to do live tests with, although they are small numbers.

routing_byte_rate = 3.5333e-9 for 1 byte routing_service_rate= 3.5333e-7 for 1 byte exit_byte_rate = 1.7667e-8 for 1 byte exit_service_rate= 1.7667e-6 for 1 byte

I don't think at this time we need to be concerned about the units being in wei, unless its not easy to convert within the code itself for signing the data types

What number width would be the biggest concern here? Are these decimal numbers too small in GWEI to be used?

The numbers in the RatePack are integers, not floating-point. That means the smallest rate that can go in a RatePack is 0, and the next smallest amount is 1 gigawei.

What we need to do is modify the RatePack to be denominated in wei, not gigawei, As part of that operation, we may also want to modify the database tables to be denominated in wei instead of gigawei.

I have no instinctive feel for it: is it reasonable to round payables and receivables to the nearest gigawei, or do we want 1-wei resolution for that?

@dnwiebe - Ah ok that makes a lot more sense to me now, with what you were describing. I think given the scale and granular approach to routing costs relating to bytes themselves, its probably best to go for 1-wei resolution.

My early estimates show that we are actually nearing to a below 1-wei cost per byte in terms of MASQ with the way real-world values appear.

I think we would need to find an even wider range too to track this beyond 1-wei, if possible.

Only thing is, we are constrained on the token side by only 18 decimals. That may not be an issue if the financials can be accurately recorded and them summed together once it passes a certain minimum payable threshold anyways, which would trigger an alerted debt payable by accountant in general.

I think we should discuss this tonight in a design meeting

Hi @bertllll

Here are the general thoughts on the structure of the financials message, which can be used for dynamic aspects of this command

It would be useful to have options within the usable financial command, which can outline several aspects

It would be fine to explore this a bit more in discussion if needed, to refine the options below:

Default outputs Starting with totals would be preferred, as the current response is delivered now. Additionally, it would be preferred that the default response to a financials message would also contain a payload with the largest 20 payables and largest 20 receivables in order of largest to smallest based on size
Specific options Previously, the financials message had optional parameters that would allow a min/max age of debt/receivable and min/max amount in gwei (soon to be wei) That would be desireable to put back in.

We also need to be mindful on the size of the response payload so maybe we need to also limit to to a certain number of results

Test plan:

How to characterize what in the end filled this card? Not easy to me.

I can start with the most obvious, that is, all old money computations switched from Gwei units to Wei. You can see very granular balances on the accounts of your debts to the others or of the others to you. (A technical note: the transition was allowed only by engaging a more complex way of storing data in the database. The SQLITE language has integers as data types but only up to i64 which means 64 bits, saying how big number maximally we can store in it. Wei counts are much bigger though, it requires suddenly multiplication with 10^9, a bigger number used to give the same number in these other units. The possibility we chose was to reach another data type offered by SQLITE, which is blob. It is kind of an unspecific binary object that can store any kind of information, you just have to know what kind of it it is. Rust's SQLITE library, which we use, rusqlite, gives one more option, very handy for us; they automatically convert Rust's native big signed integer type i128 to a blob and it also can convert it back to the number, without any big fight, well, only as long as you don't want to do in-database numeric calculation. Sadly, we cannot do without it and so I had to extract a certain portion of computation out of the database responsibility and compute that by our code and then give the pre-processed data back to the database as parameters for an upcoming query or queries. It's not like it just sounds cumbersome it actually is, however this way is still less cumbersome than other options.)

as a side product of work on the previous, I had to deal with these things too: I revised the are of payment thresholds, especially the equation that makes the sloped part of it (if only somebody knows what I'm talking about; if not, that thing is a formula making the threshold to start paying or banning descend with time). This equation did not seem right to me, so I made a correction that could possibly be sensed even from outside but I doubt that you could compare it with any old state easily and also, under the conditions of various imperfections in the whole accounting machinery, it may stay undetectable for testing. Anyway, I also made sure that from then we compute it the same way at all various places and not always again by using a new definition, with new code for it.
another part was painful but necessary. Because of the blob thing, instead of a valid integer, it's become almost impossible for people to read the values out of the database, because there are no numbers anymore. This would affect both the testers and maybe also the future users. The way out was in enhancement of the 'financials' command that now can also provide a view on records enlisted in the payable and receivable tables. There are three new features regarding that.

Disclaimer: this is intended purely for CLI users.

1) you can simply order retrieve me back first N top records, the command looks like financials --top 45. It will automatically pick from both tables.

2) you also can take a detailed query using either --payable or --receivable or both, as the first part of this separate variation. The ending part are two mandatory arguments which signify a) min-age/max-age (e.g. 5600/25000) b) min-balance/max-balance

3) The default of the financials command (without any additional argument) is statistics with paid and unpaid money, to be received and to be paid, as it was also before this card. However, it now allows you to write --no-stats and if you do that together with another argument it will leave the statistics out. If you use it alone it's going to be an error because there wouldn't be anything left which the response could bring back, therefore this option makes zero sense.

one of the last things I have to mention is that I had to write a database migration for the obvious reason of changes in the database definitions. You can try to catch the migration in flight in order to check it's true.
and finally, as I remember slightly, this huge amount of work was introduced a long ago with a dialog with Kauri where we discussed new defaults for the rate pack. The required values were way too small and we even couldn't represent them accurately with our means being available, having just Gwei units to do it, no way to drag it down to Weis. And so with the changes made recently I could fit in the goal to bring those desired values from back then or any other values that work with fragments of Gwei. And so I did. I think Kauri may fill you in further on this as he said there would be more consideration about what the defaults exactly should be like, probably decided during the testing. You can check them for example with the configuration command or if you go live with on test network you will be likely to see the impacts of lower rates clearly.

Have fun with testing

An extension after I revamped a few things:

The most serious one was that those messages commuting between the UI and the other end-point (Daemon or Node) needs to be serialized and it did not work as expected. When talking about serialization, that means that the message is transposed from its unique Rust-like structure into another form where it can be easily reproduced and handled by any other applications. It can be taken over and the receiver, because it's in a JASON format, can deserialize it and identify parameters and their values that are in that message.

Our Rust library that does this dirty job for us probably doesn't allow for serialization of really huge numbers, even bigger than the big but standard 64-bits long integer. However, as passing through integration of Wei, I also put these big u128 and i128 integers in the structs belonging inside the UI-Node messages that later need to be serialized.

The interesting thing is that I didn't know a long time it was a problem. In the sphere of my tests I unfortunately never wrote an integration test, capturing the activity, transformation, between those serielized and deserielized data, with use of big enough numbers to get over the upper limit of u64 (which is already big anyway). So this awakening had to wait up to other testing, where I was told about certain errors and later I deduced that the big numbers of Wei did not do good to the serialization library, making it confused and I think it tried to represent the number in a scientific notation with powers over tens. Parsing of that, in the end caused a panic.

So I had to back off and have the messages carry only balances of Gwei, getting rid of nine digits. It still has good information for the user when they eventually get the values printed because we discussed that they aren't going to be driven by these small values at all, mostly interested in things even more worthy than one Gwei. So that's the first change. I had to work carefully with unit conversions and I feel like I don't want to live it through again.

As an automatic reaction to the things laid previously, I knew I had to make changes in the CLI where the returned values need to be displayed to the user in a fine manner. And to be honest, I overestimated my strengths or at least the momentary amount of energy I had in disposal. So I brought in certain fancy features which I believe other developer didn't bother to think about. You can get adequate assistance from the help of the 'financials' command if needed but I will write a small summary also here.

First, I wanted to allow the user to decide whether they want to display the amounts of money as MASQ (with two decimals) or as Gwei, with the full precision which we can get from the original message. See more in the help.

Second, I did not want to make the user feed the query for specific ranges of accounts with values in Gwei. It would mean extremely long typing of a single number. So I decided that I need to change that to values in MASQs, allowing to use less digits to maybe express the same thing.

On the other hand, thinking of the most demanding users, I prepared an option that they still can use, if they feel need for it, to get down to precision recognizing one Gwei. It threw me into more and more troubles with validating and processing such a variety of options and so it drained all my energy out and the card became bigger than it deserves (maybe). I'm confessing to have a bad feeling coming to submit this work for a code review, because it's going to consume a really lot of time to bite through it to the end.

Please give it your time to try some extreme numbers. In general I cannot put perfect defense against memory overflow in the backend, what I can do is either to laboriously check on it and not to let the operation proceed or I cannot more than make a meaningful message to be thrown out during a panic. If the Node is intentionally fed with too big numbers, under certain circumstances, it's not going to handle it (during conversions especially) and an immediate panic is going to follow (because that's actually your fault that you pushed those numbers in somehow. The Node is designed to handle standard values in that can occur in time parameters and Gwei - never fewer than how many MASQ tokens will exist and so it must be done on purpose if a bigger number is to be processed). The other thing is that you should not be let send these "explosive" values via CLI. This front application should catch it and stop you, with a considerably good message.

Well, hopefully, you will find the arrangement a bit nice.

PS: I've been still keeping in my mind what I saw while I was connected to a network with multiple Nodes, the annoying every-so-often-happening log messages broadcasted to the UI that flows like a flood to the CLI engraved into my memory. I'd appreciate if you found a moment for you to write a brief message to me about your experience with this disturbance, and the main thing, whether isn't that actually an obstacle to carrying out tests effectively. If so, we might have to consider to design a way how to disable it until we mitigate the number of warn and error logs showing up frequently, because the number of them is unbearably and unnaturally high.

Thanks

Third time lucky

I did more work on features like the financials command, enhancing it fairly, but somewhat unfortunately, I also had to rework even the core thing of this card. The way how we treat big integers that cannot be normally stored in the database as numbers which need stems from the transfer to Wei counts which make everything a big number out of sudden. We've known we need to be able to read and write the values, what more, we also need to be able to perform mathematical operations over the values, and solving each was kind of a chapter on its own, from adding over subtracting (in minor also multiplication) and then comparisons a lot. What made this difficult has been our wish to keep these operations in the competence of the database manager, so that we don't have to assist him with computations on our part, within the MASQNode application, which would cause longer execution of those procedures, transferring responsibility back and forth. Because of the change of our paradigm lately, where we decided for a different technical solution, all sort of other things have changed and I don't want to go into details because it's already turning into a too long story.

Perhaps the main thing for you to know is that we've begun to store big integers (exactly 128-bit integers, big enough to swallow all MASQ tokens even if represented in Wei) as a pair of i64-bit integers. That implicated that all columns with balance needed to be doubled to newly stand as high and low bits of the big integers. We've expected from this quite complicated solution that database operations are going to be faster than like it was with my previous solution. Most of operations now still happens in the database except cases where we add or subtract such big values that in combination with the values having already been in the table would overflow at the low bits (the second column). When that happens another feature comes to play, where we, without excuses, have to compute it by ourselves and then hand it back over to the database manager (because he alone wouldn't be able to get through it). We have faith that the overflow is going to happen rather less frequently, only if big values move around, so we probably are going to safe some execution time, which is going to be true the more the further in the future we will be, with appreciating of the token that will cause billing to run down to smaller amounts of the token for pieces of serving/routing services charged for an average CORES package. We have to do quite many computations with each packet and so the impact of the single procedure on keeping times lower might be strong.

The above has also a disadvantage with it, that is, people since now won't be able to read values out of the database so easily. The specific representation in two columns makes it distorted and quite unreadable for a human. However, at least for our payable and receivable tables, one can use the financials command to get it back on with a readable manner. Even this has a flaw though, because, and I don't know to what extend, the financials command has been becoming more and more complex and, as the result, there's been a large amount of time needed to process the request...and at some situations, or all, at worse, the present limit for waiting on responses within the CLI won't be long enough and we will see it stop waiting without any response.

There is a solution for that but that'd be another card. Now I'm thinking maybe nothing holds us off increasing the limit temporarily even though it will look awkward if you meet the long waiting. For now, it would allow to have the information from the database come to us reliably. @kauri-hero

Testing should be rather wider for this card. I cannot name everything but I'd try the normal operating of the Node (if a panic or whatever weird happens). Then I'd look specifically at functions of the financials command - well - I'm afraid it might not give you anything sensible because of that previously said. The only chance to make it work if so is lower the Node's serving/consuming activity or even if you restart it and then demand a look into the database in peace.

Regarding CLI there is quite a lot to test. You should try help for the financials command, run it in various modes, try to see financials statistics, toprecords or customized query with two ranges. The top records version offers different ordering of the retrieved records, so please try that too. And at last but not least you can train to switch between displaying values in whole MASQs or in Gwei.

I wish you not to be bored with the test

Sorry for English mistakes, I improvise sometimes haha

MASQ-Project / MASQ-Node-issues

Database allowing storing big binary data for Wei counts + supporting machinary for computations in Wei #497