Proposal of core change about EventTypes related Exchanges

[bum2]

Actual exchanges structure:

The actual exchange system structure, with a common buy exchange type as example (commitments are always optional and events can refer to them or directly the transfer):

(see better diagrams at https://board.net/p/OCP-changes )

Modelling (five types):

• ExchangeType:
  • TransferType-1: give payment
  • EventType-a: Give (for events or commitments)
  • EventType-b: Receive ( '' )
  • TransferType-2: receive resource
  • EventType-a: Give (for events or commitments)
  • EventType-b: Receive ( '' )

Real life single exchange instance (11 records, in a typical exchange with one commitment and one event in each transfer):

• Exchange:
  • Transfer-1:
  • Commitment-1a:
  • Commitment-1b:
    • Event-1a
    • Event-1b
  • Transfer-2:
  • Commitment-2a:
  • Commitment-2b:
    • Event-2a
    • Event-2b

Thanks to the GA verbs and opposite relations, the action names (give, receive, etc) can be swapped depending on the context agent viewing the info, and so we don't need anymore to store double commitments and events for every transfer in a exchange.

As seen in the code, the main difference between 'a' and 'b' versions of either commitments and events is the event_type defining 'give' or 'receive' core actions, the 'from' and 'to' agents and other information is the same, and this duplicity can be avoided, reducing the number of db records a lot.

We can define the 'give' action as primordial and just derive the 'receive' naming when showing the info to the destination agent. The same goes for the 'buy' action which is the main initial motive of the exchange and the 'sell' action can be shown only when needed. No more need to configure and manage buy'n'sell and give'n'receive versions of the same exchange types.

The only exception I've found to this is the faircoin transactions events, which define a different resource in 'a' or 'b' events, to refer the faircoin accounts of sender and receiver agents. This is used to easily find events related each account resource, but this can be solved by finding the events using the fair address itself.

In the case of other resources without a fairaddress, like the shares accounts, their related events can be also derived even without a direct FK.

Other types of resources, not accounts, are transfered from agent to agent and both 'a' and 'b' events refer to the same resource.

Proposed simplification (7 records per exchange instance):

• Exchange:
  • Transfer-1:
  • Commitment-1a:
    • Event-1a
  • Transfer-2:
  • Commitment-2a:
    • Event-2a

When there are faircoin_transactions related the events, we can set just one faircoin_transaction as it is just one event, instead of the two faircoin_transactions that are now created in prod.

This simplifyed structure is now being tested in the easy-exchanges branch, an extension of the fdc-migration branch, to prove the concept in TestOcp.

The debate question is obvious: Do you agree on going to this direction? probably other parts of the ocp code needs to be changed to implement this change in all cases, but if everyone agrees on going that way we can gradually review the code to stop duplicating events and commitments one day.

We keep the possibility of many transfers of same transfer-type per single exchange, to allow various payments for the same resource like when renting or for fractional payments, and for giving resources in various parts for the same single payment.

This new approach will affect only the exchanges, so, related exchanges, only single events are required (and their 'duplicates' are ignored). The other multiple uses of the Event model using the two events for every economic fact can coexist with this 'single' events approach in the same system, as long as the other parts of OCP don't try to manage or represent 'exchanges' the old way, assuming the double event is always there.

When a payment involves transfer fees, we better represent them as another event of the same transfer, but related the agent receiving the fee, instead of relying on different quantity values for the 'a' and 'b' events... With the approach i'm proposing there will be only one event to represent each economic fact.

The other needs i can think of now about the use of the double event structure, is when importing the accounts of a company into ocp-rea (one more todo?): their double accounting lists perhaps can be imported as the 'b' events of the matching existent transfers (if present) and later try to depurate the differences between 'a' and 'b' events, to find out the best representation of reality (there were fees in a transfer missing to be represented? etc)

Do you think is possible to go step by step with this approach to other sides of OCP or it just will work for exchanges? I think the many other verbs (event_types) used in ocp, apart from 'give' and 'receive', either don't need to be flipped (just use their inverse_label when needed) or can be flipped (matching opposite action name and swap it when needed). Do you think is that true?

. . . . .

Exchanges Table:

Another key change to make exchanges more usable is the new table listing of the exchanges, structuring the exchange data much more eficiently for easily work with them (now using datatables js plugin).

This table, in its first simple version has only this 10 columns:

• State: to see the exchange state
• Date: the exchange creation date
• Action: the shortest version of the main verb defining the exchange type, also show link to a related join-request if existent.
• Agent: the other agent (from the point of view of the reading agent)
• Give: a group of columns to show the giving part of the exchange (from the point of view of the reading agent):
  • Qty: the amount to give
  • Resource: the resource type given
  • Last date: the date of the last event or commitment in this giving transfer, includes the give transfer state as a colored dot.
• Receive: a group of columns to show the receiving part of the exchange (from the point of view of the reading agent):
  • Qty: the amount to receive.
  • Resource: the resource type received
  • Last date: the date of the last event or commitment in this receiving transfer, includes the receive transfer state as a colored dot

This table covers good the usecases of the buyer or seller agents, but is not enough to show them to other agents (and show both related agents). I'm assuming here an agent-centric philosophy and getting rid of platform-centric or 'admins' point of views, at least in the work app.

The exchange details about how many transfers, commitments or events and their particular state, value and date is not shown now in the exchanges_all list (only shows the calculated state and totals for the whole transfer type or side of the exchange). The agent can open the exchange (the exchange_logging page) to see all details of the exchange parts, transfers, commitments and events.

I believe this change will not be much a dev team debate beacuse is mainly at the layout UI level, but anyway: What you think about this change?

[fosterlynn]

I made some comments at the bottom of the doc, https://board.net/p/OCP-changes. Repeating here, since there is now an issue.

Some history and explanation of the existing data structure:

• I initially wrote the exchange part of NRP to not have both a 'give' and 'receive' event for a transfer. I think 'transfer' was the event itself maybe, can't remember now.

• I re-wrote it as it is now to experiment with this data structure. One reason was that an event (currently) affects zero or one resource, and a transfer most often has 2 resources, one of the from_agent and one of the to_agent. This happens whenever there are "stock" resources which don't care what the exact instances of the things are, just that you have N of them. This is true of faircoin and all currency accounts, but also lots of other things - bins of apples, pallets of computers, etc. The other reason was to mirror the process - event structure (as a transfer - event structure) so that the value flows kept that pattern whether encountering a Process or a Transfer (R - input E - P/T - output E - R - input E - P/T - output E - R ....)

ValueFlows: Currently maps to the existing structure you see in OCP. But we continue to discuss, and it is not clear where we will end up. There are reasons to get rid of the "duplicate" events but also downsides. And if the from_agent and the to_agent are in different organizations and different namespaces (which they will be more in transfers than processes), there could be reason to have the events separated. Even in OCP, say in Sensorica, if they made a purchase from outside the network, they only recorded the 'receive' event, and if they made a sale, they only recorded the 'give' event, as the only things in their context. There would be (theoretically) a matching 'give' or 'receive' in someone else's namespace, as their record of the same transfer, with their resource id, which is not really the business of the other context.

In terms of effect on other parts of the system, there will be some, and it will require complete regression testing. If the test suite runs without errors, that is a good start. But I'm pretty sure there will be several things to fix. One example is the ability to trace backwards to see the history of a resource. Unless we don't plan on using other parts of the system much longer, then that could make it easier to test and fix. But as far as I know, we need it at least for some months.

Also, I don't see how you can do it without adding that extra resource to the one event. Then you have different attribute naming for resources on an event, which is pretty much everywhere in the app. As well as requiring a conversion script. But maybe I am missing something?

My general opinion: I'm not sure of the best way to structure transfers. It is possible this suggested way is better, although I think it would take a lot of analysis and experimenting to make sure of that, judging from the ongoing discussions in ValueFlows. But I also don't think it is worth changing it. There are lots of real priorities that make a difference to users. This doesn't make any difference to users, as far as I can tell. Right now, you can ask the Transfer for any information, and it will find it in its 'give' and/or 'receive' event(s) and return it, so it shouldn't be difficult for developers. You also for sure don't need to show the more complex structure on the UI if you don't want to. In addition, I think we should be looking towards retiring OCP, and making this kind of change doesn't really seem to make sense, as it will use a lot of developer and tester resources that could be put to use on something needed. If we are not going to retire it, perhaps bringing it up to a recent Python version would be a better use of time?

[bum2]

Ok, that was just a proposal to start this debate... i've added (uncommented) the duplicated events, commitments (and faircoin_transactions when using internal faircoins) in the generic share payments system (joinrequest.update_payment_status(status)), so the exchanges produced now have again all the expected items.

I was trying to simplify the structure a bit in order to reduce somehow the number of needed records to represent exchanges (from 13 to 8), because its a key structural part we are just using a bit for FreedomCoop and maybe others but still not used yet in BotC. I think is good to introduce exchanges to BotC, to represent the payments and the shares transfers (like in FreedomCoop), and the CoopShares and other projects that can be using OCP soon will need exchanges too.

The main conceptual difference i think it comes from the assumption of having separate contexts, like with single 'receive' events when the context is outside OCP, which is a certain situation that of course is maintained in the proposal, but I'm working toward a common-data, common REA platform where the agents are mostly identified, even if not participants in OCP. When someone creates a transfer from unknown agent origin, the system allows to search or define a new agent to represent the provider, so step by step we will have less single-sided events. When a provider company comes into OCP maybe his agent is already created by someone, with an existent economic history, and what they do is an OCP user to manage that agent, or set their individual agents as managers of that existent 'company' agent. The same goes for unknown destinataries or customers, the system should make easy to identify them as known or semi-known agents.

Now the duplicates are created again in the easy-exchanges branch, but the code i'm doing can work only with single events, commitments and faircoin_transactions. The duplicates are ignored (only manages 'give' events). If we keep developing with that in mind, maybe one day we can stop using this duplicates and so stop producing them. I'll like my part of the code to be ready for this...

About the double 'resource' FK you mentioned, let's talk about this: the different resource referenced in 'a' and 'b' events is used in the faircoin transactions, to reference the origin and destination fair-account resources, but as said this is solved by searching the uses of the fairaddress itself (as origin or destination). All cases related accounts behave similar, and their events can be retrieved even without a direct FK. Other usecases that i can think of don't need this differentiation in the event.resource field, but please show me examples of scenarios where a double FK with the resource model can be needed if we use single events. Or any case where the difference between 'a' and 'b' events is not only the event_type, and they are not accounts. I want to see if the proposed simplification have chances or not, but this can be a longer debate and so let's continue 'like always' meanwhile and investigate more the subject.

About retiring the actual code, I'm very interested in building a new distributed backend and one day get rid of this postgres and later (or together) also get rid of this django (because data and functionalities are migrated). The actual release now in prod can be called 0.9.8 and when easy-exchanges (fdc-migration) is in master it should be 0.9.9... The 1.0 version I think will come when all the custom freedomcoop code is deleted, so a first somewhat 'generic' version with much less lines of code is achieved. That 1.0 milestone is the time when i really want someone else to take care of it and i want to focus completely in the refactoring as a distributed system, integrating all the learnings and the best generic approaches we can, but with better technologies.

[fosterlynn]

Ok, that was just a proposal to start this debate... i've added (uncommented) the duplicated events, commitments (and faircoin_transactions when using internal faircoins) in the generic share payments system (joinrequest.update_payment_status(status)), so the exchanges produced now have again all the expected items.

Thanks!

The main conceptual difference i think it comes from the assumption of having separate contexts, like with single 'receive' events when the context is outside OCP, which is a certain situation that of course is maintained in the proposal, but I'm working toward a common-data, common REA platform where the agents are mostly identified, even if not participants in OCP.

Small clarification: In OCP, the "external" agents themselves can/should be identified the instance, with relationships to define how they fit into the internal contexts (trading partners, suppliers, etc). But the matching event is not there.

But I totally agree we want to move beyond the "platform" paradigm.

Another possibly interesting note from VF related to "duplicate" events on a transfer. In VF we introduced the concept of being able to exchange using any event type (action in VF). So for example you can exchange your work for faircoin or food or other person's work or use of their equipment or whatever. So it becomes conceptually less about the resources and more about the events in terms of exchanges. One implication of that is that say for work events, you in fact don't have the duplicate events, and need another way for different agents in an agent-centric distributed model to keep copies of the event, verify its accuracy, etc. So that part of the rationale for having 2 events on transfers becomes not important.

please show me examples of scenarios where a double FK with the resource model can be needed if we use single events. Or any case where the difference between 'a' and 'b' events is not only the event_type, and they are not accounts.

Any resources treated as "stock" resources will follow the same pattern as currency accounts. That is almost all of them, but specifically substitutable resources where you don't care which exact things you have, you just care how many of that kind you have, the individuals aren't important and aren't represented that way in the model. So resources like documents that are all different from each other are not treated as stock resources. But if you have a bin of apples and a pallet of bags of flour for making apple pie in your bakery, you only keep track of how many (or weight) of apples, and how many bags of flour. If an orchard transfers some of their bin of apples that were harvested, they have an identified resource (their bin of apples) and you have your identified different resource (your bin of apples). Say 200kg of apples are transferred from the orchard to your bakery. Their separately identified resource is decremented 200kg and your separately identified resource is incremented 200kg. Nobody cares which exact apples were transferred in terms of the model, and they are not identified themselves. It works just like currency.

[bum2]

I understand the difference between ResourceType and Resource since long ago, and when a resource is treated as 'stock' (its resource_type defines it as substituable)...

The exchanges become more easy if we just track the resource_type being transfered in the transfers, without identifying a particular 'resource', but when the resource is a unique piece (with serial number, etc) is perfect that the transfers refer to this same resource (both in 'a' and 'b' events).

The weird part is when we transfer a resource_type (substituable) but we want to track the 'containers' of origin and destination as unique and different 'resources'. That 'resource_type containers' treated as unique resources can make sense for 'accounts' (money or shares), but as said it can be avoided and use other means of finding the account related transfers.

The use in transfers of this 'resource_type' + different 'resource' in 'a' or 'b' events, when refering other resource types (not accounts), starts to became weird for the domestic scale: If the agents are industrials can be ok to reference the exact container from where the resources are transfered (say the blue container number x) or the container where we store an incoming resource_type in a transfer, but it can make no sense to define a bag, box or container as a unique resource, treated as an account (say for apples), specially if the agent has just one container.

To know the exact containers at each transfer event is ok but not essential imho, at least in the initial 'exchange' data. The info about where each agent stores the resource_type received or sended is useful for each agent's internal organization, but don't seem necessary to track that info in the transfers of an exchange with another agent (at least when creating them).

I think my proposal of single events can only work if we don't use the event.resource field except when the transfered resource is a non-substituable unique piece (in such transfers the 'resource' is the same in 'a' and 'b' events). The case for accounts, with substituable resources, can work also without using that field.

Imagine for one moment the exchanges referencing only the resource_type being transfered (and not the exact resource if its not the same). If one agent has more than one container for the same resource_type (say two faircoin accounts or two apple containers) and a transfer is received, the system should ask the agent in which of the containers she/he wants to store the received resource_type. Or when sending resources and the agent has various containers of them, the system let him choose from which container (or account) the resources are taken to make the transfer. Normally its a decision of each agent, and to make a new transfer one agent don't needs to know about the other agents containers, they just send a resource_type to the other agent and that's all.

• When we transfer a unique resource (say with serial number or an art piece), the resource field is used to refer the same resource in 'a' and 'b' events, but the resource_type field is not needed (because we can get the resource.resource_type).

• When we transfer substituable resources we can use only the resource_type field and not use the resource field of the events. A useful reference to the origin container 'resource' can be defined by the sender for the 'give' event if she/he has more than one container of same type (if not it is not necessary). Similarly, the receiver agent can define for the 'receive' event the exact container where she/he will store the received resources, but again only makes sense if this agent has more than one container for that type of resource.

I think we can let the use of the 'resource' field in the events only for the cases where its a unique resource, and for the cases where agent A or agent B has more than one container or account for that resource_type, and let all other events work simply with the 'resource_type' field defined.

The debate is open... but let's try to simplify the exchanges as much as we can!

[fosterlynn]

Thanks @bum2 this is an interesting discussion, especially since I am thinking again about Transfers in ValueFlows, triggered by your current efforts as well as other projects that need them.

Since OCP is a platform that supports both the "independent view" between agents, and the operational needs of each agent, we do need to consider both of those in the one database we have.

I think the identification of resources is up to the agent who has custody of them or responsibility for them (or ownership if you prefer). Agents may wish to deal only with the resource type, deriving current quantity from the events. Or, as in some of the situations you described, more than one more than one resource of the same type for example, they may need the resource itself. And some resources, like faircoin accounts, have an identifier that needs to be stored somewhere.

The weird part is when we transfer a resource_type (substituable) but we want to track the 'containers' of origin and destination as unique and different 'resources'. That 'resource_type containers' treated as unique resources can make sense for 'accounts' (money or shares), but as said it can be avoided and use other means of finding the account related transfers.

I have come to understand that in REA, we shouldn't think of resources very much at all as the material thing irrespective of agents. Even for serialized resources. We need to carry the serial number or lot number or document name or whatever on the resource for other kinds of identification. But the unique identification of a resource in REA is more about the resource as controlled by an agent. Even thinking technically, most agents have their own namespace, and also their own historical ways of identifying things. So when an actual touchable resource (or say rights and responsibilities for that resource) passes from one agent to another, it almost always gains a new unique identifier, so becomes a new instance, and a new REA resource, even though in the real world it is the same thing - whether managed as stock or not as far as I can tell. The real worldness can be tracked in OCP by input-process-output flows and by the tracking identifier (serial number or lot or unique name etc).

I think we can let the use of the 'resource' field in the events only for the cases where its a unique resource, and for the cases where agent A or agent B has more than one container or account for that resource_type, and let all other events work simply with the 'resource_type' field defined.

If we keep the same transfer structure (give - Transfer - receive), then what you say makes sense - Transfer doesn't need to know about the specific agent identification of the resource, resource type may be plenty in many cases. And the resource information itself may be only the business of each agent.

But in terms of simplification, if there are some cases where someone needs to actually have some identifying information about the resource (stock or not), isn't it simpler to just leave the structure the same for everybody, and let people choose whether to include the resource or not in their data?

Another piece to consider in terms of simplification is that transfers need to in some way be able to be part of value flows that also include processes so that complete tracking and tracing can be done. This is how we ended up with all that information duplicated in the give and receive events, rather than just putting most of it (everything except the resource maybe) into the transfer. We already have economic events with those data fields - is it simpler to keep those more consistent or simpler to more normalize transfers to something simpler for transfers themselves? I don't know the answer. There are several trade-offs.

Here's some earlier write-up on resource identification, done mostly by @bhaugen . https://valueflo.ws/introduction/resources.html , scroll down. This probably needs updating too, but does contain experience from the manufacturing world.

Anyhow, this is to continue the discussion, not to give a firm opinion.

[fosterlynn]

@bum2 More discussion in ValueFlows in this issue, starting with this comment. https://github.com/valueflows/valueflows/issues/270#issuecomment-472456382