New BSIP: Add reference data to operations and database objects

[abitmore]

Example 1: user-defined ID for limit orders https://github.com/bitshares/bitshares-core/issues/556 https://hackernoon.com/the-ideal-crypto-trading-api-b1bbb2675875

Note: also need to include the ID in fill_order_operation.

Asking for deterministic account IDs: bitshares/bitshares-core#1398

[abitmore]

Another example: a merchant wants to label all its outgoing transfers for tracking/accounting purpose. Right now he need to store the data in an external database, or send a custom_operation with each transfer. It would be convenient if he can add some data in the transfer_operation itself.

Updated title of this issue to be more generic.

[abitmore]

My thoughts:

• add these optional fields into extensions of certain operations

  uint64_t op_ref_id;
  string   op_ref_data;
  uint64_t obj_ref_id;
  string   obj_ref_data;

• op_ref_id and op_ref_data will be stored in operation history (1.11.x), so a user can directly know each action she had done was for what purpose by simply looking at her account history.

• obj_ref_id and obj_ref_data will be stored in objects, mainly for the ease of system integration.

  Generally, when an operation creates an object, the obj_ref_id and obj_ref_data would be saved to the object; a follow-up operation that updates or removes the object can use the op_ref_id instead of a "traditional" object ID (e.g. 1.7.x), and update the obj_ref_data if it presents. Users can put arbitrary data in the data field. We index the ID for performance. We don't index the data.

  For example, if a limit_order_create_operation specified an obj_ref_id, that ID will be stored in the new created limit order and be indexed; if a limit_order_cancel_operation specified an obj_ref_id, it means to cancel the order with that ID.

[xeroc]

From the other issue:

So, adding a user-defined order-id would IMHO require

* a hardfork to add the user-defined id to the `create_limit_order` operation

* an index over user defined order id that maps to object ids

* an API that allows to query for the object id by using the user-defined order id

Doesn't sound too complicated to me

@dimfred, would this be a topic for you to look into?

[Dimfred]

This seems like a useful feature for users. I support @abitmore idea, adding this fields to certain operations. Maybe calling them *_user_*.

But as I understand the *_id should be unique, meaning there should always be a check whether the selected id exists or not. If it exists the user has to pick another one. Isn't that kinda inconvenient? Because of this, isn't it then the same for the user just to use the default object_id assigned by the database? Appears for me the same.

Or should the given id just be unique among the users context?

In the specific case of limit_order_cancel the id could also be not unique and we would search for limit_orders only created by this user.

And my next question would be, which operations should get this fields?

• transfer
• limitorder*
• ???

Then I could get a broader picture of that.

[clockworkgr]

this is a great feature. i can think of multiple cases where it could come in handy

[pmconrad]

IMO the id need not be unique. Certainly not globally, but also not per user. I can imagine use-cases where users would want to set the same ID for different operations. If they want them unique they have to take care themselves.

[abitmore]

My thought was the ID should be unique per user op_type or per user object_type. If uniqueness is not enforced, it would be complex to define behaviors of operations which update the objects by that ID.

My thought was, in the end we'll add the fields to every operation and every object.

isn't it then the same for the user just to use the default object_id assigned by the database

"use the default object_id assigned by the database" means the client application need to wait for the transaction to be included in a block (or even wait for it to become irreversible) to use the object_id reliably, which is not good enough for clients who need to do follow-up operations on the object. For example, a faucet may want to transfer an amount of funds to a newly registered account immediately (see https://github.com/bitshares/bitshares-core/issues/1398), or add the account to a whitelist, etc.

[pmconrad]

If uniqueness is not enforced, it would be complex to define behaviors of operations which update the objects by that ID.

Not necessarily. E. g. for limit_order_cancel it would make sense to cancel them all. Operations that make sense only with a unique ID could throw if there was more than one object.

One more thing:

string op_ref_data; string obj_ref_data;

These can be arbitrarily long. Not all operations have a per-kb fee. We should severely limit the length of these strings. (Perhaps configurable by committee?)