Now the op_return limitation has been removed .

[carpemer]

The SV community is hoping to use OP_RETURN as the most important OP to move all the valuable data into the block.

Now, most of the major miner has increased the restriction. Most of the major wallet service provider also increased the number as well.

We believe at least 99k is a reasonable value for now.

https://github.com/moneybutton/bsv/blob/50ca3fa2032478895f222cd6cc235df5b9387728/test/script/script.js#L338-L349

[AustEcon]

I'll see what I can do.

[AustEcon]

I need to find an API that can accept the txs... forming 100kb op_returns is fine and I figured that out ages ago but I haven't commited it because the APIs on the network reject it...

Anyone know of a REST API?

[carpemer]

You mean bitindex.network send api does not support it, big op_return payload?

[AustEcon]

If you go here: https://github.com/AustEcon/bitsv/blob/master/bitsv/network/services.py

You can see that I am currently using BitIndex as the preferred method of broadcasting... So I'm not sure why it's not working - I just figured that it's not supported yet. Anything over 220 bytes gets rejected.

[AustEcon]

Here's a basic script doing the broadcast manually if you want to play around with it... There are 55 x "hello" which gets rejected but if I do 44 it works. (bearing in mind that my development version has removed the 220 byte limit and replaced it with 100kb)

import bitsv
import json
import requests

my_key = bitsv.Key('PUT YOUR OWN PRIVATE KEY IN HERE TO TEST')
my_key.get_balance()

def broadcast_rawtx(rawtx):
    """broadcasts a create_rawtx as hex to network via BitIndex api"""
    json_payload = json.dumps({"hex": rawtx})

    headers = {
        'Content-Type': 'application/json',
        'Accept': 'application/json'
    }
    r = requests.post('https://api.bitindex.network/api/v2/tx/send', data=json_payload, headers=headers)
    return r.json()

rawtx = my_key.create_op_return_rawtx([('hello'*55, "utf-8")])

broadcast_rawtx(rawtx)

[carpemer]

Strang, It supposes to support that. I had sent a mail to the author for this, what is your email address? I could loop you in the thread.

[AustEcon]

Okay thanks. cyborg.napster.0922@gmail.com

I must be doing something wrong with forming the transaction then...

[ghost]

Hello @AustEcon , I'm the BitIndex developer.

I just performed a test of a large OP_RETURN using the datapay composer here: https://unwriter.github.io/datapay/example/composer

It uses BitIndex underneath and the transaction was successfully broadcast here: https://api.bitindex.network/api/v2/tx/6c0caf0b31887b4aae85db20e9569ce8bf6fce3df44a6af31477ccc1c2c4e571

Can you share what the example rawtx hex payload is? Then we can troubleshoot it.

I'm suspecting the encoding of my_key.create_op_return_rawtx([('hello'*55, "utf-8")]) is incorrect.

Thanks,

[ghost]

I found this error returned by the BitIndex node:

{
    "message": {
        "code": 1,
        "message": "the transaction was rejected by network rules.\n\nExpected type string, got null\n[None]"
    },
    "errors": []
}

I'm investigating now and I'll report back shortly

[ghost]

I found the problem.

You must set the content-type: application/json HTTP header.

That solves it:

Sample request:

POST https://api.bitindex.network/api/v2/tx/send 
Content-Type: application/json

{
  "rawtx": "0100000001db9ce916c1793e221bc3e0d1f9386a2a425de1f107bc9b948b1b4b57dd6134d2000000006b483045022100ddc7039ff4f03a59b238d5d22479334a0647684ffddf1c17c674e29a371b36b30220320a4ee9b60a6a806a0219c0eff89d5826b75e37a655b3c6aab504bae6d2822041210298c007d3d44d5cab19ae1f74192829f1ec4a5c8dcf57a57827d60265893897defeffffff02c8320000000000001976a914c29e6da4dd4825306149fc0e0a73f0b8265bf76788ace84f0400000000001976a9145b710db5bf74bd19952cd2486a2c3c195524c36b88aca4c50800" 
}

Response:

{
    "txid": "09cea59654d4605389427bb2968e7da4079482c0ba1889b820745dbe08ee98ec"
}

[ghost]

@carpemer @AustEcon Please see above for the solution. You must set Content-Type: application/json

[ghost]

Actually, I created a test script and I get an error:

Scripttest.py:

import bitsv
from bitsv import Key
my_key = Key('my key here')
bal = my_key.get_balance()

lst_of_pushdata = [('6d01','hex'), ('This is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo postThis is a very long memo post', 'utf-8')]
c = my_key.send_op_return(lst_of_pushdata)

print("out")
print(c)

Response (I've added some extra logging'):

$ python3 broadtest.py 
raw headers
{'Content-Type': 'application/json', 'Accept': 'application/json'}
out
{'message': {'code': 1, 'message': 'the transaction was rejected by network rules.\n\nTX decode failed\n[010000000139d6839363d17b8fbdef94365c58b59f0f7db1e410eae2b53e7595b7d3480683000000008a473044022058078608fbfbdb75559db2348bad9e09b4b851f53d18f88ba7626ada07f4fbf202205463a9b0878ae01b8cc38e57a98245fe4248e3555b3f181ed48a9243e35d13104141043cf0a503fd150ad112de4503f7dd17dcdba99e41cd7f8b52315fa1a4f9e499b9493fddcc15a594022f9734b8cf12a068d51328664192f351c3b618e52ae1f85fffffffff028dc40700000000001976a9149467df677dc153a88243465d09ca5fe8f7ba8cf988ac0000000000000000390a6a026d014d320a5468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f73745468697320697320612076657279206c6f6e67206d656d6f20706f737400000000]'}, 'errors': []}

[ghost]

It's saying that the transaction was rejected by network rules.\n\nTX decode failed\n[

Something is not right about how it's being constructed

[ghost]

After investigating...I'm able to broadcast large tx's directly with bitindex composed with datapay and also manually.

The problem is going to be in this function: https://github.com/AustEcon/bitsv/blob/master/bitsv/op_return.py#L11

Something is not correct about the encoding of the transaction when it gets more bytes. Please let me know if I can help with the BitIndex troubleshooting side.

Right now I'm using https://unwriter.github.io/datapay/example/composer and able to copy-paste the right panel hex encoded transaction into POST https://api.bitindex.network/api/v2/tx/send and it successfully works.

[AustEcon]

Thanks for the help so far... just got back home. This is my error message for OP_RETURN followed by OP_PUSHDATA2 then 0x0113 and then "Hello" x 55:

{'message': {'code': 1, 'message': 'the transaction was rejected by network rules.\n\nTX decode failed\n[0100000001eb1aa94e2582bc76861bdd1f956fea17902d91b742edd4e76d776b7ba2dda342000000006b483045022100fecfa355dff698c83ba1fa648084c891498756608785ab7ac2e591b02356217f022053badc7d05eb94ea5d3a96e18fd78952bc37e5eda4c24462979bf53ae96f4f7441210340458ebe64df0c5f11c490440700d5e1c2b52e8bd4f3cd296a7950bf898a65bcffffffff023e6d9800000000001976a914449e033884e6c9c5f10adb04a5a796b5cffa122988ac000000000000000017016a4d130148656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f48656c6c6f00000000]'}

I too had come to the conclusion that I must be doing something wrong with forming the transaction when I step it up into OP_PUSHDATA2 range ?? But it looks right to me.

This is an OP_RETURN (6a) followed by 4d1301 which is OP_PUSHDATA2 and then 0x0113 which is hex for 275 bytes (hello x 55 = 275)... so I don't know where the problem is at this stage.

[ghost]

Just an idea.... try generating the same packet with datapay composer and then inspect the hex to see where they differ. It may give a clue as to the difference

[AustEcon]

Yep... That's what I've been doing... I can get the 275 byte op_return tx to broadcast using bitsv (via BitIndex just fine) so the API is working fine... I just have to find the bug! I think the op_return.py is okay... or at least the OP_PUSHDATA2 etc. seem to match... very confused as to why it won't work... but I MUST BE constructing the tx wrong. I have proved it to myself...

[AustEcon]

I've spotted something that I can try... I noticed that there was an "fd" prior to the two bytes indicating the length of OP_RETURN... (i.e. this is a "var int" and is expressed as \fd + two hex bytes)...

My version does not have the \fd preceeding byte - I actually flagged this to myself a while back but I thought that maybe the original creator of this repo @ofek had some knowledge in this area that I didn't have access to. (There's a mention of "CVarInt" in the bitcoin wiki and I thought maybe it was something like that... so I left it alone because it wasn't causing any issues)...

[AustEcon]

Okay that has worked (changing everything over to VarInt)!!! :)

... but my electrum SV wallet and block explorers seem to not show the transaction until after 1 confirmation... is this normal behavior for large OP_RETURNS?

[carpemer]

My understanding is the block explorer needs the tx to be in the block to build the index. You could use babel to query the specific tx in memory and block. https://babel.bitdb.network/query/1DHDifPvtPgKFPZMRSxmVHhiPvFmxZwbfh/ewogICJ2IjogMywKICAicSI6IHsKICAgICJmaW5kIjogeyJ0eC5oIjogIjE3ZGMzZDg4YjY3ZjQ3NjAxZjQ0ZDQ2ZjBkZGNmOGIyNTFiYzhmM2I0YmQwODA0MTk1NjY1NDA5ZTcwYjBiYTAifSwKICAgICJsaW1pdCI6IDEwCiAgfQp9

[AustEcon]

Thanks @carpemer for bringing attention to this issue and @atteeela for the help.

@carpemer - if you are able to help me with testing this latest repo version, I can look at maybe doing a new release 0.5.4 tomorrow.

[AustEcon]

Oh wow!! I just checked out Babel... that works! Problem solved! :) Thanks

[carpemer]

My blocked tx has been sent to the network, as well, Thanks, @AustEcon ! https://babel.bitdb.network/query/1DHDifPvtPgKFPZMRSxmVHhiPvFmxZwbfh/ewogICJ2IjogMywKICAicSI6IHsKICAgICJmaW5kIjogeyJ0eC5oIjogIjczZTc1YWI5YWViNjYwZTAwM2VkYzRlMjIyYThiYjZmYmRkMmZiMjY3OWE3NzFlOGVjYjE0MWRiODg4MTc0ZTgifSwKICAgICJsaW1pdCI6IDEwCiAgfQp9

[ghost]

Nice work! @AustEcon @carpemer (ps, I just noticed I'm replying from my 2 different github accounts atteeela and this one)

BTW, the UTXO's in BitIndex are up to date. It is always good practice to get UTXO's and send from the same node so there are no race-conditions (some other public nodes sync slower/faster) and it can lead to errors.

ALSO, the reason it did not show up in the block explorer is that many SV nodes are not propagating mempool large OP_RETURNs between BitIndex and whatsonchain (and vice versa)

However, after 1 confirmation the network is flushed and all services will be in sync. As time goes on and everyone is runing Bitcoin SV 1.1.0+ . then this will not be a problem.

[AustEcon]

That makes a lot of sense! That was my intuition about what was likely causing random issues from time to time...

Thanks @attilaaf this will be the next thing to fix. Hopefully can get it purring like a kitten.