search

MarshalX / atproto

The AT Protocol (🦋 Bluesky) SDK for Python 🐍
https://atproto.blue
MIT License
326 stars 33 forks source link

decode_dag_multi does not decode fully #274

Closed azulika closed 9 months ago

azulika commented 9 months ago

Maybe related: https://github.com/MarshalX/atproto/issues/53

data = bytearray(b'\xa2atg#commitbop\x01\xaacops\x81\xa3ccid\xd8*X%\x00\x01q\x12 \xc5\xa5\xcc\xba\x86\xb2\xf5\x0cS\x02\xf6\xf21\xe74.\x94\xaa)\x8b\x99a\xf1uf7\x19[}SH\x83dpathx app.bsky.feed.like/3jx22vyioas27factionfcreatecseq\x1a\x01\xe2\x998dprev\xd8*X%\x00\x01q\x12 \xc9\x10g\x9c0\xa0v\x8d\xf3\xeau\x17\xb7%\xec\xe7lP52\xbc-4\xde\xaeoe\x0f}\x80\xbc+drepox did:plc:idifviortq7ein242dx32dizdtimex\x182023-05-31T16:57:43.352Zeblobs\x80fblocksY\x13@:\xa2eroots\x81\xd8*X%\x00\x01q\x12 &/\xb8\xd7\x01\xbc@4\xabq;\x00R\xa6\x95)r\x14C%\x1a=\x9el\x7f\x11\xe7U\'\r\x9e,gversion\x01\xf8\x01\x01q\x12 \xc5\xa5\xcc\xba\x86\xb2\xf5\x0cS\x02\xf6\xf21\xe74.\x94\xaa)\x8b\x99a\xf1uf7\x19[}SH\x83\xa3e$typerapp.bsky.feed.likegsubject\xa2ccidx;bafyreickzoel2fiybtmqszwxtmd4fclxawlprlibphnqrpsbz3p5kaxilmcurixFat://did:plc:y5ziidtxv6yvci7b7tvei3ea/app.bsky.feed.post/3jx22vqs4m225icreatedAtx\x182023-05-31T16:57:43.003Z\xd1\x01\x01q\x12 \xad\x89\x91b\x89i\x8c\xe4m0\xeb\xe7\xb9\x99\x8b\xcc\xf1I\xdd,\xb0\xafV\x88\xf7\x96\x90\xfe\xb8\xce\x90\xdf\xa2ae\x81\xa4akX app.bsky.feed.like/3jv67nmeeuu2jap\x00at\xd8*X%\x00\x01q\x12 \xaf\x1d\xd8`\x95\x1d\xe891\xa7\x8e\xf1\xc92\xd6P&\xe5X(\\r\xd9\x1a\xb3\x0f18\x86\xff.\xf3av\xd8*X%\x00\x01q\x12 U\xeb<\x1aP\xa0\xc0\x80&\x91\x82\xe8\x9d\x10\x00\xddxk\x08\xe1\xb6\xf8J\x0cm\xa2\x05\x87\x18\x82\x8caal\xd8*X%\x00\x01q\x12 X\x9a\xceh\xa5\xf2\xdbM\xad$\xdfW\xec\xc9\xae\x1e\x0b\xd8\xcc6\x8f\x84N>\x16\xbb\x07MmV[\x9bS\x01q\x12 \xaf\x1d\xd8`\x95\x1d\xe891\xa7\x8e\xf1\xc92\xd6P&\xe5X(\\r\xd9\x1a\xb3\x0f18\x86\xff.\xf3\xa2ae\x80al\xd8*X%\x00\x01q\x12 \xd2`\xd4i\xe4~P\xfd\xaed\x0eT\xca*\xdc\x16\x82\xf5\x13P\x88\\\x03?\xf1\xc1/i\xd4\xd3\xf9WS\x01q\x12 \xd2`\xd4i\xe4~P\xfd\xaed\x0eT\xca*\xdc\x16\x82\xf5\x13P\x88\\\x03?\xf1\xc1/i\xd4\xd3\xf9W\xa2ae\x80al\xd8*X%\x00\x01q\x12 \x95{\xa3UN\xa1\xb7\x8ae\x85\xdf~\x9b\x86\xc72n\xbd\xd6\xc7\xe5%\xbb\xa9\xee\rH\x1f\xb7^A\x0eS\x01q\x12 \x95{\xa3UN\xa1\xb7\x8ae\x85\xdf~\x9b\x86\xc72n\xbd\xd6\xc7\xe5%\xbb\xa9\xee\rH\x1f\xb7^A\x0e\xa2ae\x80al\xd8*X%\x00\x01q\x12 S\x815\x9e\x18\x96\x05Ku\xf8B\xc7I\xc3\x8d\xfe\x17\xcf\x1e\xf0{\x1c\xe1\xf3\xf7\x06\xb6\xb9\x8c\xb5\x10\xb7\xee\x05\x01q\x12 S\x815\x9e\x18\x96\x05Ku\xf8B\xc7I\xc3\x8d\xfe\x17\xcf\x1e\xf0{\x1c\xe1\xf3\xf7\x06\xb6\xb9\x8c\xb5\x10\xb7\xa2ae\x86\xa4akX app.bsky.feed.like/3jvsxqnb3fk2iap\x00at\xd8*X%\x00\x01q\x12 O0\xba\xce\xfc\x80\x9e \xeb.B\x11#8AFs\xb2K5\x892=\\\xb1\x15g\xa3N>\x18pav\xd8*X%\x00\x01q\x12 \xd5\xf9{\x87\xa1\xd5\xb7{\xe7K\x87T\xab\xf4(\x85\xb1\x89!\xdc1\x92\xb2\x93\x91\xbd\xe3\x06\x9c\xe9.y\xa4akRpost/3jvimk4ldnk2gap\x0eat\xd8*X%\x00\x01q\x12 \x99\x04\xdd\\\xe5\xba\xa1\xbc\xa3|\xc1\xc8X\xbb\x8a>\xc6^\xa2\xd5\xfc\x8c\xb2h\x9d\xedC\x80N\xec\xcb\rav\xd8*X%\x00\x01q\x12 o\xc0\x07V\xff\xfe< /ouVu\x7f3\xfdm\xd4\x1eEE\x93\xd3\'\xe5x]x\xbfR\xa1x\xa4akKw6rrfvy2m22ap\x15at\xd8*X%\x00\x01q\x12 \xdbP7\x9a9Js\xa1Bba}N^K\xcb2Mj\x98\xd7\xed\xb0n\xb7\xa6O\xdc)\xcd\xf3\xc1av\xd8*X%\x00\x01q\x12 \xab\xf9\xf3U\xb31+\x8b\xb2\xbe\xd0S\x8eT\x10L,\xfdP\xc42\xdf\xfb\xc7`\xe6\x83\xf0\x8e\xaez\xba\xa4akJj6pum5gs2cap\x16at\xd8*X%\x00\x01q\x12 D\xcd\xbdV}\xe4\xf94d>/j\x02\xa5e"\xe7\x038o^*\x062\xfdo\x9aF\x8b\xd3\x0f{av\xd8*X%\x00\x01q\x12 \x0fr\xb8\x7f\xde\xa6U\x9d\x0f\xac\x9d\x8fb\xef\xe5\x0fp\x144\x18\xf3-\xc5\x89\xe2\xac@\xd3?7p\xbc\xa4akJlob47vs22fap\x16at\xd8*X%\x00\x01q\x12 \xf5\xf2[\xc19\xed\xe4d<\xaf\xb3UZ\xc5\xd3\x93"\x14((\xbf\xf6\xf8\x1b\x9d8\xb1zi\xea\xf24av\xd8*X%\x00\x01q\x12 t\x8a\x08\xaf\xa6\x17\x16\xcc\x1cT\xaf\xcc\xda\xea\x87\x86\x93\x9b\xe2\xbe-\xf6\x18\xe3\'X8}\x97A\x89P\xa4akX\x1agraph.follow/3jw2l3srjvk2yap\tat\xd8*X%\x00\x01q\x12 \xfb\xca\xb9\xe2\xc3X!\x05c\'/\xadF\xca\x16\xa1Rd\xd90[\xccb\xe4\x81\xed\xc2\xf9\xdet\x7f\xc5av\xd8*X%\x00\x01q\x12 mM#\x06\xd3*\xb6S\x15\x16\xd8\x03\x16`Y\xa8\xeds\x06F\x95\xd6\xb5\xa2\xa5j\x01P\xef.D\x8fal\xd8*X%\x00\x01q\x12 Zef\x94\x94\xc4\xb9\xba\xa2\t\x17k\xe7\x95\x1b3~0\x15\xeb\xa66\xbda0\xc1+\x14p\xe6\xfd(\xc8\n\x01q\x12 O0\xba\xce\xfc\x80\x9e \xeb.B\x11#8AFs\xb2K5\x892=\\\xb1\x15g\xa3N>\x18p\xa2ae\x8c\xa4akX app.bsky.feed.like/3jvxznyl55k24ap\x00at\xd8*X%\x00\x01q\x12 \x86\xba\xc5\x8a\xf5\xa9\x80\xb8\x16\xac<\x98\xf8\x87\x89\x8f}\xb7\x84\xdbQ\x11\xa5\x94\xc81\x88\xc4\xca\r\xcdTav\xd8*X%\x00\x01q\x12 \xe0\n\xbd(-\xfd\x10\tF~\xfb:\x93\xf64\xf2\xba\xe1K\xf1\xd8Z\xb2\xa7\xcfW/\xbf\xb9;\x9b\xc2\xa4akKwjljagh7k2map\x15at\xd8*X%\x00\x01q\x12 \x83 \x1fJd/\x83;\xad\xf4>\xd5=\x1d\xaf>n\xec_cnB\'\x96\x1b\xbdK\\\xf0\x17f`av\xd8*X%\x00\x01q\x12 \xef\x06\xee\xf1\xb4\xfc\xd6\x06T\xac\xf7\x8a\x19\x11n\xe6_s\x9c\xad\x7f\x1e\xb1\xf6\xa2}\x1a\xd0\xb7[\xaf\xb8\xa4akJnju3rvps2bap\x16at\xd8*X%\x00\x01q\x12 \xaa\xadnz0\xc2\xbb\xb4\xd7\xe8}\xc4\xbbZ\x12\xbb\xacU:\x13\xbe\xf6\x1d$y\xf3\x90y\xb3\xba\xb8\x14av\xd8*X%\x00\x01q\x12 \x8c\xaf\xd5\xdb\x88\xc7\\0\xdc\x9d\xec\x15uHp\xff\x8a"\x06I\x14\xe4\xdfh\x00l\x91\xb83\x91\xc4\x02\xa4akJq4nf7cuq2fap\x16at\xd8*X%\x00\x01q\x12 \x1d\xd7\x07\x18\xbd%\x1c66@)Z\xc5=\xae\x12\x8cD\x87\x93%\xac\xae\x1e\x9f\xb6\x0258\x97r1av\xd8*X%\x00\x01q\x12 \xaf\xa8@\x1d\x1a\x01\xcfk\x84-\xd2\xe7\x06\xb4\x12G\x0f/G:\x02\xa1\x04\xee\xa0\x12\x84\xed2\xae\xe3d\xa4akJwfamw6bl2zap\x16at\xd8*X%\x00\x01q\x12  \x12I\xa7}.\x8a\xc6\x8ck\x98b\xa5z\xc7\xbc\xeb\xb5U\xcb\xfb\xa2Qn_a6\x16\xb1\xba\xd7gav\xd8*X%\x00\x01q\x12 \xff>@M\x8b$"V\x8d\xd0\x97\x91\xaf\xd5\xa6\xf8\xfb\xe2\x94\xef\xe9\xa4\xc5\xa4R\x9e\xc7Y9A.\xa0\xa4akJxv63njz32hap\x16at\xd8*X%\x00\x01q\x12 \xb2\xc6\x15\x0c,\xaa\xf7Y\x13\x95\xf17\xff-\x93\xa5\xb8\x8d\xdc\x84\x92\xa7\x86{\x05\xef\xb7\x8c\x16|_\xfeav\xd8*X%\x00\x01q\x12 \xfc\xa4\xfc\xb2"\x0f^\xb9\xa62\xbe\t\xc4\xe61\x8df{\xe3\x86#\xa1x\xc1-\xc5E!\xc0\xd4\xab\xac\xa4akJyn5klqfu2sap\x16at\xd8*X%\x00\x01q\x12 \x1d\xc0gG\xefk\x01E\xfb\x11$\x9e:\xed\xf2\x8f1\x9c:\xc8Y\xcc\x98\x8a\x14\xee\x17\xe2\x0b\n\xea\xcfav\xd8*X%\x00\x01q\x12 U@\x17lFWw\x87\x8fwN\x04P\xdf\x8e\xa9MY(&s\x8c\xcb\xda\xf6\x0c\xcb\x0c6\xb5\xb6\n\xa4akRpost/3juugscu2e22cap\x0eat\xd8*X%\x00\x01q\x12 \x94\xcd\xc6v\xf6\x11w\xdc\xf92Q2\xd73\xbf\x99\x066\xca\x9f`\xf4\x8f\xd1\x84?v\xfeIi\x85\xbeav\xd8*X%\x00\x01q\x12 9\xd7\xb9^\x1dyr\x1dl\xd4\xce\x0c,\x1f\x07\x05!\xd4S\x84j;\xa9L]\xd5/SV2\xe5\xc5\xa4akKv47vn7cqi2dap\x15at\xd8*X%\x00\x01q\x12 1>a\x94]ev\xf4*\x8f~\xa1\x86\x01\x1a7\x81\xa6X\xa6g\x7f\xc9\x960\xf4"L\x81l\xacgav\xd8*X%\x00\x01q\x12 \xcb{&\xafF\x87\x05h\xdaA|\x12\x00\xd4z\xab\xccM\n\x18\x1c0@\\\xaa\x02UJ&\x0e\xbe\xf7\xa4akJ66zsl5ue2tap\x16at\xd8*X%\x00\x01q\x12 \x01B\x05O\xcc\x02*4S\xa3l\xff\xfa\xe1\x12Z\xa7\x1e\xbe0@\xd3\xcb,\x98\xb4O\xd0\xe1:c]av\xd8*X%\x00\x01q\x12 \x83\x10\x7f\xfc\x9aEE\x9a=\x1b\xbe/p\x00\x04\xa3\xb8\x80\xea\x91\xd9H\xd6\xfew\n\x12\xe2\x03\x1a\xc2\xf7\xa4akJdkh5db6v2fap\x16at\xd8*X%\x00\x01q\x12 \xf5\xcc\xa4d6\x15N\xbb\xf6\xdfc\xe9\xea\x10o\xc6\x1d\xf5\x9a\xb5P\xe1\xa6\xf9\rug\xec_\xf8\x1aWav\xd8*X%\x00\x01q\x12 \xfd\xda\x03\x1a\x8e\x97\x1f\xad\xd5_\x08\xc7\xb3\xb6\xe3\xe3i\xc1\xc8CE\xeb\xd7\x8f\x89\xcavr\xe5\xe7\x17\xc3\xa4akJhl3hgmmc2map\x16at\xd8*X%\x00\x01q\x12 Z\xab\x0f\xeaCJ\xde\x17\x0f\x0f\xd3k\xc80\xecc K\xf0,\xe4\xbb+\xbag\xe9\x10\x18\x97\x17\xf6\xfbav\xd8*X%\x00\x01q\x12 \xf4\xe2\xdca\xd0"\x04\x8f\x18\x87\xc4\x99Y0\x18\x99\xfa\xcd\xdf\x08_\x90{\xfe\xeb\xbf\x91\x97\xb0Mrcal\xd8*X%\x00\x01q\x12 6c\x120\xb0\xb6\xddRV&\x84\xa1L\xd6f\x18\x87\xcf7$\x916_\xf6Zi\n\x96\xd4\xf2\xc2\xea\x8b\x08\x01q\x12 \x1d\xc0gG\xefk\x01E\xfb\x11$\x9e:\xed\xf2\x8f1\x9c:\xc8Y\xcc\x98\x8a\x14\xee\x17\xe2\x0b\n\xea\xcf\xa2ae\x89\xa4akX app.bsky.feed.like/3jwysed2ofn2fap\x00at\xd8*X%\x00\x01q\x12 \xb6\x17\xce\x02bS\xda\xe7MZh\x15\xd0`\xd4\xad\xa0 \x0f,\xf4\xe47\r\xcd\xb1s}\x11{\x85yav\xd8*X%\x00\x01q\x12 \xa5\xc2^s\x83ekK\x0eG\\E\xea\x16\xa9.\xcd\xfd\xc1J\x06-\xe3\xaeZA\xb5\xaa\x87\x1b\x13\xb3\xa4akRpost/3jumie2dd4f2rap\x0eat\xd8*X%\x00\x01q\x12 \xe9\x99\xca\x8a\xb6\x94\x9e\'\xfb\x141\xa0Yuy\x16&\x9cZ\xf4\xee%{J2\x9d\x1b!\xf2\xa4\xfe\xe6av\xd8*X%\x00\x01q\x12 \xa9y\xb2\xbc\x1d\xa4\x9f\xe1\x08\xfd\xe7b\x12\r\xe6\xed\xe9w\xdfT\x9b\x91\x91R\x88\x08\xa9wl@\xec\xf1\xa4akJr4xwcsl62hap\x16at\xd8*X%\x00\x01q\x12 \xbd\rM\x12t\x03\t\xfc\xae\x11\xb3,\x1a\xa2\x1c\xe4\xc6\xdar\x8ad_p@s\xd2\xa1\x8a\xd8\xf8\xec\x14av\xd8*X%\x00\x01q\x12 \xed\x1cG\xc3\xc1\x17\xe4{8\x13w&\xe3\xc3\xec\r0#d>W\xbd\xae\x8f\x00\xd3\\\xdfg\t[\xdd\xa4akIbx7dsmk23ap\x17at\xd8*X%\x00\x01q\x12 :\x83\xf8YL\xd6\x1e{lL\x1c\xbb\xb0\xe8"\xbd\xea\x15f\xe6\xc9\xb8&j\x8f\xd1\xc6\x9e\x8f\xeb9\xa1av\xd8*X%\x00\x01q\x12 \xa7\rlr\x9e\x13|C\xf2\x7f\x91\xce\xb9\x86}\xd3\xe7%\xc9\xda\xa5\x0f\xb7\xa9\xbd\xce\xf3\x888\x1bK\xe9\xa4akIlrowu3i2sap\x17at\xd8*X%\x00\x01q\x12 \x18\xc6\xfbn\xb0%2a\x1eH\xf8\x0b\xe7\\\x18\xa4\'\x7f\x98\xb5L\xd7\\\xd84cP\x91Q\x99P\x88av\xd8*X%\x00\x01q\x12 \x0f\xb9\x13\xf2>\x96F62\x0e\xf9\x9b\x1fE\x1e4&\xc0\xbdP\x84.\xce\xece0\xe3\xa9\xac\xcb\xc3\x8b\xa4akIo7yayc52cap\x17at\xd8*X%\x00\x01q\x12 F\x1fd\xe9X\xcfV\x89+=$\x81X\x96\xf5)\x18\xd1l%\x86\x81\xb2x\x87\x9f\xed\\\x9e\xb7\twav\xd8*X%\x00\x01q\x12 \xfb8E\xe8\xe93\x80\x1a=\xed;\xa6\x88\xa2\x80\x04\x9e],\x13\xc5\x90\x12\xb1\x1c\x98!\xfb\x81_\xef\x9d\xa4akIrvz3ugc2bap\x17at\xd8*X%\x00\x01q\x12 \x14\xc4\xae\xa6\xf3{*\xad\x8e\xd0ca\x92\xd7\xda\x0b\x11AK\xfeW\xf2\xc3\xc1\xd6|<\xb2\xb9b\xc1Hav\xd8*X%\x00\x01q\x12 \xe4\xe0\x98Q\xf3\'\x99\xd7\x8b=\xfev\xc2\x1c\x82%/\x02\xc2\x14o\xff\xc3>e2\xf3T\xdfxm\xa1\xa4akIui3comc2nap\x17at\xd8*X%\x00\x01q\x12 0\xcf\x8d\x14\x1f\xf2\xf75T\xc4\xd5\xf5Ov\xd0\x0b\x94')

from atproto_core.cbor import decode_dag_multi
dag = decode_dag_multi(data)

only decodes the first part

{'t': '#commit', 'op': 1}
MarshalX commented 9 months ago

tried via js @ipld/dag-cbor and got more details

throw new Error(`${decodeErrPrefix2} encountered invalid minor (${minor}) for major ${data[pos] >>> 5}`);

should not be related to issue no 53 because SDK changed lib for websocket. and the current _MAX_MESSAGE_SIZE_BYTES is 5mb

azulika commented 9 months ago

I see. Perhaps I should've chosen a different example.

import requests
from atproto_core.cbor import decode_dag_multi

url = "https://bsky.network/xrpc/com.atproto.sync.getRepo?did=did:plc:ewvi7nxzyoun6zhxrhs64oiz"
data = requests.get(url).content
dag = decode_dag_multi(data)
print(dag)

This only returns:

[26, {'version': 1, 'roots': ['bafyreig67suqxcebtiv453ujpveydtv4doolnebl5yfnqx54evlg6u3tn4']}, None, 1, 1]

Decoding the same PDS instance via js @ipld/dag-cbor works perfectly. Is decoding a bsky.social instance not supported in decode_dag_multi or am I missing something?

MarshalX commented 9 months ago

@azulika you are missing the difference between the CAR file and DAG-CBOR

the right code:

import requests
from atproto import CAR  # same as from atproto_core.car import CAR

url = "https://bsky.network/xrpc/com.atproto.sync.getRepo?did=did:plc:ewvi7nxzyoun6zhxrhs64oiz"
data = requests.get(url).content
car = CAR.from_bytes(data)
print(car.root)
print(car.blocks)
azulika commented 9 months ago

@MarshalX Thanks!