Open mockthebear opened 1 week ago
MarshalX
commented
1 week ago Super duper interesting. Probably fails on https://github.com/MarshalX/python-libipld side.
Could you please export problematic data bytes to the file? And prepare reproducible example smth like with open(f.bin.....) as f: Frame.from_bytes(f.read())....
It will help a lot! Thank you
mockthebear
commented
1 week ago Does this help?
from typing import Union
from atproto import CAR
import ast
def read_binary_dump(file_path: str) -> Union[bytes, bytearray]:
with open(file_path, "r") as file:
human_readable_string = file.read().strip()
binary_data = ast.literal_eval(human_readable_string)
if isinstance(binary_data, (bytes, bytearray)):
return binary_data
else:
raise ValueError("The parsed data is not bytes or bytearray.")
file_path = "crashy.txt"
data: Union[bytes, bytearray] = read_binary_dump(file_path)
CAR.from_bytes(data)
# python3 rd.py
Segmentation fault (core dumped)@mockthebear thank you! Super strange. I run repos firehose from 3720770000 cursor. Nothing happens locally for a 3+ min. How much should I wait to reach problematic frame?
Speaking of your example it gives errors, but ast and literal_eval scares me. So I just saved it in non-human readable format like this:
with open('crashy.txt', 'r') as file:
binary_data = ast.literal_eval(file.read().strip())
with open('data.bin', 'wb') as file:
file.write(binary_data)and now it does not segfaults, but gives proper errors about wrong varints, etc
this is the code that I use to reproduce with cursor (pls run locally):
from atproto import models, FirehoseSubscribeReposClient, firehose_models, parse_subscribe_repos_message
client = FirehoseSubscribeReposClient(models.ComAtprotoSyncSubscribeRepos.Params(cursor=3720770000))
def on_message_handler(message: firehose_models.MessageFrame) -> None:
_ = parse_subscribe_repos_message(message)
print(message.header)
client.start(on_message_handler)
okay, found thanks to discord
@DavidBuchanan314 executing the reported issue https://github.com/MarshalX/python-libipld/issues/9
mockthebear
commented
1 week ago Usually a few seconds, it never passes trough the xxx7000 to xxx8000
Sorry for that, i'm not as experienced in python, so i asked for gpt to parse that output for me x.x
I see i see. Maybe because the machine i'm running it has only 1gb of ram left it happens?
The code i'm running is this one and it crashes on this line: https://github.com/MarshalX/bluesky-feed-generator/blob/main/server/data_stream.py#L19
Changing just a bit your code, i get the crash:
from atproto import models, FirehoseSubscribeReposClient, firehose_models, parse_subscribe_repos_message, CAR
client = FirehoseSubscribeReposClient(models.ComAtprotoSyncSubscribeRepos.Params(cursor=3720770000))
def on_message_handler(message: firehose_models.MessageFrame) -> None:
commit = parse_subscribe_repos_message(message)
print(message.header)
CAR.from_bytes(commit.blocks)
client.start(on_message_handler)MessageFrameHeader(op=1, t='#commit')
MessageFrameHeader(op=1, t='#commit')
MessageFrameHeader(op=1, t='#commit')
MessageFrameHeader(op=1, t='#commit')
MessageFrameHeader(op=1, t='#commit')
MessageFrameHeader(op=1, t='#commit')
MessageFrameHeader(op=1, t='#commit')
MessageFrameHeader(op=1, t='#commit')
Segmentation fault (core dumped)It takes about 9889 lines of 'MessageFrameHeader' to get to the problematic message
MarshalX
commented
1 week ago Reproduced, thank you! Known edge case, but this is first time when someone exploits it in the network
mockthebear
commented
1 week ago Aparently the post came from https://bsky.app/profile/david.dev.retr0.id
Li-WeiCheng
commented
1 week ago Hi,
I encountered the same issue.
Currently, my approach is to skip the data/cursor after Segmentation Fault occurs.
Is there a better solution for this?
Thanks :)
mockthebear
commented
1 week ago Hi,
I encountered the same issue.
Currently, my approach is to skip the data/cursor after Segmentation Fault occurs.
Is there a better solution for this?
Thanks :)
Thats the same solution i did, and thats THE solution so far. Until someone fixes it. Lets just hope someone does not abuse this bug again XD
pai911
commented
1 week ago encountered the same issue here. Does it help to upgrade the https://github.com/MarshalX/python-libipld in use?
MarshalX
commented
1 week ago @Li-WeiCheng hi, no better solution yet :( @pai911 we need to fix python-libipld first
MarshalX
commented
1 week ago I finished with fix. Gonna release it soon https://github.com/MarshalX/python-libipld/pull/51
i was consuming data from the firehouse, specifically at cursor: 3720770000, every time i get a consistent Core Dumped
I managed to find the crash is coming from: .venv/lib/python3.12/site-packages/atproto_firehose/client.py: frame = Frame.from_bytes(data)
If i print the data i get:
There is no verification of invalid data or too big data, causing the crash