search

archiecobbs / s3backer

FUSE/NBD single file backing store via Amazon S3
Other
538 stars 77 forks source link

"Broken Pipe" errors when running in NBD mode #182

Closed Nikratio closed 2 years ago

Nikratio commented 2 years ago

Hi,

I've been experimenting with NBD mode, and I'm at times getting errors like these:

nbdkit: s3backer.8: error: write reply: NBD_CMD_WRITE: Broken pipe
nbdkit: s3backer.15: error: write reply: NBD_CMD_WRITE: Broken pipe

I think they're not related to the NBD code itself, but issues in the code that are triggered by the way that NBD requests are coming in.

What do you think?

archiecobbs commented 2 years ago

I don't think s3backer is generating those messages.

The strings "write reply" and "NBD_CMD_WRITE" don't appear anywhere in s3backer.

Also s3backer doesn't (itself) originate any EPIPE errors.

That doesn't mean it's not an s3backer bug of course.

Nikratio commented 2 years ago

You're right of course about the origin of the messages. I mixed this up with the other bug (assertion failure).

What is happening here (I think) is that s3backers pwrite handler sometimes return EPIPE errors, which causes nbdkit to write these messages.

On the other hand, it seems that should not happen either?

I'll try to instrument the code to check (if I find some free time)

Nikratio commented 2 years ago

Seems the instrumentation is already present in the newest branch. I'll report back when this happens with the most recent code.

Nikratio commented 2 years ago

Looking at https://gitlab.com/nbdkit/nbdkit/-/blob/master/server/protocol.c#L382, it looks as if this could be errors when sending something to the NBD client. I've asked on the nbdkit mailing list for advice.

archiecobbs commented 2 years ago

That error makes sense in the scenario where the client has disconnected abruptly for some reason. Maybe something in your testing was causing that to occur sometimes?

Nikratio commented 2 years ago

Having looked into it, it seems this problem is with the kernel's NBD client. Followed up on https://lists.debian.org/nbd/2022/05/msg00003.html.

archiecobbs commented 2 years ago

Great - thanks.