search

rschroll / rmfuse

FUSE access to the reMarkable Cloud
MIT License
95 stars 8 forks source link

Crash when trying copying more than one pdf #19

Closed f3fora closed 3 years ago

f3fora commented 3 years ago

When copying with

cp /path/to/*.pdf remarkable/

I get 

cp: cannot stat '/path/to/file.pdf': Transport endpoint is not connected

and rmfuse crashes with

WARNING:root:Getting Item e945f1a0-740b-4735-9e20-96134bb1948e during upload.  This may lead to odd behavior!
   unique: 78, success, outsize: 16

Multiple selection works with filemanager instead.

rschroll commented 3 years ago

I was worried about this, as you can probably tell from the warning message. There's a little gap in time between when a file upload starts and it finishes where your system will obviously expect a file to exist, but the RM cloud doesn't yet have a entry for that file. It's short, and I hadn't managed to trigger that state. But it looks like you found a way!

One thing to try would be to remove the for item in self.uploading.values(): block in get_by_id, starting here: https://github.com/rschroll/rmfuse/blob/master/rmfuse/fuse.py#L146. This will cause rmfuse to immediate return an error in this case, instead of returning an object in a possibly inconsistent state. This doesn't solve the problem, per se, but the failure may be something that the system can deal with in a not-too-broken way.

(This is how rmfuse used to work, but in #2, I was worried that this might be a problem. Turned out it wasn't the issue, so we might be able to just undo the fix.)

If that doesn't produce satisfactory behavior, do you get any traceback from the current code? Or does it just die without any message other than the warning. (Annoyingly, this does happen, which makes figuring out where the failure actually is pretty annoying.) I'll keep trying to reproduce the failure, but I've had no luck so far. I suspect it's an issue of OS + shell + upload size + upload speed.

One possible solution would be to put a lock files being uploaded, and then wait for that lock to be released in get_by_id before responding. Blocking access isn't great, but it's probably better than trying to pretend everything's okay.

f3fora commented 3 years ago

One thing to try would be to remove the for item in self.uploading.values(): block in get_by_id, starting here: https://github.com/rschroll/rmfuse/blob/master/rmfuse/fuse.py#L146. This will cause rmfuse to immediate return an error in this case, instead of returning an object in a possibly inconsistent state. This doesn't solve the problem, per se, but the failure may be something that the system can deal with in a not-too-broken way.

Removing this part of code actually seems to solve the issue, for me. But I don't know if it is useful somewhere else.

If that doesn't produce satisfactory behavior, do you get any traceback from the current code? Or does it just die without any message other than the warning. (Annoyingly, this does happen, which makes figuring out where the failure actually is pretty annoying.) I

It just crash. I have report all the output I got. No traceback.

I'll keep trying to reproduce the failure, but I've had no luck so far. I suspect it's an issue of OS + shell + upload size + upload speed.

I use Arch Linux + Bash 5.1.4 + 2 PDF of 20K and 25K. I haven't checked the upload speed. How can I eventually get the data you are interested in? I can reproduce the issue with pip and poetry.

One possible solution would be to put a lock files being uploaded, and then wait for that lock to be released in get_by_id before responding. Blocking access isn't great, but it's probably better than trying to pretend everything's okay.

I think that actually showing a file such test.pdf.uploading can be useful especially when uploading a large file and the file manager takes a while before display the actual document.

rschroll commented 3 years ago

Thanks for checking! The uploading file state was put in to try to solve a Mac problem, but that turns out to be unrelated. So getting rid of it is a reasonable solution.

That said, I like your idea about a file indicating the upload is happening. Here's a couple ideas:

  1. Put the content "Uploading..." into the file name being uploaded. Pros: Clear what is happening, not creating odd file names. Cons: Upload may result in a file with a different name (eg epub file in annot mode), file viewers may be confused by file contents not matching extension, thumbnail creation may be messed up.
  2. Make the file being uploaded have zero length. Pros: Clear that contents are missing, doesn't create odd file name. Cons: Upload may result in a different file name, meaning of empty file may be unclear, thumbnail creation may be messed up.
  3. Make an empty file at filename.uploading. Pros: Won't be confused with eventual real file, file browsers won't try to open with a PDF viewer. Cons: Files will appear and disappear during upload, won't help tools that are trying to check whether the new file just got created.

What do people think? Any preferences among these?

On Mon, Mar 15, 2021 at 2:22 AM AF @.***> wrote:

One thing to try would be to remove the for item in self.uploading.values(): block in get_by_id, starting here: https://github.com/rschroll/rmfuse/blob/master/rmfuse/fuse.py#L146. This will cause rmfuse to immediate return an error in this case, instead of returning an object in a possibly inconsistent state. This doesn't solve the problem, per se, but the failure may be something that the system can deal with in a not-too-broken way.

Removing this part of code actually seems to solve the issue, for me. But I don't know if it is useful somewhere else.

If that doesn't produce satisfactory behavior, do you get any traceback from the current code? Or does it just die without any message other than the warning. (Annoyingly, this does happen, which makes figuring out where the failure actually is pretty annoying.) I

It just crash. I have report all the output I got. No traceback.

I'll keep trying to reproduce the failure, but I've had no luck so far. I suspect it's an issue of OS + shell + upload size + upload speed.

I use Arch Linux + Bash 5.1.4 + 2 PDF of 20K and 25K. I haven't checked the upload speed. How can I eventually get the data you are interested in? I can reproduce the issue with pip and poetry.

One possible solution would be to put a lock files being uploaded, and then wait for that lock to be released in get_by_id before responding. Blocking access isn't great, but it's probably better than trying to pretend everything's okay.

I think that actually showing a file such test.pdf.uploading can be useful especially when uploading a large file and the file manager takes a while before display the actual document.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rschroll/rmfuse/issues/19#issuecomment-799259037, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACZ2H3SHD2YIAAHOSSNQILTDXGXDANCNFSM4Y6XSC2A .

f3fora commented 3 years ago

Thanks for checking! The uploading file state was put in to try to solve a Mac problem, but that turns out to be unrelated. So getting rid of it is a reasonable solution.

I sent a PR.

  1. Make an empty file at filename.uploading. Pros: Won't be confused with eventual real file, file browsers won't try to open with a PDF viewer. Cons: Files will appear and disappear during upload, won't help tools that are trying to check whether the new file just got created.

I will go with this. Easy and do not touch in anyway final files.

rschroll commented 3 years ago

I think the reversion should solve this problem. If we want to add a placeholder file, I'm still open to that.