Clarifications

[f3fora]

fusermount -u ~/remarkable is useful only to unmount rmfuse which runs in background (eg rmfuse ~/remarkable &), otherwise a ctrl-c is sufficient. Right?
In the last case, I suggest to manage the KeyboardInterrupt exception.

The file size for annotated files is just an estimate before the file is first read. This can confuse some tools which use the file size to determine how much to read. After reading the file once, the file size will be correctly reported going forward; rerunning these tools a second time is usually enough to get them working.

I am not able to open myFile.pdf with xdg-open myFile.pdf if myFile.pdf is in the mounted directory and is a notebook or a annotated pdf. However if I copy it in another directory is ok. Is this the way rmfuse is suppose to work?

Is there a way to enable/disable the render of templates without manually add/remove the ~/.local/share/rmrl/templates directory? Eventually, can a mode be added?

[rschroll]

On Sat, Feb 13, 2021 at 5:25 AM AF notifications@github.com wrote:

fusermount -u ~/remarkable is useful only to unmount rmfuse which runs in background (eg rmfuse ~/remarkable &), otherwise a ctrl-c is sufficient. Right? In the last case, I suggest to manage the KeyboardInterrupt exception.

That's correct. RMfuse catches all errors and unmounts the filesystem before shutting down. I typically use ^C in development, and I haven't noticed any problems with this.

I am not able to open myFile.pdf with xdg-open myFile.pdf if myFile.pdf is

in the mounted directory and is a notebook or a annotated pdf. However if I copy it in another directory is ok. Is this the way rmfuse is suppose to work?

Ideally, no. I've noticed some viewers (evince) are okay with the incorrect file size, while others (chrome) are not. But in all cases, trying to open the file a second time succeeds, since the file size is corrected after it was rendered. Is that what you're seeing? Am I correct in reading between the lines that non-annotated PDFs open without a problem?

If not, some debugging questions:

1. What PDF viewer is xdg-open trying to use on your system?
2. Does xdg-open succeed in opening the viewer, only for it to fail, or does xdg-open itself fail?
3. Do you get an error message? Does it crash? Or does it just hang?
4. If it hangs, for how long have you waited? As you may have noticed, RMfuse is a bit slow in rendering annotated PDFs. Running with -vv will output a whole mess of information, but it will show you if something is happening while the viewer hangs.

Thanks!

[f3fora]

That's correct. RMfuse catches all errors and unmounts the filesystem before shutting down. I typically use ^C in development, and I haven't noticed any problems with this.

Thanks. I think I had no problem with ^C. Maybe instead of rising KeyboardInterrupt, getting a rmfuse unmounted correctly will be more informative and user-friendly ;-)

But in all cases,trying to open the file a second time succeeds, since the file size is corrected after it was rendered.

Actually I get always the same error.

Am I correct in reading between the lines that non-annotated PDFs open without a problem?

Yes, you are.

1. What PDF viewer is xdg-open trying to use on your system?

zathura. Actually I get the same error using xdg-open myFile.pdf or using zathura myFile.pdf explicitly.

2. Does xdg-open succeed in opening the viewer, only for it to fail, or does xdg-open itself fail?

xdg-open works. Neither zathura or firefox can show it.

3. Do you get an error message? Does it crash? Or does it just hang?

$ zathura Test.pdf
error: read error: No data available
warning: read error; treating as end of file
error: cannot find startxref
warning: trying to repair broken xref
warning: repairing PDF document
error: read error: No data available
warning: read error; treating as end of file
error: read error: No data available
warning: read error; treating as end of file
warning: object missing 'endobj' token

And as output on zathura GUI I get Document does not contain any pages

Some similar message I get in firerfx GUI.

It just doesn't show anythings.

4. If it hangs, for how long have you waited? As you may have noticed, RMfuse is a bit slow in rendering annotated PDFs. Running with -vv will output a whole mess of information, but it will show you if something is happening while the viewer hangs.

Typing cp Test.pdf .. and then opening ../Test.pdf works just fine. Typing zathura Test.pdf doesn't work, but I get the same log as the previous command.

[rschroll]

Good thought on the ^C exit -- I've turned off the traceback in that case.

Thanks for the detailed debug info. The "read error: No data available" is exactly the error that RMfuse throws when a program tries to read past the end of a file. It often occurs when a program is using the inaccurate file size to figure out how much to read. It looks like zathura is hitting that and then feeling that the PDF file is incomplete. Perhaps it's counting on the xref table being somewhere in the last $n bytes, and instead of backing up when it can't read them, just gives up?

I've installed zathura locally, but I'm unable to reproduce the problem. Perhaps it's a version issue; here's what I have:

$ zathura --version
zathura 0.4.3
girara 0.3.2 (runtime: 0.3.2)
(plugin) pdf-poppler (0.2.9) (/usr/lib/arm-linux-gnueabihf/zathura/libpdf-poppler.so)

If you're willing to dig around a bit, I'd be curious to hear what happens if you comment out lines 264 and 265 of rmfuse/fuse.py. This will prevent the error in the read-past-end case, and just return an empty buffer. This caused problems in some clients when I was testing, but maybe it will work for zathura. (If so, I suspect we might be able to get things to work by returning the empty buffer the first time a read goes past the end, and then raise an error on subsequent reads.)

[f3fora]

I've installed zathura locally, but I'm unable to reproduce the problem. Perhaps it's a version issue; here's what I have:

Zathura uses plugins. If you try zathura-pdf-mupdf, you should be able to reproduce the issue.

I find a workaround for the problem. It is related with the pdf rendering library. The reader based on poppler library works, while the one on mupdf not.
From what I understood, Firefox uses instead PDF.js, and doesn't works.

Most of PDF viewers in Linux are based on poppler, so they should work fine.

[f3fora]

Is there a way to enable/disable the render of templates without manually add/remove the ~/.local/share/rmrl/templates directory? Eventually, can a mode be added?

[rschroll]

Good to hear there's a work around. I'm not finding zathura-pdf-mupdf in the raspbian archives. I did try mupdf proper, but no luck triggering a problem. I learned a bit more about PDF files, and there's a startxref attribute near the end, designed to let programs know where to find the xref table. I suspect mupdf (or at least the plugin) is being clever by reading the end of the file to find startxref, then reading the xref table, and then reading the parts it needs, rather than just reading the whole file from the start. As a result, it tries to read past the end of a file and gets confused.

I could imagine an option where RMfuse didn't guess about annotated file sizes, but rendered everything to give proper measurements. That would slow down things at the beginning, but the caching would save those sizes for the future.

As a stop-gap, I think running something like wc * on the directory would cause all the files to be rendered and therefore their sizes to be properly calculated. After that point, any viewer ought to be able to open them without any problem. (It may take a while, though, depending on the number of files present.)

Feel free to open another issue about including the templates. My long-term plan is to add an option to set the alpha on the templates, to better reflect what they look like on the screen. So setting alpha = 0 would be an option there. No timeline on that, though.

[f3fora]

Good to hear there's a work around. I'm not finding zathura-pdf-mupdf in the raspbian archives.

zathura-pdf-mupdf is available only on Arch or Gentoo repos.
Can you reproduce the problem with firefox?

I suspect mupdf (or at least the plugin) is being clever by reading the end of the file to find startxref, then reading the xref table, and then reading the parts it needs, rather than just reading the whole file from the start. As a result, it tries to read past the end of a file and gets confused.

Ok.

I can say that the problem is not related with rmrl. ;-)

As a stop-gap, I think running something like wc * on the directory would cause all the files to be rendered and therefore their sizes to be properly calculated. After that point, any viewer ought to be able to open them without any problem. (It may take a while, though, depending on the number of files present.)

Maybe I was not clear. With this two pdf reader I was never able to open the document in the fuse mount directory. (And wc * do not solve the problem.)

Feel free to open another issue about including the templates. My long-term plan is to add an option to set the alpha on the templates, to better reflect what they look like on the screen. So setting alpha = 0 would be an option there. No timeline on that, though.

Thanks!

[rschroll]

I missed the part about this failing even on the second (or later) attempt. That throws my idea about the wrong size confusing things out of the window. I'll try to reproduce on Firefox later. In the meantime,

• As a sanity check, can you see if the files read from rmfuse and copied into another directory are identical? Checking md5sums would be an easy check.
• Does the size of a file reported by rmfuse agree with the file size of the file copied to another directory?
• Can you try running master, and see if that does anything to fix things? I've made rmfuse a little less eager to throw ENODATA when someone reads beyond the end of a file.
• If none of that suggests anything, can you try running with -vv? This will spit out a whole load of debugging information. If you can post that here, it may help me figure out what's going wrong. (I don't think there would be any personal info in this output, if you prefer, you can email it to me at my Github handle at gmail.)

At the moment, my hypotheses are:

• RMfuse is messing up the output bytes somehow. (Thus the check of the md5sum.)
• Something is caching the filesize downstream of RMfuse, keeping it from being updated. (Thus the check of the file size.)
• These viewers are unhappy with a ENODATA error. (Thus the check of the new version.)
• These viewers are calling a filesystem operation that RMfuse doesn't implement. (The logs should show this.)
• These viewers are timing out waiting for the PDF to be rendered (and the cache of the most-recently-accessed file isn't working).

[f3fora]

Using the master branch:

As a sanity check, can you see if the files read from rmfuse and copied into another directory are identical? Checking md5sums would be an easy check.

Assuming that we are in the remarkable directory,

$ cp Test.pdf ..

That's cool because it doesn't show cp: error reading 'Test.pdf': No data available . I was already hopeful.

$ md5sum ../Test.pdf
140f8ccf433757c2207f83e55f1cd869  ../Test.pdf

$ md5sum Test.pdf
140f8ccf433757c2207f83e55f1cd869  Test.pdf

With the release version, instead, the last command does not produce any results.

However, after restarting rmfuse

md5sum Test.pdf
75c6f4732d90cd296c782c4b95a77906  Test.pdf

That actually doesn't mean anything, except that the rmfuse does not produce reproducible documents.

Does the size of a file reported by rmfuse agree with the file size of the file copied to another directory?

$ ls -lh ../Test.pdf
-r--r--r-- 1 user users 184K Feb 15 10:10 ../Test.pdf
$ ls -lh Test.pdf
-r--r--r-- 1 user users 184K Feb 15 10:01 Test.pdf

I don't know if these permissions are set on purpose. I think not having the writing permissions is a bit annoying, especially after copying in another directory.

However, I can't understand which is the right result:

$ ls -sh ../Test.pdf
184K ../Test.pdf
$ ls -sh Test.pdf
0 Test.pdf

Can you try running master, and see if that does anything to fix things? I've made rmfuse a little less eager to throw ENODATA when someone reads beyond the end of a file.

Now I can open document both with zathura-pdf-mupdf and firefox.

If none of that suggests anything, can you try running with -vv? This will spit out a whole load of debugging information. If you can post that here, it may help me figure out what's going wrong.

I just send you the log. i think it is useless. rmfuse.log

I run

$ poetry run rmfuse remarkable/ -vv

and stop it with ^C, at the end.

On another terminal

$ cd remarkable/
$ zathura Test.pdf
$ cd ..

These viewers are unhappy with a ENODATA error. (Thus the check of the new version.)

This was the issue. And I think it's solved. Thanks!

[rschroll]

Good to hear! If you run into similar problems in the future, even if they're transient, do let me know. The current fix is pretty naive, but I hope it's good enough. If not, we can be more careful.

Also, I think I've reproduced the bug in Firefox, so I can test it in the future.

Thanks for all the help debugging!