search

reHackable / scripts

[ THIS PROJECT IS CURRENTLY ON HALT :( ] A set of bash scripts that may enhance your reMarkable experience
GNU General Public License v3.0
236 stars 30 forks source link

repush.sh: Filetransfer broken due to new naming scheme #31

Open cherti opened 3 years ago

cherti commented 3 years ago

With version 2.5.0.27, repush.sh is no longer able to transfer files onto the remarkable. The reason for this is a change in the setting of teh visibleName in the metadata field. While before this was the filename including the extension, now the metadata is generated with the filename without the extension in the visibleName-field. This causes repush.sh to hang indefinitely as internal checks fail.

The fix for this has to be applied in two locations: This line has to be changed to placeholder_basename="$(basename "$1" .$extension)", as $extension is available from a couple of lines above.

The second location is this, in principle the same fix applies, however, $extension is not set. A quick test of mine using "$(basename "$f" | cut -f1 -d '.')" works like a charm, however, this will fail the moment the filename contains another . besides the one separating the extension. As I have just briefly investigated the problem, I am not sure what's the cleanest fix for this location, hence an issue describing the fix instead of pull requesting a fix. The hack with cut, while working, is certainly not the optimal solution.

Addendum: Due to the handling of directories (dropping a placeholder pdf and turning that into a directory), the aforementioned fix is likely not that straightforward. I currently settled for using basename twice at that very location to first remove a .pdf-extension and then removing a .epub-extension, which works well enough for me. However, this is likely breaking directory pushing, so it likely needs to be implemented a little more carefully, with consideration for the directory pushing implementation.

mb64 commented 3 years ago

These fixes worked for me. The common Bash way to get the extension would make it "$(basename "$f" "${f##*.}")". (There's probably a way to simplify it even further, removing the basename, but I don't know it.)

Another issue is that the thumbnails did not show up for me -- they were all blank. This was quick to fix by deleting the thumbnails that were already there (.local/share/remarkable/xochitl/<uuid>.thumbnails/*.jpg) and running systemctl restart xochitl.service. My hypothesis is that when the thumbnails are rendered may have changed to be earlier, rendering the blank placeholder PDF by accident.

mb64 commented 3 years ago

These changes aren't quite sufficient when the basename has a dot in it. For a file this.pdf, using a placeholder_basename of either this or this.pdf makes an entry called this. But for a file this.that.pdf, using this.that.pdf results in an entry this.that and using this.that results in an entry this.

There's also some wackiness with EPUBs; I'm not exactly sure why.

trou commented 3 years ago

I've included your fixes (among others) in this branch: https://github.com/trou/scripts/tree/remarkable_2

I'll do a PR if I get to work around the dots problem.

ghalfacree commented 3 years ago

@trou I've run into this problem, and I'm trying to use your modified repush.sh - but it seems to be hitting an infinite loop.

+ VERSION=4.1
+ SSH_ADDRESS=10.11.99.1
+ WEBUI_ADDRESS=10.11.99.1:80
+ PORT=9000
+ shopt -s nullglob
+ getopts :vhdr:p:o: opt
+ shift 0
+ '[' -z '/media/RAM Disk/edge/Edge 001.pdf' ']'
+ for f in "$@"
+ check_file '/media/RAM Disk/edge/Edge 001.pdf'
++ file -F '|' '/media/RAM Disk/edge/Edge 001.pdf'
+ file_cmd_output='/media/RAM Disk/edge/Edge 001.pdf| PDF document, version 1.3'
+ echo '/media/RAM Disk/edge/Edge 001.pdf| PDF document, version 1.3'
+ grep -q '| directory'
+ '[' '!' -e '/media/RAM Disk/edge/Edge 001.pdf' ']'
+ '[' -z ']'
+ echo '/media/RAM Disk/edge/Edge 001.pdf| PDF document, version 1.3'
+ grep -q '| PDF'
+ '[' -z ']'
+ echo '/media/RAM Disk/edge/Edge 001.pdf'
+ grep -qi '\.pdf$'
+ echo '/media/RAM Disk/edge/Edge 001.pdf'
+ grep -q '"'
+ return 1
+ '[' 1 -eq 0 ']'
+ '[' '' ']'
+ ssh -o ConnectTimeout=1 -M -S remarkable-ssh -q -f root@10.11.99.1 -N
+ SSH_RET=0
+ '[' 0 -ne 0 ']'
+ for f in "$@"
++ basename '/media/RAM Disk/edge/Edge 001.pdf'
+ uuid_of_root_file 'Edge 001.pdf'
+ RET_UUID=
+ rmtgrep lF '"visibleName": "Edge 001.pdf"' '~/.local/share/remarkable/xochitl/*.metadata'
+ escaped_regex='\"visibleName\": \"Edge 001.pdf\"'
++ ssh -S remarkable-ssh root@10.11.99.1 'grep -lF "\"visibleName\": \"Edge 001.pdf\"" ~/.local/share/remarkable/xochitl/*.metadata'
+ RET_MATCH=
+ matches_by_name=
+ '[' -z '' ']'
+ return
+ '[' '!' -z ']'
+ rm -rf /tmp/repush
+ mkdir -p /tmp/repush
+ OUTPUT_UUID=
+ '[' '' ']'
+ TOTAL=0
+ SUCCESS=0
+ for f in "$@"
+ ROOT_UUID=
+ push '/media/RAM Disk/edge/Edge 001.pdf'
+ (( TOTAL++ ))
++ file -F '|' '/media/RAM Disk/edge/Edge 001.pdf'
+ file_cmd_output='/media/RAM Disk/edge/Edge 001.pdf| PDF document, version 1.3'
+ echo '/media/RAM Disk/edge/Edge 001.pdf| PDF document, version 1.3'
+ grep -q '| \(PDF\|directory\)'
+ extension=pdf
+ echo '/media/RAM Disk/edge/Edge 001.pdf| PDF document, version 1.3'
+ grep -q '| directory'
+ directory=
++ basename '/media/RAM Disk/edge/Edge 001.pdf'
+ placeholder_basename='Edge 001.pdf'
+ '[' ']'
+ placeholder='/tmp/repush/Edge 001.pdf'
+ [[ pdf == \p\d\f ]]
+ create_placeholder_pdf '/tmp/repush/Edge 001.pdf'
+ echo %PDF-1.
+ echo '1 0 obj<</Pages 2 0 R>>endobj'
+ echo '2 0 obj<</Kids[3 0 R]/Count 1>>endobj'
+ echo '3 0 obj<</Parent 2 0 R>>endobj'
+ echo -n 'trailer <</Root 1 0 R>>'
+ true
+ curl --connect-timeout 2 --silent --output /dev/null --form 'file=@"/tmp/repush/Edge 001.pdf"' http://10.11.99.1:80/upload
+ true
+ uuid_of_root_file 'Edge 001.pdf'
+ RET_UUID=
+ rmtgrep lF '"visibleName": "Edge 001.pdf"' '~/.local/share/remarkable/xochitl/*.metadata'
+ escaped_regex='\"visibleName\": \"Edge 001.pdf\"'
++ ssh -S remarkable-ssh root@10.11.99.1 'grep -lF "\"visibleName\": \"Edge 001.pdf\"" ~/.local/share/remarkable/xochitl/*.metadata'
+ RET_MATCH=
+ matches_by_name=
+ '[' -z '' ']'
+ return
+ '[' '!' -z '' ']'
+ true
+ uuid_of_root_file 'Edge 001.pdf'
+ RET_UUID=
+ rmtgrep lF '"visibleName": "Edge 001.pdf"' '~/.local/share/remarkable/xochitl/*.metadata'
+ escaped_regex='\"visibleName\": \"Edge 001.pdf\"'
++ ssh -S remarkable-ssh root@10.11.99.1 'grep -lF "\"visibleName\": \"Edge 001.pdf\"" ~/.local/share/remarkable/xochitl/*.metadata'
+ RET_MATCH=
+ matches_by_name=
+ '[' -z '' ']'
+ return
+ '[' '!' -z '' ']'

That last chunk - from +true to + '[' '!' -z '' ']' just repeats ad infinitum, filling over 20,000 lines in the log file in a matter of seconds before I hit Ctrl-C, and the file is never transferred - though an empty skeleton file appears on the Remarkable.

Clues welcomed, 'cos I have no idea what's broken here!

I'm running the script on an Ubuntu 20.04 64-bit installation, standard Bash shell, and connecting to a Remarkable 2 running 2.5.0.27.

trou commented 3 years ago

I had the same bug, which was related to the "." problem @mb64 mentioned. I updated my branch with a fix. Can you check if it works ?

ghalfacree commented 3 years ago

I had the same bug, which was related to the "." problem @mb64 mentioned. I updated my branch with a fix. Can you check if it works ?

Seems to be working here - the file transferred correctly, no infinite loop. Many thanks!

ghalfacree commented 3 years ago

Tell a lie, there's a new glitch: the file transferred correctly, but there's also an empty skeleton file with the same name on there. Tried it again with a second PDF, same thing happened: there's a working copy of the file, and an empty skeleton file which just opens up to a blank "Page 1 of 1".