Exception when using boar on a win-sshfs drive if meta file data is >64MB

[GoogleCodeExporter]

I'm not sure how to reproduce this, but maybe the given error message is enough?

This just happened to me when I tried to update my documents repository... I 
ran this on my notebook from a different location than usually; the boar 
repository is hosted on a machine at my home and mounted using win-sshfs.

This is the output:

NOTICE: Repo is write protected - only read operations can be performed
Looking for files: 17230
Verifying checksum cache: 17230
Loading session...
Traceback (most recent call last):
  File "boar", line 1340, in <module>
  File "boar", line 1272, in main
  File "boar", line 673, in cmd_update
  File "workdir.pyc", line 233, in update
  File "workdir.pyc", line 547, in get_changes
  File "workdir.pyc", line 401, in get_bloblist
  File "front.pyc", line 322, in get_session_bloblist
  File "blobrepo\sessions.pyc", line 449, in get_all_blob_infos
  File "blobrepo\sessions.pyc", line 392, in get_raw_bloblist
  File "blobrepo\sessions.pyc", line 422, in __load_raw_bloblist
  File "common.pyc", line 75, in read_json
IOError: [Errno 22] Invalid argument

I'm not sure why the repo should be write-protected, but I guess that doesn't 
matter here.

Just tell me if there are any files that could help nailing down the problem...

Original issue reported on code.google.com by jannis.u...@stoppe.de on 19 Dec 2012 at 4:58

[GoogleCodeExporter]

When opening a repository, Boar attempts to create a temporary file to check if 
it is writable. If that fails, Boar assumes it is because the repository is 
read-only for some reason. It is possible that the write fails for some other 
reason and if so, the "errno 22" probably has something to do with it. What is 
the path to your repository? Is there any unusual characters in it?

Original comment by ekb...@gmail.com on 19 Dec 2012 at 8:20

[GoogleCodeExporter]

The path is "W:\boar family repository". This has worked several times before, 
on all kinds of computers, I really guess it's not that.

Original comment by jannis.u...@stoppe.de on 21 Dec 2012 at 3:21

[GoogleCodeExporter]

I just deleted the contents of the tmp folder, at least I got rid of the write 
protection message. Still get that error when I try to update though... Any 
idea how to fix this?

Original comment by jannis.u...@stoppe.de on 21 Dec 2012 at 4:05

[GoogleCodeExporter]

To be sure it's not the connection I tried this on WiFi, connected via cable 
and finally even using my phone's WiFi hotspot (to make sure it's not the 
router). All the same, on both update and co.

Could it be that it's some kind of timeout? I'm dealing with a large repository 
here and I'm on a rather slow connection (it's a DSL of some sort, could be 2 
Mbit/sec downstream, 192 kbit/sec upstream)

Original comment by jannis.u...@stoppe.de on 21 Dec 2012 at 11:23

[GoogleCodeExporter]

I'm thinking of two likely causes. Windows will give error 22 if a program 
tries to open an invalid path, that is a path containing illegal characters 
like tab or newline. That could be caused by a bug in Boar or perhaps the 
interaction between the command shell and boar. Could you please try to run 
exactly this command at an ordinary cmd.exe command line and append the result 
here:

  boar --repo="\\?\W:\boar family repository" verify -q

Secondly, there may be an issue with sshfs itself. We'll try that next 
depending on the output of the command above.

Original comment by ekb...@gmail.com on 22 Dec 2012 at 11:48

[GoogleCodeExporter]

Running on the local machine that connects via ssh:

Verifying 30 snapshots
Snapshot 1 (d41d8cd98f00b204e9800998ecf8427e): All 0 blobs ok
Snapshot 2 (d41d8cd98f00b204e9800998ecf8427e): All 0 blobs ok
Traceback (most recent call last):
  File "boar", line 1340, in <module>
  File "boar", line 1256, in main
  File "boar", line 525, in cmd_verify
  File "boar", line 179, in verify_repo
  File "front.pyc", line 322, in get_session_bloblist
  File "blobrepo\sessions.pyc", line 449, in get_all_blob_infos
  File "blobrepo\sessions.pyc", line 392, in get_raw_bloblist
  File "blobrepo\sessions.pyc", line 422, in __load_raw_bloblist
  File "common.pyc", line 75, in read_json
IOError: [Errno 22] Invalid argument

When running on the machine that houses the repository (using ssh, running on 
the remote shell):

D:\Share\Pictures>boar --repo="D:\Share\boar family repository" verify -q
Verifying 30 snapshots
Snapshot 1 (d41d8cd98f00b204e9800998ecf8427e): All 0 blobs ok
Snapshot 2 (d41d8cd98f00b204e9800998ecf8427e): All 0 blobs ok
Snapshot 3 (72ef6fd5d651439e0e3bb35a47a4568b): All 20383 blobs ok
Snapshot 4 (fe073e7bceb130a5a8e148876e5f948b): All 41026 blobs ok
Snapshot 5 (c2551ea8f5a3964dc62f2caf3d30195c): All 20612 blobs ok
Snapshot 6 (764bc4fa951f870afda218bcd19650f4): All 20607 blobs ok
Snapshot 7 (09d75804c64196bac8c9e5df19871336): All 17044 blobs ok
Snapshot 8 (1bb512b2e7c43e384f91c159e0c2de77): All 17023 blobs ok
Snapshot 9 (ecfed5a53c9c27b57867962e80ee2848): All 41022 blobs ok
Snapshot 10 (14b71ba54393e4783355d7b06cf7d89a): All 39327 blobs ok
Snapshot 11 (edf16d7af8b94a6d2a06bbcc07cf959d): All 39298 blobs ok
Snapshot 12 (d73b86b0b2b61eb5569f7baaf8d02638): All 16967 blobs ok
Snapshot 13 (a30d63c879c2ddb40e2ddce2f1116819): All 16920 blobs ok
Snapshot 14 (010f559bdd40402cfd01f46e993be2dd): All 39275 blobs ok
Snapshot 15 (4749a2406e0e1147249df8ee74e8e6ed): All 39057 blobs ok
Snapshot 16 (d41d8cd98f00b204e9800998ecf8427e): All 0 blobs ok
Snapshot 17 (8595e43c26dc3f0fb38374f86e94f17c): All 16925 blobs ok
Snapshot 18 (1773cc2264b5aeb0f4d5a953166bb303): All 5746 blobs ok
Snapshot 19 (e0e7ebb33d8f79a164d400bdb2dd8438): All 39187 blobs ok
Snapshot 20 (b960f45f33713fefcf7f19d773f671b5): All 17231 blobs ok
Snapshot 21 (0f56e0bde80635339fbf94e390043033): All 17233 blobs ok
Snapshot 22 (25ac08fdcb5fedf0dab8082bef7222fc): All 17233 blobs ok
Snapshot 23 (d09f51a2a62d172584b27c37d06153ba): All 17234 blobs ok
Snapshot 24 (565346065409bec097ded9faa63a4c21): All 30522 blobs ok
Snapshot 25 (abaa6c0c7a376e69b2b8edabcf3ce9d5): All 30379 blobs ok
Snapshot 26 (d41d8cd98f00b204e9800998ecf8427e): All 0 blobs ok
Snapshot 27 (fa22a4ddb26dc1f7d931ec2aa5ca4827): All 6 blobs ok
Snapshot 28 (363e7a87f635d18df9709c02917c5aec): All 17242 blobs ok
Snapshot 29 (d41d8cd98f00b204e9800998ecf8427e): All 0 blobs ok
Snapshot 30 (735cd4fd9311790b8f0ee865d3d92b5c): All 17230 blobs ok
Skipping blob verification
Finished in 118.08 seconds

Running the former took hours (literally).

Original comment by jannis.u...@stoppe.de on 22 Dec 2012 at 9:36

[GoogleCodeExporter]

(Sorry for the delay. Holidays, you know)

Looking at the exception in your previous comment, I'm doubtful that any bug in 
Boar could cause these symptoms. I realize it may take a long time, but it'd be 
very useful to check if any file is unreadable through ssh-fs for any reason. 
One simple way to do that is to try to make a full copy of the repository 
(without using Boar) This command should do the trick:

  xcopy /E "W:\boar family repository" "C:\repo copy"

If it succeeds, follow up with a "boar verify" on the copy to make sure it is 
actually correct. (Make sure you have enough room on c: You can delete the copy 
immediately afterwards of course.)

Original comment by ekb...@gmail.com on 3 Jan 2013 at 7:35

[GoogleCodeExporter]

Sorry for the delay, too... holidays, you know ;)

I was able to update again when I got back home. Updating locally wasn't a 
problem... Even when I got back to work, updating the computer there wasn't a 
problem, too. Apart from a really slow connection and old hardware at my 
parents' house, I don't know what might have caused this.

Original comment by jannis.u...@stoppe.de on 16 Jan 2013 at 12:25

[GoogleCodeExporter]

Okay, I got this problem again. To the point where I haven't been able to 
checkout/update my pictures folder on a remote PC. I looked into it and it 
seems to be a problem with Windows that somehow can't read chunks bigger than 
64 MB from networked folders or something. The issue is known and outlined 
here: http://support.microsoft.com/default.aspx?scid=kb;en-us;899149 . I tried 
to apply the solution, the suggested fix of using w+b when opening files didn't 
work though. What *did* work was reading the json files in smaller chunks:

diff -r 87c96c575374 common.py
--- a/common.py Sun Sep 22 21:56:41 2013 +0200
+++ b/common.py Fri Sep 27 16:11:28 2013 +0200
@@ -72,7 +72,17 @@

 def read_json(filename):
     with safe_open(filename, "rb") as f:
-        return json.loads(f.read())
+        # This workaround is for Windows networked files that do have a problem
+        # when getting too big. Large bloblists can become several GB, crashing
+        # if the read process isn't split up.
+        # 8192 is of course arbitrary. But it needs to be something << 64 MB
+        fullResult = ''
+        tempResult = f.read(8192)
+        while tempResult != '':
+            fullResult += tempResult
+            tempResult = f.read(8192)
+            
+        return json.loads(fullResult)

 """ This file contains code that is generally useful, without being
 specific for any project """

Original comment by jannis.u...@stoppe.de on 27 Sep 2013 at 2:16

[GoogleCodeExporter]

To avoid copypasting hassle, there's a patch...

Original comment by jannis.u...@stoppe.de on 27 Sep 2013 at 5:43

Attachments:

• boar_1186.patch

[GoogleCodeExporter]

Windows never seizes to amaze me... Thanks for looking into this! But for some 
reason I cannot replicate this problem on my windows 7 64-bit installation. 
What version are you using?

Original comment by ekb...@gmail.com on 28 Sep 2013 at 6:27

[GoogleCodeExporter]

Windows 7 Pro 64, connected via win-sshfs to a server running Windows 7 Pro x86 
and winsshd.

Original comment by jannis.u...@stoppe.de on 28 Sep 2013 at 6:24

[GoogleCodeExporter]

Ah yes, there we go. Ordinary network drives work fine, but win-sshfs mounts 
suffers from the problem you describe. Likely the same problem will show up for 
other types of application backed mounts.

Original comment by ekb...@gmail.com on 29 Sep 2013 at 8:54

• Changed title: Exception when using boar on a win-sshfs drive if meta file data is >64MB
• Changed state: Accepted