Closed ross-spencer closed 8 years ago
richardlehane
commented
8 years ago Hi Ross Yuck! Sorry about this!
if you use -log p you'll log the progress of the scan, which will indicate the last file that runs.
You can keep error reporting too by adding that i.e. -log p,e.
It looks like the windows long filepath code (added for your issue with long file paths on win) may be causing this.
richardlehane
commented
8 years ago May not be necessary to track this down to a file Ross: it looks like I've made a recursion bomb, causing infinite loop. Reading the code is giving me a bit of a headache but should be able to puzzle out from the code alone over weekend.
ross-spencer
commented
8 years ago Noted :) Thanks Richard.
The flag I was going to ask for anyway, it is useful for the other bits i'm testing. I look forward to a later version for this. No rush though!
On Fri, Jan 15, 2016 at 5:36 PM, Richard Lehane notifications@github.com wrote:
May not be necessary to track this down to a file Ross: it looks like I've made a recursion bomb, causing infinite loop. Reading the code is giving me a bit of a headache but should be able to puzzle out from the code alone over weekend.
— Reply to this email directly or view it on GitHub https://github.com/richardlehane/siegfried/issues/66#issuecomment-171869751 .
richardlehane
commented
8 years ago cool. You may find the doc of logging helpful: https://github.com/richardlehane/siegfried/wiki/Identifying-file-formats#logging
richardlehane
commented
8 years ago Hi Ross this change now implemented on the develop branch.
No perfect fix here unfortunately. My first fix for your long path error on Win was to try to fix for both files and directories. If it was a directory, it was a lot more complicated: had to try to save old path, then restart recursion with a long form path, then rebuild old paths for all files and dirs within that directory. This I think caused infinite recursion (gave me a headache just trying to understand that old code).
The new fix just retries long file paths for files only. If the long path issue affects a directory, scanning should now cease with a fatal error (figured this better than silently skipping directories during scan). Although not ideal, this fix is a lot simpler & shouldn't cause further problems (I hope!).
If you can check the develop branch out and build and test, much appreciated!
ross-spencer
commented
8 years ago Hi Richard,
Sounds like a tough one. I'm not sure how Java is handling it in DROID.
Will Siegfried continue onto a sub-directory or other item in the file/dir list at all?
Will test from the develop branch over this week, need the source files to work with.
Thx,
Ross
On 17/01/2016, at 19:26, Richard Lehane notifications@github.com wrote:
Hi Ross this change now implemented on the develop branch.
No perfect fix here unfortunately. My first fix for your long path error on Win was to try to fix for both files and directories. If it was a directory, it was a lot more complicated: had to try to save old path, then restart recursion with a UNC path, then rebuild old paths for all files and dirs within that directory. This I think caused infinite recursion (gave me a headache just trying to understand that old code).
The new fix just retries long file paths for files only. If the long path issue affects a directory, scanning should now cease with a fatal error (figured this better than silently skipping directories during scan). Although not ideal, this fix is a lot simpler & shouldn't cause further problems (I hope!).
If you can check the develop branch out and build and test, much appreciated!
— Reply to this email directly or view it on GitHub.
richardlehane
commented
8 years ago DROID should be unaffected - this is a golang issue (https://github.com/golang/go/issues/3358). Unfortunately unlikely to be fixed soon as the underlying problem is with the windows API.
Now the behaviour is: if there is an OS error, sf will attempt to retry with a long form path if the path is a file. Scanning will continue (and into sub-dirs) if the path is a file, regardless of error (either identifying or reporting the error). If it is a directory, scanning will halt immediately with error. This allows the user to pick up the scan from that point (e.g. by mapping the directory to a shorter path) and prevents silent skipping of directories.
In practice, I suspect that this should be fine as you'd expect most of the long paths to be files anyway (otherwise you'd have really long dirs, with really short file names - as there is still that hard limit anyway and we are only talking paths between 255 and 260 chars).
ross-spencer
commented
8 years ago Thanks Richard, I should have re-phrased as to "I wonder what Java's 'trick' is to do the same thing" - aware it is a Golang issue.
I'll have to do some research myself. Using Siegfried is a good option, but any risk of a scan stopping is a tough one to deal with.
While I am trying to generate an entire collection overview in a single report, it is conceivable I could pick up a scan from a halt point, but do we have a guarantee as to how the directories have been listed in memory and are being accessed by the language? - we don't make the same assumption of ordering in Java for example. And would SF best practice to be always run in -log p,stderr mode?
Given enough content too, right now I'm working with 100k+ files and I don't know at what frequency it might stop.
Ideally, behaviour as robust as DROID in not dropping paths from a report is what I guess I'm looking for, and am will have a look at workarounds to implement in Golang too.
The paths incidentally, when set up, were set up using bad digital records management practice, BUT, they are the paths we have to deal with and changing them is not something I can do in practice.
I appreciate the help. Will test as mentioned before.
On Sun, Jan 17, 2016 at 7:54 PM, Richard Lehane notifications@github.com wrote:
DROID should be unaffected - this is a golang issue (golang/go#3358 https://github.com/golang/go/issues/3358). Unfortunately unlikely to be fixed soon as the underlying problem is with the windows API.
Now the behaviour is: if there is an OS error, sf will attempt to retry with a UNC path if the path is a file. Scanning will continue (and into sub-dirs) if the path is a file, regardless of error (either identifying or reporting the error). If it is a directory, scanning will halt immediately with error. This allows the user to pick up the scan from that point (e.g. by mapping the directory to a shorter path) and prevents silent skipping of directories.
In practice, I suspect that this should be fine as you'd expect most of the long paths to be files anyway (otherwise you'd have really long dirs, with really short file names - as there is still that hard limit anyway and we are only talking paths between 255 and 260 chars).
— Reply to this email directly or view it on GitHub https://github.com/richardlehane/siegfried/issues/66#issuecomment-172296297 .
richardlehane
commented
8 years ago Hi Ross As I understand it, other langs have kind of hacked around this problem with the win API - it apparently is a messy area! I'm hesitant about going further with applying a fix to sf here simply because it is so messy and a complex solution like I tried at first may in practice do more harm than good.
Golang's file walk is lexical / deterministic (https://golang.org/pkg/path/filepath/#WalkFunc) & restarting on error guarantees you won't miss any results.
If you get hit with a lot of long directory name errors, you could also try using a long path in the initial command you give sf. I.e. give sf an argument in the form \\?\ + absolute dir. e.g. sf \\?\c:\path\to\files. This should avoid the long path issue altogether (though will mean you get these long form paths in your results).
Re. logging: use sf -log p,e [or long version sf -log progress,error], don't use stderr. stderr just controls the destination for the logging (stderr or stdout) and is the default. sf -log p,stderr logs progress to stderr but it doesn't log error messages! Sorry if confusing. For big scans I do tend to use sf -log p as I like to know what is going on. You might actually like to use sf -log p,e,w,t (this logs progress, errors, warning messages and gives you the time elapsed for the scan). I like to leave this to the user to choose as for short scans it is cleaner just to see if there are any errors.
ross-spencer
commented
8 years ago Hi Richard, as you know the codebase better, have you managed to wrap any of this up into unit tests? As noted above, I'm interested in testing this further outside of SF but anything I can borrow from the SF unit tests can help me drive that development/testing in a way that might be useful to SF in future.
Thanks, Ross
richardlehane
commented
8 years ago Hi Ross very happy for you to take this one further!
Unfortunately, the sf cmd tests (https://github.com/richardlehane/siegfried/blob/develop/cmd/sf/sf_test.go) are more end-to-end than unit.
Fortunately, the code you are interested in is fairly small and is contained in just two funcs: multiIdentifyS() and identifyFile(), found here https://github.com/richardlehane/siegfried/blob/develop/cmd/sf/sf.go#L143
The Win long path fix resides in this file: https://github.com/richardlehane/siegfried/blob/develop/cmd/sf/longpath_windows.go (this file is only built for windows version of sf, hence the _windows.go - other platforms just get empty funcs in longpath.go)
Let me know how you get on Rich
richardlehane
commented
8 years ago OK I've tried one last time with this fix :)
https://github.com/richardlehane/siegfried/commit/f73dc100f8ecd53b73b7150fd42c5e403d601b22
Now restarts the recursion for long directories (and restores short path names after long fix applied). Code cleaner than the original version of that fix, but still a recursive design so a little scary.
Appreciate if you can test! cheers Richard
ross-spencer
commented
8 years ago This one still seems to be happening, I'm wondering if the cause is just a memory leak/bad garbage collection, or something around that... the paths involved do not look long, per se:
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AI Release Budget 2003\03-05 - training - data - gateway senior school numbers.xls
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AI Release Budget 2003\Youth training Auc City.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AK03 Congrats 6Oct03oc.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld Employment Stats 11Nov03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld Tourism Stats 12Feb03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld crime stats 14Aug03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld teen parent unit pr15Dec03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld transport a team effort 5Aug03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld-Govt Mayoral Forum pr8Oct03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AkldBudget200315May03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AkldMuseumBill passed pr3Sept03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Auckland unemployment fall a welcome signal.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Backup of Auckland unemployment fall a welcome signal.wbk
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Backup of Govt takes commonsense approach to tertiary fees.wbk
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Blues Trains congratulated 23May03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Coddington desperate pr4Nov03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Economic buoyancy in Akld 12Dec03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Get to the game early 15Aug03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Good year for Akld-Govt 19Dec03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Govt takes commonsense approach to tertiary fees.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Health support for Older Aklers 1Oct03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\New Mayoral Forum chair pr6Oct03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Tizard on Akld Transport & LTMB 17Jan03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Tizard on Westhaven sale 20Aug03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Waiheke Heartland service 5Feb03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\akld PHO pr1Oct03.doc
[ERROR] winapi error #4
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\billpassingMuseumrelease.doc
[ERROR] winapi error #4
fatal error: runtime: cannot allocate memory
runtime stack:
runtime.throw(0x7e9300, 0x1f)
c:/go/src/runtime/panic.go:527 +0x7f
runtime.persistentalloc1(0x100, 0x40, 0x9a3e78, 0xccff00)
c:/go/src/runtime/malloc.go:878 +0x253
runtime.persistentalloc.func1()
c:/go/src/runtime/malloc.go:831 +0x35
runtime.systemstack(0x6378fbcc)
c:/go/src/runtime/asm_386.s:283 +0x81
runtime.persistentalloc(0x100, 0x40, 0x9a3e78, 0xccff00)
c:/go/src/runtime/malloc.go:832 +0x4e
runtime.getempty(0x4cf628a, 0x0)
c:/go/src/runtime/mgcwork.go:292 +0x87
runtime.getpartialorempty(0x2a, 0x123fb6ba)
c:/go/src/runtime/mgcwork.go:332 +0x60
runtime.(*gcWork).put(0x127a4928, 0x160d9450)
c:/go/src/runtime/mgcwork.go:83 +0x34
runtime.greyobject(0x160d9450, 0x16ddfbac, 0xc, 0x123fb6ba, 0x0, 0xcb325c, 0x127a4928)
c:/go/src/runtime/mgcmark.go:941 +0x295
runtime.scanblock(0x16ddfbac, 0x1c, 0x83027c, 0x127a4928)
c:/go/src/runtime/mgcmark.go:798 +0x108
runtime.scanframeworker(0x6378fda4, 0x0, 0x127a4928)
c:/go/src/runtime/mgcmark.go:485 +0x162
runtime.scanstack.func1(0x6378fda4, 0x0, 0x1)
c:/go/src/runtime/mgcmark.go:384 +0x57
runtime.gentraceback(0x4161dd, 0x16b9b1ac, 0x0, 0x127a8000, 0x0, 0x0, 0x7fffffff, 0x6378fe30, 0x0, 0x0, ...)
c:/go/src/runtime/traceback.go:336 +0x8af
runtime.scanstack(0x127a8000)
c:/go/src/runtime/mgcmark.go:406 +0x305
runtime.scang(0x127a8000)
c:/go/src/runtime/proc1.go:422 +0xd6
runtime.markroot(0x12794190, 0x5)
c:/go/src/runtime/mgcmark.go:133 +0x164
runtime.parfordo(0x12794190)
c:/go/src/runtime/parfor.go:95 +0x10e
runtime.gcscan_m()
c:/go/src/runtime/mgcmark.go:36 +0xab
runtime.gc.func1()
c:/go/src/runtime/mgc.go:996 +0x107
runtime.systemstack(0x127a5400)
c:/go/src/runtime/asm_386.s:267 +0x57
runtime.mstart()
c:/go/src/runtime/proc1.go:674
goroutine 5 [garbage collection scan]:
runtime.systemstack_switch()
c:/go/src/runtime/asm_386.s:222 fp=0x127b1d3c sp=0x127b1d38
runtime.gc(0x0)
c:/go/src/runtime/mgc.go:1005 +0x229 fp=0x127b1fc0 sp=0x127b1d3c
runtime.backgroundgc()
c:/go/src/runtime/mgc.go:896 +0x41 fp=0x127b1fd8 sp=0x127b1fc0
runtime.goexit()
c:/go/src/runtime/asm_386.s:1662 +0x1 fp=0x127b1fdc sp=0x127b1fd8
created by runtime.startGC
c:/go/src/runtime/mgc.go:869 +0x15e
goroutine 1 [syscall, locked to thread]:
syscall.Syscall(0x76b943d5, 0x2, 0x164a0000, 0x16b9b214, 0x0, 0x2, 0x3, 0x3)
c:/go/src/runtime/syscall_windows.go:128 +0x4a
syscall.findFirstFile1(0x164a0000, 0x16b9b214, 0x965168, 0x0, 0x0)
c:/go/src/syscall/zsyscall_windows.go:349 +0x5f
syscall.FindFirstFile(0x164a0000, 0x164a2000, 0x164a0000, 0x0, 0x0)
c:/go/src/syscall/syscall_windows.go:922 +0x59
os.openDir(0x14771e30, 0x63, 0x0, 0x0, 0x0)
c:/go/src/os/file_windows.go:98 +0xe7
os.OpenFile(0x14771e30, 0x63, 0x0, 0x0, 0x1, 0x0, 0x0)
c:/go/src/os/file_windows.go:145 +0x19f
os.Open(0x14771e30, 0x63, 0x441bb2, 0x0, 0x0)
c:/go/src/os/file.go:246 +0x43
path/filepath.readDirNames(0x14771e30, 0x63, 0x0, 0x0, 0x0, 0x0, 0x0)
c:/go/src/path/filepath/path.go:402 +0x45
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16485e50, 0x16b9b674, 0x0, 0x0)
c:/go/src/path/filepath/path.go:361 +0x15c
path/filepath.Walk(0x14771e30, 0x63, 0x16b9b674, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16485e00, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16485cc0, 0x16b9b834, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9b834, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16485c70, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16485b30, 0x16b9b9f4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9b9f4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16485ae0, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x164859a0, 0x16b9bbb4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9bbb4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16485950, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16485810, 0x16b9bd74, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9bd74, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x164857c0, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16485680, 0x16b9bf34, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9bf34, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16485630, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x164854f0, 0x16b9c0f4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9c0f4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x164854a0, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16485360, 0x16b9c2b4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9c2b4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16485310, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x164851d0, 0x16b9c474, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9c474, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16485180, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16485040, 0x16b9c634, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9c634, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484ff0, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16484eb0, 0x16b9c7f4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9c7f4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484e60, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16484d20, 0x16b9c9b4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9c9b4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484cd0, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16484b90, 0x16b9cb74, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9cb74, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484b40, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16484a00, 0x16b9cd34, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9cd34, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x164849b0, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16484870, 0x16b9cef4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9cef4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484820, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x164846e0, 0x16b9d0b4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9d0b4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484690, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16484550, 0x16b9d274, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9d274, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484500, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x164843c0, 0x16b9d434, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9d434, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484370, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16484230, 0x16b9d5f4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9d5f4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x164841e0, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x164840a0, 0x16b9d7b4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9d7b4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16484050, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16465ef0, 0x16b9d974, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9d974, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16465ea0, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16465d60, 0x16b9db34, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9db34, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16465d10, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36b
path/filepath.walk(0x14771e30, 0x63, 0xc94590, 0x16465bd0, 0x16b9dcf4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:363 +0x1c7
path/filepath.Walk(0x14771e30, 0x63, 0x16b9dcf4, 0x0, 0x0)
c:/go/src/path/filepath/path.go:396 +0xb7
main.multiIdentifyS(0x330801c8, 0x12c3e600, 0x12c38050, 0x14771e30, 0x63, 0x14771e30, 0x63, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:176 +0x84
main.multiIdentifyS.func1(0x14771e30, 0x63, 0xc94590, 0x16465b80, 0x0, 0x0, 0x0, 0x0)
C:/Source/git/go/src/github.com/richardlehane/siegfried/cmd/sf/sf.go:161 +0x36bHi Ross this is the longpath stuff: all the memory is getting soaked up by repeated calls of the multiIdentifyS function (at line 161), which was the latest fix I tried.
The longpath fix is pretty blunt: any error is treated as a possible long path error & results in a retry (so it might be worth type asserting the error and only catching that particular error). I.e. you might have some network issue causing a file to be inaccessible and it will still be treated as a potential long path error (on the principle that it doesn't hurt to retry). The problem is that when retrying on a directory the file walk has to restart and at present it seems to be retrying infinitely.
richardlehane
commented
8 years ago Hi Ross Still not sure what is going on here exactly. Can you see if this bug always affects the directory immediately following [FILE] \?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\billpassingMuseumrelease.doc ??
I've just pushed a fix that tidies a couple of things up: it will only retry once now with a long file path (if the file path is already long it will not retry) & now use Lstat instead of Stat (to be consistent with the filepath.Walk func). Lstat vs Stat may have caused the issue if the problematic directory is a symbolic link.
Sorry about these bugs and the scattergun approach to fixing! If you can pin it down to a particular file or dir I can add a test.
cheers rich
ross-spencer
commented
8 years ago Hi Richard,
Using the develop branch before the previous commit described above - If i just put in:
sf -droid -hash=md5 "Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003"
The entire containing folder for that collection of files, SF returns just fine, which is what made me wonder about it just being a plain memory/garbage collection error (or something else not long paths). I can rebuild and try that, but will have to start the scan from scratch to see if it recreates at the point it gets here...
What's your thinking?
On Tue, Jan 19, 2016 at 1:41 PM, Richard Lehane notifications@github.com wrote:
Hi Ross Still not sure what is going on here exactly. Can you see if this bug always affects the directory immediately following [FILE] \?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\billpassingMuseumrelease.doc ??
I've just pushed a fix that tidies a couple of things up: it will only retry once now with a long file path (if the file path is already long it will not retry) & now use Lstat instead of Stat (to be consistent with the filepath.Walk func). Lstat vs Stat may have caused the issue if the problematic directory is a symbolic link.
Sorry about these bugs and the scattergun approach to fixing! If you can pin it down to a particular file or dir I can add a test.
cheers rich
— Reply to this email directly or view it on GitHub https://github.com/richardlehane/siegfried/issues/66#issuecomment-172693156 .
ross-spencer
commented
8 years ago So, to clarify, the version of develop I am using contains that status update, and was the one that caused the last crash. What's curious, is the long path syntax (\?) doesn't get put into the output. It only happens if I run this scan from scratch:
C:\Working\sfrogues>sf -droid -hash=md5 -log p,e "Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003" > test.txt
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\2003 - Advisories\Alexandra Advisory 17April03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\2003 - Advisories\SmartGrowthday 8Mayl03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\2003 - Advisories\TopArt Advisory 3Junel03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\2003 - Advisories\VehicleEmissions Advisory 30Sept03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\2003 - Advisories\VehicleEmissionsLaunch Reminder 1Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\ArtsBudget14May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\CNZ Appointments pr5Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Chch arts survey response 6Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\ClarkScholarship 13Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Coddingtons facts wrong 24Oc03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Confident Arts Voice pr9Sept03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Cultural tourism funding MBJT pr10April03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Cultural wellbeing pr20Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Dylan Taite 23Jan03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Edwin Carr 27Mar03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Flight of the Conchord - Edinburgh Festival.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\GATS cultural issues letter draft.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Hirini Melbourne 8 Jan 03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\LOTR Baftas 24Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\LOTR OscarNom 12Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ Music Month Tizard 1May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ Music Month Tizard 2May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ Music Quiz ANSWERS 2003.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ Music Quiz Questions 2003.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ at Midem 30Jan03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ music Xmas pr14Dec03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ music a hit in New Yorkl.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ music to Nth America 3March03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZ music worldwide pr19May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZFC Appointments pr5Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZMusic Quiz results 8May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\NZSO Board appointments pr4Sept03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\National Band Members recognised 24Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\New Zealand Music Code 2003 results (070803).doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\PACE statistics pr 18June03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Rip It Up Top50 13June03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Sarjeant Gallery 28Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Screen Announcements 10Nov03\Anderton response to taskforce.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Screen Announcements 10Nov03\Clark-Tizard NZ Film Commission.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Screen Announcements 10Nov03\Horomia - Screen Council statement.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Screen Announcements 10Nov03\Maharey - taskforce response final.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Screen Announcements 10Nov03\Media fact sheet.DOC
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Screen Announcements 10Nov03\Screen production taskforce response - main statement.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Screen Announcements 10Nov03\ScreenCouncilMembership22Nov.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Te Papa 5th birthday 14Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Te Papa reappointments pr4Sept03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Te Waka Toi apts 18Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\Tizard to SXSW pr7March03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\TopArtExhibition 3June03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\War memories campaign 10Jan03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Arts PR 2003\nzfcnotes.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Blues train summary.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AI Release Budget 2003\03-05 - training - data - Finalised Local Authority by Industry 31 March 2
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AI Release Budget 2003\03-05 - training - data - gateway senior school numbers.xls
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AI Release Budget 2003\Youth training Auc City.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AK03 Congrats 6Oct03oc.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld Employment Stats 11Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld Tourism Stats 12Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld crime stats 14Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld teen parent unit pr15Dec03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld transport a team effort 5Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Akld-Govt Mayoral Forum pr8Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AkldBudget200315May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\AkldMuseumBill passed pr3Sept03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Auckland unemployment fall a welcome signal.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Backup of Auckland unemployment fall a welcome signal.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Backup of Govt takes commonsense approach to tertiary fees.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Blues Trains congratulated 23May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Coddington desperate pr4Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Economic buoyancy in Akld 12Dec03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Get to the game early 15Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Good year for Akld-Govt 19Dec03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Govt takes commonsense approach to tertiary fees.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Health support for Older Aklers 1Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\New Mayoral Forum chair pr6Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Tizard on Akld Transport & LTMB 17Jan03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Tizard on Westhaven sale 20Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\Waiheke Heartland service 5Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\akld PHO pr1Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\billpassingMuseumrelease.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Akld Issues comment 12Dec03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Akld getting fair share 8Dec03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Another milestone for Auckland transport - Britomart.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Auckland roading projects start 21Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Backup of Another milestone for Auckland transport - Britomart.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Backup of Tizard welcomes opening of another key Auckland road project.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\Ak Tizard.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\Ak governance Carter.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\Ak transport Swain.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\Land Transport package - package at a glance.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\Land Transport package - pm statement.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\governance fact sheet.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\transport fact sheet one.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\transport fact sheet three.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\December 12 Announcement\transport fact sheet two.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Draft Coddington Ak Transport repsonse.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\NthShoreBuswaypr19Mar03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Tizard welcomes opening of another key Auckland road project.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Transfund 30 June 2003.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland Transport 2003\Yet more progress 4Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Anzac WIPO 15April03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Copyright Tribunal Harvey Appt 17Jan03.DOC
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\CopyrightAmdmtBill Passed 22Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Digital Amendments to Copyright Act 25 June 03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\EngineersAppts pr27May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\IPEG NZ mtg pr10March03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\MotorVehicleTradingpassed pr3April03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Patent - internet retailers advised to seek legal advice Final.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Patent - internet retailers ministers seek briefing Draft.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Patent attorney changes 17April03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Patent&PlantAmdmts pr6Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\PatentTermXtn DiscussionPaper pr11June03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Performers Rights 18Dec03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Pharmaceutical xtsn deferred Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\Trade marks regulations discussion document.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\TradeMarksAct into force NZRFU pr20Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\TradeMarksAct into force pr20Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\WIPO Vale media release - 15 Apr.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Commerce 2003\~
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\DRAFT alister taylor.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\2.1 percent drop in violent crime in central Auckland.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\AccessRadioFM 20May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Akld WM Museum 25 June 200303.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\AkldCentralEducation15May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\AkldCentralTraining15May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\AkldWarMemorialBill Intro 12June03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Backup of Akld WM Museum 25 June 200303.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Backup of Backup of Akld WM Museum 25 June 200303.BK
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Backup of PHOs have 291703 friends in Akld DHBs area.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Broadband for Gulf Islands pr14Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\CentreofInnovation pr8May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\E_learningTeacher6Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\FranklinRd Lights pr1Dec03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Gateway programme expands 28Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\ICT Schools funding update pr18Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Literacy&NumeracyinEducation pr18Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Low Unemployment pr16April03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\PAM comment 28March03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\PHOs have 291703 friends in Akld DHBs area.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Smokefree profitability Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\TM new school for Waiheke Island.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Teacher boost pr30April03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Tizard on dog attack 4Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Tizard on victims' rights 4Feb03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\TizardwelcomesnewWaihekeschool3April03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Waiheke NBR Award congrats pr9Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Generics 2003\Akld Car Conversion rates pr19June03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Helen Clark Arts PR 2003\Backup of Film Anncmt 10Nov03.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Helen Clark Arts PR 2003\Film Anncmt 10Nov03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Helen Clark Arts PR 2003\heritage budget 7May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Helen Clark Arts PR 2003\oralhistory09.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Helen Clark Arts PR 2003\whale rider 27Jan03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Kathryn Hutton on ladders Aug 03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\MVSA - Notes for 15 Dec 2003 launch.DOC
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\SpeedLimitRule - March 2003 .doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\03-06-30-roading-tizard2.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\03-07-02-roading-tizard2.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\AkldRailPlanresponse15May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Backup of Govt work on reducing transport emissions moves ahead.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Backup of VehicleEmissionsRule QAs 18Sept03.wbk
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Govt work on reducing transport emissions moves ahead.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\PraiseforRailSafetyProgrammepr21Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\RailSafetyProgrammepraised21Aug03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Vehicle Emissions Anncmt Oct03\Govt announces Vehicle Emissions Policy 1Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Vehicle Emissions Anncmt Oct03\VEP Speaking Points.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Vehicle Emissions Anncmt Oct03\Vehicle Emissions Policy Bkgrd Info 1Oct03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Vehicle Emissions April 2003\Govt programme on Vehicle Emissions pr3May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Vehicle Emissions April 2003\MOTfact_sheet-vehicle_emissions_jan03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\Vehicle Emissions April 2003\Vehicle Emissions Q&A May03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\VehicleEmissionsRule QAs 18Sept03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Transport PR 2003\VehicleEmissionsRule18Sept03.doc
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\mayoral forum communique notes.docHi Ross the issue is definitely just stack filling up with all those multiIdentifyS calls (lines 161 and 176): you can see them all in your stack trace. This is a recursion problem caused by my longpath fix (I had to introduce the recursion there in order to try to recover a directory walk for a directory that fails due to a long path). If you upgrade with that latest commit I pushed to develop it will hopefully be fixed. Shame couldn't pin it down to a particular file or dir and sorry if you have to spend ages redoing this :(
ross-spencer
commented
8 years ago Cool, that's built and running now. I should have results in the morning.
I can see what you're referring to in the stack trace now.
One question, maybe answered, maybe not (i'm not entirely clear on it) but why was the long path syntax appearing for something not a long path?
[FILE] \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\billpassingMuseumrelease.doc
[ERROR] winapi error #4In the full 'Master Copies' scan.
vs.
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003\billpassingMuseumrelease.docin the previous standalone scan of: 'Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\'
richardlehane
commented
8 years ago Ah ... I'd assumed you'd added the long path prefix (\?\Z:) yourself.
The current behaviour is just to try a long path on any error... because, if it fixes things, why not?
But those long paths shouldn't be exposed to users ... there is a corresponding shortpath function to split them off again. I'll look into why those appeared in your logs. Did they appear in your output file too?
ross-spencer
commented
8 years ago Hi Richard,
My managed desktop was restarted over night (unexpectedly) but it did get up to the set of files it normally halts, here is an example row:
84479,,file:///%3F/Z:/Master%20Copies/Judith%20Tizard%20-%20E2/E2/Press%20releases%20%26%20Advisories/2003/Auckland%20PR%202003,\\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Auckland PR 2003,Auckland PR 2003,,Done,,Folder,,2008-11-13T15:12:32+13:00,false,,,,,,You'll see the example longpath syntax in there.
I'm rerunning now to see how it halted/if it did. But that looks like another thing to search through.
richardlehane
commented
8 years ago Hi Ross in your output, are you only seeing those long path prefixes for the folders ... are files coming out OK?
What is happening is there is a new shortpath function that restores original paths after the long path fix has been applied. I neglected to apply this before droid output prints folders but it should kick in for files (and for all the other output formats which don't print folders at all). Can fix this tonight.
richardlehane
commented
8 years ago Hi Ross have pushed a further commit to the develop branch to ensure that in Droid output the original names of folder paths are also restored.
ross-spencer
commented
8 years ago Cool, that's built and on its way. Will report back on how it goes. Thx Richard.
On Thu, Jan 21, 2016 at 10:50 AM, Richard Lehane notifications@github.com wrote:
Hi Ross have pushed a further commit to the develop branch to ensure that in Droid output the original names of folder paths are also restored.
— Reply to this email directly or view it on GitHub https://github.com/richardlehane/siegfried/issues/66#issuecomment-173372014 .
ross-spencer
commented
8 years ago Hi Richard,
Still erroring, but I guess this is now a #WONTFIX ? - But it looks better now as an error (just the long path syntax in the final output path):
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Backup of Backup of Akld WM Museum 25 June 200303.BK
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Backup of PHOs have 291703 friends in Akld DHBs area.wbk
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Broadband for Gulf Islands pr14Nov03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\CentreofInnovation pr8May03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\E_learningTeacher6Nov03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\FranklinRd Lights pr1Dec03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Gateway programme expands 28Aug03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\ICT Schools funding update pr18Aug03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Literacy&NumeracyinEducation pr18Aug03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Low Unemployment pr16April03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\PAM comment 28March03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\PHOs have 291703 friends in Akld DHBs area.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Smokefree profitability Nov03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\TM new school for Waiheke Island.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Teacher boost pr30April03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Tizard on dog attack 4Feb03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Tizard on victims' rights 4Feb03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\TizardwelcomesnewWaihekeschool3April03.doc
[ERROR] winapi error #4
[FILE] Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Electorate PR 2003\Waiheke NBR Award congrats pr9Oct03.doc
[ERROR] winapi error #4
2016/01/21 18:35:03 [FATAL] walking \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advisories\2003\Generics 2003; got open \\?\Z:\Master Copies\Judith Tizard - E2\E2\Press releases & Advi
sories\2003\Generics 2003: winapi error #4 Just a heads up, I'll run this code without MD5 generation now, as I think I'm getting different behavior that way too. I'll report that in a different ticket.
I hope these tickets are helpful! :)
richardlehane
commented
8 years ago I feel like we're making progress!
I've got a couple of ideas:
richardlehane
commented
8 years ago Hi Ross
I've added a new -copy flag to sf.
This causes sf to fully copy files into memory before processing, rather than access files dynamically and only read as much as necessary. Obviously this will run more slowly in normal usage but if you are generating hashes anyway you will need to load the full file so shouldn't impact speed in those cases.
Advantage of using this flag is that if you are accessing files over an unreliable network you can copy them once into memory in a single operation. This should reduce IO errors and also its makes retrying possible: if you use both the -throttle and -copy flags, sf will automatically retry failed read operations.
Downside is you need enough memory to load files completely into memory. I.e. this won't work with 1GB files on 32 bit architectures. But if you are working with 1000s of moderate sized files, should work fine.
I've got a bit of code tidying to do after all of these piecemeal fixes to sf but is ready for trialling now.
Just use e.g. sf -droid -throttle 50ms -copy -hash md5 DIR
ross-spencer
commented
8 years ago Hi Richard,
I have created a file that exacerbates the issue.
The behaviour for the data I am working with never seemed right and so I watched the logs very closely until I could get this down to a set of files that constantly made sure SF didn't let go of file descriptors.
I got it down to a set of 2,000 files and with throttling on, watched a very specific set take the descriptors up. I found a set of PSD files. The handles incremented by one for each file processed.
The attached file is a skeleton file for a PSD file. I don't know much more about what's happening here other than it is simply on the checksum generation part of your code the issue happens:
var b *siegreader.Buffer
var cs []byte
if checksum != nil {
b = s.Buffer()
var i int64
l := checksum.BlockSize()
for ; ; i += int64(l) {
buf, _ := b.Slice(i, l)
if buf == nil {
break
}
checksum.Write(buf)
}
cs = checksum.Sum(nil)
checksum.Reset()
}I can comment this out and the issue disappears. I tried to narrow it down further but I'm now no further along and decided to just make sure I could recreate this in Linux as I was sure I'd be able to.
All I know about the files is that it is not divisible by 64 (MD5 blocksize) and so it will trigger the EOF err each time. But i can;t imagine that's the issue seeing your handling in the dev branch of the slices.
The file is not much more than 3 mb. I've attached a collection of them - as skeleton files they zip well.
The file will re-create the behaviour wherever you place it removing (it seems) any consideration of recursive long-path corrections, and you can create multiple copies of the file and it will simply make the number of descriptors go up, 1-by-1.
I am monitoring this on Linux too using the lsof command and a 2000ms throttle as GC won't remove the descriptors even in that time.
SF cmd:
sf -hash=md5 -log p -throttle=2000ms zeroes > test.txtProcess monitoring:
lsof -p PIDExample output after ~30 files scanned. Continuing to go up:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sf 28037 goatslayer cwd DIR 8,1 4096 14180409 /home/goatslayer/Desktop/zeros-all
sf 28037 goatslayer rtd DIR 8,2 4096 2 /
sf 28037 goatslayer txt REG 8,1 10489512 15859769 /home/goatslayer/git/go/bin/sf
sf 28037 goatslayer mem REG 8,1 3295093 14159170 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-006 (4th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159113 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-006 (3rd copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14155818 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-005.PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159027 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-005 (copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159083 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-005 (another copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159221 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-005 (5th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159169 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-005 (4th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159112 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-005 (3rd copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14155817 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-004.PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159026 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-004 (copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159082 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-004 (another copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159220 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-004 (5th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159168 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-004 (4th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159111 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-004 (3rd copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14155815 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-003.PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159025 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-003 (copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159081 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-003 (another copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159219 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-003 (5th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159167 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-003 (4th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159109 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-003 (3rd copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14155811 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-002.PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159024 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-002 (copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159080 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-002 (another copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159218 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-002 (5th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159166 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-002 (4th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159108 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-002 (3rd copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14155806 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-001.PSD
sf 28037 goatslayer mem REG 8,1 3295093 14157355 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-001 (copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159079 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-001 (another copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159217 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-001 (5th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159163 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-001 (4th copy).PSD
sf 28037 goatslayer mem REG 8,1 3295093 14159106 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-001 (3rd copy).PSD
sf 28037 goatslayer mem REG 8,2 1840928 7738830 /lib/x86_64-linux-gnu/libc-2.19.so
sf 28037 goatslayer mem REG 8,2 141574 7739408 /lib/x86_64-linux-gnu/libpthread-2.19.so
sf 28037 goatslayer mem REG 8,2 149120 7738818 /lib/x86_64-linux-gnu/ld-2.19.so
sf 28037 goatslayer mem REG 8,2 217032 2361337 /var/cache/nscd/passwd
sf 28037 goatslayer 0u CHR 136,23 0t0 26 /dev/pts/23
sf 28037 goatslayer 1w REG 8,1 12288 14159931 /home/goatslayer/Desktop/zeros-all/test.txt
sf 28037 goatslayer 2u CHR 136,23 0t0 26 /dev/pts/23Descriptor count e.g:
lsof -p 28037 | wc -lI hope the set of files helps you to uncover something new.
I may be able to visit this over the weekend again I figured it good to share my progress for now. I just can't see what it might be causing the issue right now... (too long analysing!)
richardlehane
commented
8 years ago Ross - this is brilliant, should really help narrow it down! I've got a sense of what it might be, will report back soon.
In the meantime, the new -copy flag seems to prevent this behaviour. If you are doing hashes you need full file reads anyway (copy causes a full file read as an atomic action prior to processing - this should prevent leaks even on flaky networks), and so I'd recommend using this flag if your file corpus is all reasonably sized (i.e. don't use if 1GB files). You'll also get a small speed-up (maybe 10%).
So this would make your command:
sf -hash md5 -log p -throttle 2s -copy zeroes > test.txt
P.S. when throttling, you can use any time unit, so -throttle 2s works too :)
ross-spencer
commented
8 years ago Thanks Richard, I'll give the -copy flag a whirl but have been focusing on this issue amongst a few others between this and DROID.
As for the throttle then of course! - the 2000ms was so I could observe the corpus and get various process monitors up and running (probably in a rather amateur way!) and do other bits of research, documentation, and experimentation without SF completing - having narrowed it down to a couple of thousand objects, and then recreating the behaviour with one skeleton file consistently, it became more necessary to slow SF down further.
Additionally, the behaviour on my Windows machine is as you expected, it gets up to a significant number of handles, in the case of the corpus ~10,000 handles out of the first 80,000 objects it comes across, and then bugs out.
Looking forward to hearing more/looking at the code. It has been an interesting learning experience.
ross-spencer
commented
8 years ago Oh! Any time unit!!! I see! ^_^;
Thx for the tip!
richardlehane
commented
8 years ago I believe this is now fixed.
The problem was in the siegreader package. SF treats files as composite structures:
a) the file itself is an object & this just holds a small amount of memory for the first 8192 bytes of the file b) files hold references to a data which is where the rest of the file is accessed (normally via mmap)
If a file can be identified using just that initial 8192 bytes (the peek), we don't need a (b) for that file. PSD files can be identified just with the peek.
Files and datas are re-used by sf to avoid allocation/garbage collection. Both types have pools for this. Files and datas are only reset when they are retrieved from the pool (i.e. not on put) - this allows you to do a Last() call which grabs the latest file back out of a pool with its attached data. This is necessary for hashing and unzipping (because the main identification has run and has discarded the file, but we want to steal it back again).
The problem for these files was that the data (b) was not created by the initial process but by the subsequent Last() call. This had the unfortunate effect of those datas never being recycled back into the pool (because of mistaken idea that they were already in the pool).
The fix for this ensures that datas go immediately into the pool when they are created after a Last() call.
This was a pretty subtle bug! Thanks for narrowing it down to such a precise reproduce
ross-spencer
commented
8 years ago Using your throttle tip! This is how it all looks after about a minute and a half:
goatslayer@goatslayer-acer-linux:~/Desktop/zeros-all$ ps aux | grep sf
root 50 0.0 0.0 0 0 ? S< Jan26 0:00 [ata_sff]
goatsla+ 398 1.1 0.6 310180 50428 pts/9 Sl+ 14:29 0:01 sf -hash=md5 -log p,e,t -throttle=2s zeroes
goatsla+ 455 0.0 0.0 11744 928 pts/23 S+ 14:31 0:00 grep --color=auto sf
root 30244 0.0 0.0 8340 3676 ? S 10:47 0:00 /sbin/dhclient -d -sf /usr/lib/NetworkManager/nm-dhcp-client.action -pf /run/sendsigs.omit.d/network-manager.dhclient-wlan0.pid -lf /var/lib/NetworkManager/dhclient-e7399605-edb2-4aa0-bdba-d1fa74b7524d-wlan0.lease -cf /var/lib/NetworkManager/dhclient-wlan0.conf wlan0
goatslayer@goatslayer-acer-linux:~/Desktop/zeros-all$ lsof -p 398
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sf 398 goatslayer cwd DIR 8,1 4096 14180409 /home/goatslayer/Desktop/zeros-all
sf 398 goatslayer rtd DIR 8,2 4096 2 /
sf 398 goatslayer txt REG 8,1 10554160 15859769 /home/goatslayer/git/go/bin/sf
sf 398 goatslayer mem REG 8,1 3295093 14159177 /home/goatslayer/Desktop/zeros-all/zeroes/64-BLOCKSIZE-ZEROS/ZEROS-011.PSD
sf 398 goatslayer mem REG 8,2 1840928 7738830 /lib/x86_64-linux-gnu/libc-2.19.so
sf 398 goatslayer mem REG 8,2 141574 7739408 /lib/x86_64-linux-gnu/libpthread-2.19.so
sf 398 goatslayer mem REG 8,2 149120 7738818 /lib/x86_64-linux-gnu/ld-2.19.so
sf 398 goatslayer mem REG 8,2 217032 2361337 /var/cache/nscd/passwd
sf 398 goatslayer 0u CHR 136,9 0t0 12 /dev/pts/9
sf 398 goatslayer 1w REG 8,1 20480 14159931 /home/goatslayer/Desktop/zeros-all/fix-test.txt
sf 398 goatslayer 2u CHR 136,9 0t0 12 /dev/pts/9I'd say it's looking good.
I'm confident that this will push past the issues I was having with the scan back at work. You can pretty much close this off.
Happy to keep help improving the tool even if it just takes a bit of time to narrow down. ^_^
richardlehane
commented
8 years ago Great thanks again Ross - closing for now, but please re-open if necessary. I'll get a 1.4.5 release out shortly with your recent issues fixed.
ross-spencer
commented
8 years ago Aaannd! It's working perfectly on Windows 7. If you're interested I can do a performance comparison between my current scan and -copy flag when this completes. Given there are no other SF handling errors when run sans hash, I'm sure this will get through now.
Hi Richard,
Logging this here so I can collect the data more analytically and see if there are any existing clues I should be looking at or tips. Again, running through our legacy collection.
Can I confirm which SF log setting I need to see the file it was processing at the time?