Scan wiki-dirs and add contents to pollies list

[Marxsal]

May need code to prevent usage of downloads dir and/or directories with massive number of files.

[TiddlyTweeter]

• I'm not clear myself how you'd do that. I guess a PS "filter" (= exclude "downloaddir"; + exclude "item hits gt than") No idea how to do it though.

• Might help ... Providing a listing of all wiki the end-user can see via a Menu option (replaces TiddlyWiki Folder with FolderS) what is actively monitored in "wikisdirs"?

[Marxsal]

I've started to work on this.

Just discovered that by default PS will find contents of sub-directories of the target wikis. The question is, is this the behaviour that we want, or do we want to limit to the upper directory (because the lower directory may be backup files that have the same stem name).

[Marxsal]

Oops. No recursing wasn't on by default. That was me.

I'm thinking we should only do the top-level, and leave it up to the user to specify any lower levels.

[Marxsal]

See branch Wiki-Directories. Directories are not recursed. Download directory is omitted, possibly with a warning. But wikis can be restored from download dir by name per the wikis section.

Need to check -- if a name is in both a wiki dir, and by name, does it appear twice?

[TiddlyTweeter]

I'll test & comment.

[TiddlyTweeter]

The question is, is this the behaviour that we want, or do we want to limit to the upper directory (because the lower directory may be backup files that have the same stem name).

Its a complicated one in a way. My thought is IF I can manage in do-settings-ps1 to offer mass addition of re-cursed wikis to [wikis] the user can then EDIT out wikis not needed. That scenario would give user efficiency of addition then honing down.

Your auto-load tool is a really neat idea. The problem with the caveats and cautions is in a way it kinda reduces its "oomph"?? Meaning I think it might be best at full-on auto, though excluding downloads??

But I'll test properly & comment more later.

[TiddlyTweeter]

Part of this is also about having a clear mental map of what is happening. So some of it is that documentation is needed, not just protecting users?

[TiddlyTweeter]

@Marxsal ... Looks like it has great potential!

But my own feeling is it needs to be recursive--or have that option. You can see the issue below ... whilst auto-reading of "P\tw-wikidir\tricky" dir is productive, reading of "P\tw-wikidir\goose\wiki" doesn't really add much to simply specifying a full path in [wikis].

IF "P\tw-wikidir" were recursively probed I think it would be a more useful tool??

I understand the worry about detecting files that should not be monitored. How much of an issue do you think it could be in practice?

File before expansion: C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\tricky
File after expansion: C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\tricky
Comparing paths C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\tricky and C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\Users\Polly\Downloads BEFORE
Dir C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\tricky passes directory tests
Comparing paths C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\tricky and C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\Users\Polly\Downloads AFTER

    Directory: C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\tricky

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       12/08/2019     08:31        5255488 tricky(SUCCEED).html
-a----       12/08/2019     08:31        5255488 tricky-1.0.0.html
-a----       12/08/2019     08:31        5255488 tricky-accounts(2019).htm
-a----       12/08/2019     08:31        5255488 tricky-test.html
-a----       12/08/2019     08:31        5255488 tricky.htm
-a----       12/08/2019     08:31        5255488 tricky.html
-a----       12/08/2019     08:31        5255488 tricky.tw
File before expansion: C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\goose\wiki
File after expansion: C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\goose\wiki
Comparing paths C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\goose\wiki and C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\Users\Polly\Downloads BEFORE
Dir C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\goose\wiki passes directory tests
Comparing paths C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\goose\wiki and C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\Users\Polly\Downloads AFTER

    Directory: C:\bag\z_pa\PortableApps\tw_polly\dev\test\P\tw-wikidir\goose\wiki

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----       24/07/2019     07:57              0 goose-wiki.htm

[TiddlyTweeter]

Its also worth noting that when "do-settings" comes to [wikidirs] section it should be possible to display a listing of the wikis that would get automated and ask for confirmation.

[Marxsal]

I understand the worry about detecting files that should not be monitored. How much of an issue do you think it could be in practice?

I'm thinking about the scenario where people, avoiding github, do their own versioning by putting a copy of a file in sub-directory. They think they have a nice, secure, backup of their file at a particular stage, whereas, in fact, it will be written over the next time it's stem name appears in the download directory.

Technically, it's not hard to recurse (I think. I hope).

I guess having an option that the user deliberately chooses in settings.ini would be one possibility.

Another (somewhat harder) possibility is to warn users when a stem has been detected in more than one place. Then we get into the issue of whether to just report it, or to force the user to acknowledge it. The latter might get annoying. I just realized that with recursing, you would effectively have insta-Parrot, since more than one file with the same stem would could get restored if they belonged to sub-directories of each other.

I sure hope our 2.5 users (or has it gone down?) appreciate all this work ...

[TiddlyTweeter]

I sure hope our 2.5 users (or has it gone down?) appreciate all this work ...

Lol! An upside for me is that the methods we using for Polly can be applied to some other issues I have. One spin off already is I learned about "iwr", basically a "fetch" mechanism. This is handy for my work which often involves repetitive downloads.

[TiddlyTweeter]

whether to just report it, or to force the user to acknowledge it.

IF duplicate stem detection is possible I'd be perfectly happy to enforce something like ...

Polly has detected you have two wiki (html/htm/tw) files with the 
same name under this directory. That will cause problems! 

To solve this issue make sure all wikis have unique names. 
Or individually add selected wiki under [wikis] section of settings.

Meaning the process won't run.

[Marxsal]

I've pushed duplicate detection (still Wiki-Directories branch). My wording was different, but that can be worked out later. The main point is that the two files are in two different directories.

[TiddlyTweeter]

@Marxsal I've pushed duplicate detection (still Wiki-Directories branch).

Slightly delayed on this. Hope to get to it tomorrow.

[TiddlyTweeter]

@Marxsal, a comment on "stem-uniqueness-enforcement" :-). I been playing with idea that the "wiki name" ("wiki.ext") be used as the "label" for the wiki address. I'm assuming that only the last read address would exist in the "hash-table" where labels are identical? The last would over-ride the first. Not sure if this is workable but worth a thought. If I'm right it could ensure you only have one target for Roosting (restore).

[Marxsal]

@TiddlyTweeter Not sure which problem you're thinking of. It could be used to enforce uniqueness, but would do so silently. It would not help anyone loading an entire wiki directory because, of course, those files would have no label. Thanks!

[TiddlyTweeter]

@Marxsal I'm nearly finished on my Regex Holiday. I have not forgotten Polly. Soon, I hope.