Built-in externalimages build corrupts original image files

[ejheil]

Tiddlywiki version 5.1.21, running on OS X 10.14.6, node version 13.8.0.

Summary: running --build externalimages replaces the original image tiddler file in the tiddlers directory with a six-byte garbage file.

Demo:

➜  tiddlywikis git:(master) ✗ tiddlywiki test --init server    
Copied edition 'server' to test
➜  tiddlywikis git:(master) ✗ cd test
➜  test git:(master) ✗ tiddlywiki --listen          
Serving on http://127.0.0.1:8080
(press ctrl-C to exit)

At this point I drag and drop myimage.jpg, a 49K jpeg, into the wiki window and click "import."

 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 
 syncer-server-filesystem: Dispatching 'save' task: myimage.jpg 
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 
^C

The image is now in tiddles/

➜  test git:(master) ✗ ls -l tiddlers     
total 120
-rw-r--r--  1 edheil  staff    121 Mar  9 09:33 $__StoryList.tid
-rw-r--r--  1 edheil  staff  49303 Mar  9 09:33 myimage.jpg
-rw-r--r--  1 edheil  staff     90 Mar  9 09:33 myimage.jpg.meta

I build the built-in externalimages target -

➜  test git:(master) ✗ tiddlywiki --build externalimages
 syncer-server-filesystem: Dispatching 'save' task: myimage.jpg 
 syncer-server-filesystem: Dispatching 'save' task: $:/StoryList 

And the original myimage.jpg is gone from tiddlers/ replaced with a 6 byte file, though it has been correctly copied to output/images.

➜  test git:(master) ✗ ls -l tiddlers 
total 24
-rw-r--r--  1 edheil  staff   40 Mar  9 09:33 $__StoryList.tid
-rw-r--r--  1 edheil  staff    6 Mar  9 09:33 myimage.jpg
-rw-r--r--  1 edheil  staff  127 Mar  9 09:33 myimage.jpg.meta
➜  test git:(master) ✗ ls -l output/images 
total 104
-rw-r--r--  1 edheil  staff  49303 Mar  9 09:33 myimage.jpg
➜  test git:(master) ✗         

At this point if I didn't keep a copy of the contents of that output directory, I've permanently lost the original image!

It's unclear exactly what that new 6-byte file is, it appears to be garbage:

➜  test git:(master) ✗ file tiddlers/myimage.jpg
tiddlers/myimage.jpg: ISO-8859 text, with no line terminators
➜  test git:(master) ✗ cat tiddlers/myimage.jpg 
?w^~)?%                                                                                                                                                                             ➜  test git:(master) ✗ 

[Jermolene]

Hi @ejheil I'm sorry to hear that you've lost data. It's a subtle problem but now that I see it I'm surprised it hasn't cropped up before.

The 'externalimages' build target consists of the following commands:

"--savetiddlers","[is[image]]","images",
"--setfield","[is[image]]","_canonical_uri","$:/core/templates/canonical-uri-external-image","text/plain",
"--setfield","[is[image]]","text","","text/plain",
"--rendertiddler","$:/plugins/tiddlywiki/tiddlyweb/save/offline","externalimages.html","text/plain"

The two "setfield" commands modify the values of tiddlers in the store to prepare them for the final rendering step. That's usually OK because only the ephemeral copy of the wiki loaded into RAM is modified. However, with the tiddlywiki/filesystem plugin loaded the changes are synced back to the filesystem.

The easiest way to fix this to modify the tiddlywiki.info file of your wiki to remove the tiddlywiki/filesystem plugin, and then subsequently explicitly load it when you wish to run the wiki in read/write mode (e.g. with the listen command):

tiddlywiki +plugins/tiddlywiki/tiddlyweb <path-to-wiki-folder> --listen

The documentation needs to be improved here, but for the moment I've removed the externalimages build target from the "server" edition: a22438589ee76e4fc9505414bf44786238ae4bfd

[ejheil]

Hey, thanks! Don't worry, I didn't lose data; it was under version control and I could revert.

Still, it was alarming to notice that it'd been overwritten, so I filed the issue.

It might be helpful to have these build targets available in the tiddlywiki.info file in folders which are decomposed from html files. That's the only way I created folders for a long time, and there are no build targets there, and I had a hard time finding any information about how to create a build target on tiddlywiki.com. I now realize that documentation was probably lacking because those build targets would be put there for you if you created a folder with tiddlywiki --init, but I'd never done it that way (till the time I ran into this issue) so I had no idea that was the case.

[linonetwo]

It should be

tiddlywiki +plugins/tiddlywiki/filesystem <path-to-wiki-folder> --listen

with

    "plugins": [
        "tiddlywiki/tiddlyweb",
        "tiddlywiki/highlight"
    ],

[linonetwo]

I have problem on this, I tried to prevent some plugin/tiddler from loaded into the wiki, with this argument:

wikiInstance.boot.argv = ["+plugins/tiddlywiki/tiddlyweb","D:\wiki\linonetwo-wiki","--listen","port=5217","host=0.0.0.0","root-tiddler=$:/core/save/lazy-all","gzip=yes","readers=(anon)","writers=林一二","username=林一二","password=yRwyZZBRKqs-ii0Eiu_mR","anon-username=林一二","--setfield","[[$:/plugins/flibbles/relink]] [[$:/plugins/flibbles/relink-markdown]] [[$:/plugins/flibbles/relink-titles]] [[$:/plugins/tiddlywiki/markdown]] [[$:/plugins/Gk0Wk/TW5-CodeMirror-Enhanced]] [[$:/plugins/tiddlywiki/codemirror]] [[$:/plugins/tiddlywiki/codemirror-search-replace]] [[$:/plugins/tiddlywiki/codemirror-mode-xml]] [[$:/plugins/tiddlywiki/codemirror-mode-markdown]] [[$:/plugins/tiddlywiki/codemirror-mode-javascript]] [[$:/plugins/tiddlywiki/codemirror-mode-htmlmixed]] [[$:/plugins/tiddlywiki/codemirror-mode-htmlembedded]] [[$:/plugins/tiddlywiki/codemirror-mode-css]] [[$:/plugins/tiddlywiki/codemirror-keymap-sublime-text]] [[$:/plugins/tiddlywiki/codemirror-closetag]] [[$:/plugins/tiddlywiki/codemirror-autocomplete]] [[$:/plugins/tiddlywiki/codemirror-closebrackets]] [[$:/plugins/Gk0Wk/chat-gpt]] [[$:/plugins/Gk0Wk/sidebar-resizer]] [[$:/plugins/kixam/datepicker]] [[$:/plugins/kixam/moment]] [[$:/plugins/kookma/commander]] [[$:/plugins/kookma/refnotes]] [[$:/plugins/linonetwo/flow-chart]] [[$:/plugins/linonetwo/zx-script]] [[$:/plugins/linonetwo/github-external-image]] [[$:/plugins/linonetwo/markdown-transformer]] [[$:/plugins/linonetwo/source-control-management]] [[$:/plugins/linonetwo/slate-write]] [[$:/plugins/linonetwo/open-in-external-app]] [[$:/plugins/oflg/fishing]] [[$:/plugins/oflg/fishing-analysis]] [[$:/plugins/oflg/fishing-catchfish]] [[$:/plugins/oflg/fishing-zh-Hans]] [[$:/plugins/tiddlywiki/dynannotate]] [[$:/plugins/linonetwo/sub-wiki]] [[$:/plugins/linonetwo/tw-mobile-sync]] [[$:/plugins/linonetwo/tw-calendar]] [[$:/plugins/linonetwo/tw-whiteboard]] [[$:/plugins/linonetwo/tw-react]] [[$:/plugins/linonetwo/super-tag]] [[$:/plugins/tiddlywiki/tiddloid-tweaks]] [[$:/plugins/twcloud/tiddlyweb-sse]] [[$:/plugins/tiddlywiki/filesystem]]","text","","text/plain"]

But wiki will still trying to make these tiddler empty

(I have no plugins in tiddlywiki.info file, and no tiddlywiki/filesystem json plugin in tiddlers/ folder. So this should not caused by tiddlywiki/filesystem)

[Jermolene]

(I have no plugins in tiddlywiki.info file, and no tiddlywiki/filesystem json plugin in tiddlers/ folder. So this should not caused by tiddlywiki/filesystem)

That's very strange. Without the tiddlywiki/filesystem plugin I can't see how those tiddler files are being modified. It would be worth checking carefully to see if it is somehow being loaded.

[pmario]

@linonetwo ... You did post a password here. Make sure you change that one immediately for your own good.

[pmario]

@Jermolene ... I think this issue should have been closed after the fix mentioned in https://github.com/Jermolene/TiddlyWiki5/issues/4484#issuecomment-596779416

@linonetwo ... IMO resurrecting 3 year old issues should be avoided. You could create a new one and mention this one. I think it would be less confusing.

[linonetwo]

@linonetwo ... You did post a password here. Make sure you change that one immediately for your own good.

Thanks @pmario , but I'm using this technique https://github.com/Jermolene/TiddlyWiki5/discussions/7469 to utilize a one-time random password, so it's fine here.

[linonetwo]

@linonetwo ... IMO resurrecting 3 year old issues should be avoided. You could create a new one and mention this one. I think it would be less confusing.

Okay, I was keeping this link in my code, so I just click here

https://github.com/tiddly-gittly/TidGi-Desktop/blob/02999925f16ea4ec6e5049458f33f42be820fc52/src/services/wiki/wikiWorker.ts#L78

I think we can close this, the original issue is solved.

Hi, @ejheil , are you still using Tiddlywiki? :)

[linonetwo]

(I have no plugins in tiddlywiki.info file, and no tiddlywiki/filesystem json plugin in tiddlers/ folder. So this should not caused by tiddlywiki/filesystem)

My fault, sorry to alarm you. I actually have a JSON version of tiddlywiki/filesystem in tiddlers/ folder. Mis-installed from CPL. I have excluded them now.

https://github.com/tiddly-gittly/TiddlyWiki-CPL/blob/12fd237b5e9ae32c8134918868dc46c428a4739f/scripts/index.js#L306-L308

[linonetwo]

Context: I'm trying to use TidGi to start a blogging server so that an average user can set up a nodejs wiki blog on a PC.