Is it possible to upload external files into ./files/ directory using TiddlyWiki?

[pimgeek]

In this link I learned that I can link to ./files/xxx.yyy files in tiddlers using [ext[./files/xxx.yyy] syntax on Node.js backed TiddlyWiki website. 😃

But if I use TiddlyWiki's drag-and-drop / import files action to upload external files (not only image files, but also video files and pdf files), it always creates big tiddler with embedded binary data. 😑

So my question is, is there a way to upload external files directly into ./files/ directory (and auto-create a skinny tiddler pointing to that file) using TiddlyWiki? I've searched throughout GitHub Issues / Google Groups / Official Documents / Dynalist Tool Map outlines without finding a solution... 😥

[Jermolene]

Hi @pimgeek there is no support at present for uploading files from the browser directly to the ./files/ area. It is something that I'd like to add at some point.

[pimgeek]

@Jermolene Thanks for your reply, currently I consider writing a file system watching script to automatically create new tiddlers pointing to ./files/xxx , but afraid it will confuse the tiddler updating mechanism. 😅

[joshuafontany]

Bob and Bib.exe both have this option (along with serving multiple wikis, multiple users, & other great features).

https://github.com/OokTech/TW5-Bob/blob/master/Documentation/ServerMedia.tid

On Fri, Jan 3, 2020, 5:41 AM pimgeek notifications@github.com wrote:

@Jermolene https://github.com/Jermolene Thanks for your reply, currently I consider writing a file system watching script to automatically create new tiddlers pointing to ./files/xxx , but afraid it will confuse the tiddler updating mechanism. 😅

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Jermolene/TiddlyWiki5/issues/4401?email_source=notifications&email_token=ACFOMB3BLJ4MORJTGHWNH2LQ346A7A5CNFSM4KCK2LJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIBEQQI#issuecomment-570574913, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACFOMBZPJCHGU5Y6KVPC2XLQ346A7ANCNFSM4KCK2LJQ .

[pimgeek]

Bob and Bib.exe both have this option (along with serving multiple wikis, multiple users, & other great features). https://github.com/OokTech/TW5-Bob/blob/master/Documentation/ServerMedia.tid

Thanks for your recommendation, I have tried TW5-Bob, and it does exactly what I described in this issue. I have also tried other Bob features on my own server, but found it not suitable for my use case (not easy to setup password authentication , Bob's control panel contains too many tabs , documentation seems not as clear as TiddlyWiki official documentation... ).

I will need to wait for plain TW5 to support this feature, while using Bob privately on my PC. 😃

[Arlen22]

@pimgeek This is a very simple feature to implement and I thought it was already implemented in TiddlyWiki.

Actually, upon further investigation, it apparently already is. This should already work.

https://github.com/Jermolene/TiddlyWiki5/blob/master/core/modules/server/routes/get-file.js

Create a files folder in the wiki data folder (i.e. beside the tiddlers folder), then you should be able to access any files there directly. Start with a simple file with no spaces (like test.txt) to make sure it works and then go from there. If you find any bugs feel free to open another issue.

[Arlen22]

You would access the file at http://localhost:8080/files/test.txt.

If using TiddlyServer it would be http://localhost:8080/path/to/wiki/files/test.txt.

[Jermolene]

HI @Arlen22 I think @pimgeek wants to be able to upload files into the /files folder directly from the browser, which is still not currently supported.

[linonetwo]

But why do we need this? Just import them to the /tiddlers folder, and using lazy-image feature is enough. (Which is enabled by default in TidGi desktop)

[pimgeek]

using lazy-image feature is enough

From what you said, I guess there are some newly updated features on how TiddlyWiki process imported image / video / pdf tiddlers, but I didn't notice and try.

I did notice LazyLoading mechanism recorded in TiddlyWiki official documents, but that's several years ago, and at that time lazy-all would break TiddlyMap and lazy-images didn't seem to decrease page loading time dramatically.

Maybe I was wrong about lazy-images command line option's real effect, I will try using it again by importing very large files to see if it does work as expected. 🤓

[arunnbabu81]

@pimgeek I use tiddlyclip plug-in, tiddlyclip and save media add-ons by @buggyj to clip images (using screenshots in firefox or snap feature of tiddlyclip add on) directly from websites or pdf into tiddlers in this format [img[.relativepath]] on clipping. Images get saved automatically into a media folder in the subdirectory of the singe file wiki. Multiple images can be clipped into a single tiddler in this way. I have been using this for the past many months. Till TW support this natively, this is the best option I have found.

[pmario]

But why do we need this? Just import them to the /tiddlers folder, and using lazy-image feature is enough. (Which is enabled by default in TidGi desktop)

The biggest problem with unoptimized images directly imported via TW is size ... I did create a little PNG screenshot and directly imported it to TW.

Images optimized with https://squoosh.app/editor in the browser.

Image	File Size	Size Saved	Settings	Description
image.png	121 KB	--	browser default	unoptimized browser saves it
image.oxi.png	70 KB	42%	default	using OxiPNG optimisation
image.avif	26 KB	79%	quality 40%	using AVIF format

Click to expand the screenshot, that contains mainly whitespace


[pmario]

All Images to compare them, with 100% browser zoom. Otherwise the browserzoom affects the quality.

121 KB


70 KB


Sorry AVIF cannot be uploaded atm.

image.avif.zip

[linonetwo]

I use mac Automator + pngquant to auto compress screenshots/images put into desktop

https://wiki.onetwo.ren/Mac%e8%87%aa%e5%8a%a8%e5%8e%8b%e7%bc%a9%e6%a1%8c%e9%9d%a2%e4%b8%8a%e7%9a%84%e5%9b%be%e7%89%87 https://www.cnblogs.com/deppwang/p/13455495.html

I think pngquant can be embed into TidGi or TWDesktop to auto compress imgs in ./tiddlers and ./files/ , toggled by option.

This can't be made as a plugin, because JSON plugin can't deliver binary executable, and use plugin library to deliver folder based plugin is not possible yet.

Anyway, this can be useful when "auto upload external files into ./files/ directory". And you can already do it now with programs like MacOS Automator.

[linonetwo]

@pimgeek You can also try https://saqimtiaz.github.io/tw5-file-uploads/ to see if this issue can be closed.

[pmario]

And you can already do it now with programs like MacOS Automator.

On windows and Linux systems?

[linonetwo]

programs like MacOS Automator is widely used, there are many alternatives

[pimgeek]

The biggest problem with unoptimized images directly imported via TW is size

This is also one of my concerns, I really don't expect my readers having to load 10M+ data from Node.js server, just for viewing my digital garden homepage (even if the image files loads much later) 🤔

Apart from that, single image file's storage size is also a headache, but I can overcome it by reducing the png file size before uploading.

if this issue can be closed

From my own experience, I'm not sure if the proposed solutions are stable and can be used out-of-the-box. E.g. For horizontal storyriver need I have Krystal plugin, for outliner editor need I have Streams plugin. But for image upload, I don't have as much confidence on lazy-images as you have.

But I can speed up my testing process, if things work as you said, I will close this issue. 😊

[pimgeek]

@linonetwo I've finished evaluting the above solutions:

1. lazy-image solution: it works as you said, big image files are not loaded when I open the website, they are only loaded when I open the image file tiddler. (but for mp4 file tiddler, I need to use lazy-all option)
2. File Uploads plugin: it doesn't provide a filesystem uploader sub-plugin, that means I cannot use it to upload to ./files/ or ./media/ dir, so it is not currently a solution
3. Automator or AutoHotKey: it is a practical solution. I myself had written background python script to do something similar --- whenever a file is copied to ./files/ or ./media/ dir, the script would automatically create a skinny tiddler with _canonical_uri, but it feels clumsy when put into production environment.
4. tiddlyclip plug-in, tiddlyclip and save media: it's not as simple as lazy-image solution, and it does not support mp4 / pdf files upload either.

In summary, the most promising solution seems to be lazy-all option (recently I don't use TiddlyMap anymore, so no error message is popped up). I will use it in production environment for some time. Since this solution is essentially the same as uploading the file to Node.js server without increasing data-transfered & loading-time, I will close this issue.

[linonetwo]

Lazy-all seems to cause errors on some user https://talk.tiddlywiki.org/t/product-website-of-tidgi-created-with-tiddlywiki/7165/9

But I can't reproduce it, so I didn't raise a issue here. But I switch to lazy-image later. I only use lazy-all in my wiki on TidGi locally, not on a public site.

[pimgeek]

I only use lazy-all in my wiki on TidGi locally, not on a public site

I will use it in my home network, with another user. This is not a public website, but I still view it as production environment, because the other user does not have technical background, and will help discover some inconveniences which were previously omitted by me.

[pmario]

I will use it in my home network, with another user.

Which OSes do you use?

If you work with the local file systems, you can handle the access rights on the OS level, so your server setup can be much more bare bones.

It may be an option to use a local WebDav server, which can deal with single file wikis and it works with Saq's file-upload plugin. There is an into-wiki at: https://saqimtiaz.github.io/sq-tw/webdav/ which describes the "wiki-farm" project a little bit.

Plugin readme: https://saqimtiaz.github.io/sq-tw/webdav/#%24%3A%2Fplugins%2Fcommons%2Ffile-uploads%2Freadme

Saq's plugins are there: https://saqimtiaz.github.io/tw5-plugins-sandbox/

[pimgeek]

It may be an option to use a local WebDav server, which can deal with single file wikis and it works with Saq's file-upload plugin.

Thanks for your suggestion, I managed to setup WebDAV server and later noticed that it only supports Single-File TiddlyWiki, so I searched in Discourse Forum and found this solution: https://talk.tiddlywiki.org/t/use-external-images-in-node-js/3332/4 which is also implemented by Saq.

This Node.js files PUT Uploader solution is better than lazy-images (On some mobile browsers, images don't show up the first time I open a tiddler, I have to open it twice to make the images load). Saq's solution is still in Experimental mode, but it is closest to what I have been searching for. 😃