Images and PDFs are deleted when being exported from Docker-Container using lazy-images mode

[xgadscu]

I'm using tiddlywiki in several Docker-Containers (based on nicolaw/tiddlywiki). To make the TW start page load faster especially on my smartphone they run with TW_ROOTTIDDLER: $$:/core/save/lazy-images.

In order to upgrade from TW 5.2.2 to 5.2.3 I built a new container and wanted to export the content from the old to the new container. For this I used TW-Commander with the filter "[all[tiddlers]]-[is[system]]" and exported all the found tiddlers as JSON. Importing the JSON-file into the new container seemed to be ok. 277 Tiddlers were imported. I later discovered that all images and PDFs were lost. Even in the source-container all tiddlers were marked as modified and the image- and pdf-content disappeared. The size of the JSON export file was 1,5 MB. The only reason for this behavior had to be the lazy-images mode.

Fortunately I had a backup. When I repeated the process running the source container with TW_ROOTTIDDLER: $$:/core/save/all, everything worked fine. Now the JSON-export-file had a size of 49 MB. All content was imported.

I like TW Commander because of its powerful functionality. But using it in lazy-images mode it seems to be quite dangerous :-( From the users point of view there is no hint not to be able to use the lazy-images mode.

Best regards xgadscu

[kookma]

filter "[all[tiddlers]]-[is[system]]"

This can be as simple as [!is[system]]

I later discovered that all images and PDFs were lost.

Would you please repeat this using TiddlyWiki $:/AdvancedSearch -> Filter (e.g open advanced search, click the filter tab and enter your filter there). TW-Commander does not anything with Export! This may be a TW issue! Note sure, but please do the above test and let me know the results

From the users point of view there is no hint not to be able to use the lazy-images mode.

I take proper action, if this is the Commander issue

[xgadscu]

Hi Mohammad,

you are right: basically it is an TW issue. When I use the filter and export button in Advanced Search I also get a JSON file that does not contain the images and PDFs except those that were shown in the time frame from initial loading till export.

However, this is also a TW Commander issue:

1. When using Advanced Search, the content in the source container is not deleted, while this is done when using TW Commander. That was a big scare for me.

2. When using TW commander, all exported tiddlers are marked as "modified", this is not desired.

3. Sometimes there are tiddlers, which are tagged with $:/tags/Commander/working (also not desired).

I think you can reproduce this quite quickly.

This is my Docker-Compose file:

---
#
# MIT License
# Copyright (c) 2022-2022 Nicola Worthington [<nicolaw@tfb.net>](mailto:nicolaw@tfb.net)
#
# https://gitlab.com/nicolaw/tiddlywiki
# https://nicolaw.uk/
# https://nicolaw.uk/#TiddlyWiki
#

version: "3.4"

services:
  tiddlywiki03:
    container_name: tiddlywiki03
#    image: nicolaw/tiddlywiki
    restart: always
    user: 1000:992
    environment:
      TW_WIKINAME: mywiki
#      TW_ROOTTIDDLER: $$:/core/save/all
      TW_ROOTTIDDLER: $$:/core/save/lazy-images
#      TW_ROOTTIDDLER: $$:/core/save/lazy-all
    ports:
      - "8083:8080"
    volumes:
      - /Docker-Volumes/tiddlywiki03:/var/lib/tiddlywiki/
    build:
      context: https://gitlab.com/nicolaw/tiddlywiki.git
      args:
        TW_VERSION: 5.2.3
        USER: 1000
#        BASE_IMAGE: 1.19.2-alpine3.16

If you have any more questions, do not hesitate to ask me.

Best Regards xgadscu

[kookma]

I am not familiar with lazy images, but @saqimtiaz may have some idea here!

2. When using TW commander, all exported tiddlers are marked as "modified", this is not desired.

Commander does not modify tiddlers on export! You may use selective operation where a tag is added to pin tiddlers and do your operation of choice and you can deselect! If you do not need selective operation, do not use it!

1. When using Advanced Search, the content in the source container is not deleted, while this is done when using TW Commander. That was a big scare for me.

This is strange! Is it possible to share a minimal demo, I cannot reproduce it on my system.

[xgadscu]

In order to demonstrate the problem I made the following steps:

1. run mywiki under docker in lazy-images mode
2. deleted all non-system tiddlers
3. imported the jack-tiddlers from the original tiddlywiki.com wiki
4. created a Test Tiddler
5. stopped the container and zipped the mywiki folder from my Docker-Volumes as mywiki-before-export.zip
6. started the container again
7. made the export with TW commander: filter [!is[system]] (non-selective)
8. made further screenshots from status of the jack-tiddlers
9. stopped the container again and zipped the mywiki folder from my Docker-Volumes asmywiki-after-export.zip

In the following attachment you will find some screenshots, the export file and 2 backups of the wiki before and after the export: TW-Screenshots-lazyload-and-export.zip

Important: This problem occurs only in lazy-images mode.

I can imagine that it would be necessary to load the selected tiddlers before exporting them. It seems that lazy-load is not yet mature for all use cases.

Are you able to run tiddlywiki in a docker container? Then you should be able to reproduce my story.

I am not familiar with lazy images, but @saqimtiaz may have some idea here!

I do not know who is responsible for which part of the code. I'm only a user of tiddlywiki and I'm reporting here a strange behavior when I tried to export some tiddlers using the powerfull TW Commander. And I spent some time to decribe the problem and also to protect others to loose parts of their tiddlywikis. I think it will be more efficient when you discuss the lazy-load-problem with other TW-experts.

[kookma]

Thank you for the demo you provided. I believe none of these related to Commander. Unfortunately, I do not use Docker to make test with that, but on node.js and single html I cannot reproduce the issue! Please raise this in Talk.tiddlywiki and let's see other what say

[kookma]

I do not know who is responsible for which part of the code.

I recommend talk.tiddlywiki.com and https://github.com/Jermolene/TiddlyWiki5

[xgadscu]

see https://talk.tiddlywiki.org/t/strange-behavior-in-lazy-images-mode-images-and-pdfs-were-deleted/5012

[kookma]

This was a core issue and has been addressed few hours ago here

https://github.com/Jermolene/TiddlyWiki5/pull/7014