Backup omits directories starting with a `.`

[t3chguy]

Code of Conduct

• [x] I promise that i have used markdown syntax to make the code more readable which helps to fix the problem

What happened

Backup is missing printer_data/config/.theme (mainsail theme directory) with default settings

What did you expect to happen

.theme folder to not be missing

How to reproduce

path_klipperdata=printer_data/config/*

mkdir ~/printer_data/config/.theme
touch ~/printer_data/config/.theme/sample_file

Additional information

Workaround possible by explicitly listing all the hidden dirs one by one which is not ideal

path_theme=printer_data/config/.theme

[Tylerjet]

Can you try going into script.sh via nano script.sh

and on the second line place the following shopt -s dotglob

and then try running the script.sh again

[t3chguy]

That did not work as expected. It created a symlink in the git repo which does not seem to resolve

t3chguy@v0-417:~ $ ./klipper-backup/script.sh 
Klipper-backup is up to date

hint: Pulling without specifying how to reconcile divergent branches is
hint: discouraged. You can squelch this message by running one of the following
hint: commands sometime before your next pull:
hint: 
hint:   git config pull.rebase false  # merge (the default strategy)
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint: 
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
From https://github.com/t3chguy/klipper-v0.417
 * branch            main       -> FETCH_HEAD
Already up to date.
Skipping symbolic link: printer_data/config/mainsail.cfg
Skipping symbolic link: printer_data/config/moonraker_obico_macros.cfg
Skipping file: printer_data/config/printer-20240410_172050.cfg
Skipping file: printer_data/config/printer-20240411_090722.cfg
warning: adding embedded git repository: printer_data/config/.theme
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint: 
hint:   git submodule add <url> printer_data/config/.theme
hint: 
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint: 
hint:   git rm --cached printer_data/config/.theme
hint: 
hint: See "git help submodule" for more information.
[main 99ec5f9] New backup from 11/04/24 - 09:10:23
 1 file changed, 1 insertion(+)
 create mode 160000 printer_data/config/.theme
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 364 bytes | 364.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/t3chguy/klipper-v0.417.git
   a8206b7..99ec5f9  main -> main
Branch 'main' set up to track remote branch 'main' from 'origin'.

.theme is not a symlink in printer_data though

t3chguy@v0-417:~ $ ls -lah printer_data/config/
total 80K
drwxr-xr-x  3 t3chguy t3chguy 4.0K Apr 11 09:07 .
drwxr-xr-x 12 t3chguy t3chguy 4.0K Apr  9 18:09 ..
-rw-r--r--  1 t3chguy t3chguy 2.9K Apr  9 21:58 crowsnest.conf
-rw-r--r--  1 t3chguy t3chguy 2.1K Apr  9 12:00 kiauh_macros.cfg
-rw-r--r--  1 t3chguy t3chguy 3.5K Apr 10 13:49 macros.cfg
lrwxrwxrwx  1 t3chguy t3chguy   40 Apr  9 15:15 mainsail.cfg -> /home/t3chguy/mainsail-config/client.cfg
-rw-r--r--  1 t3chguy t3chguy 2.1K Apr  9 18:03 mobileraker.conf
-rw-r--r--  1 t3chguy t3chguy 3.4K Apr  9 22:01 moonraker.conf
-rw-r--r--  1 t3chguy t3chguy 2.4K Apr  9 22:02 .moonraker.conf.bkp
-rw-r--r--  1 t3chguy t3chguy  445 Apr  9 18:57 moonraker-obico.cfg
lrwxrwxrwx  1 t3chguy t3chguy   80 Apr  9 19:11 moonraker_obico_macros.cfg -> /home/t3chguy/moonraker-obico/scripts/../include_cfgs/moonraker_obico_macros.cfg
-rw-r--r--  1 t3chguy t3chguy  316 Apr  9 19:11 moonraker-obico-update.cfg
-rw-r--r--  1 t3chguy t3chguy  466 Apr 11 00:00 octoapp.conf
-rw-r--r--  1 t3chguy t3chguy  504 Apr  9 18:11 octoapp-system.cfg
-rw-r--r--  1 t3chguy t3chguy 5.2K Apr 10 14:10 printer-20240410_172050.cfg
-rw-r--r--  1 t3chguy t3chguy 5.2K Apr 10 23:50 printer-20240411_090722.cfg
-rw-r--r--  1 t3chguy t3chguy 5.2K Apr 11 09:07 printer.cfg
drwxr-xr-x  5 t3chguy t3chguy 4.0K Apr  9 20:18 .theme

Using

path_klipperdata=printer_data/config/*
path_theme=printer_data/config/.theme

it works fine but needs every dot dir individually specifying

t3chguy@v0-417:~ $ ./klipper-backup/script.sh 
Klipper-backup is up to date

hint: Pulling without specifying how to reconcile divergent branches is
hint: discouraged. You can squelch this message by running one of the following
hint: commands sometime before your next pull:
hint: 
hint:   git config pull.rebase false  # merge (the default strategy)
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint: 
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
From https://github.com/t3chguy/klipper-v0.417
 * branch            main       -> FETCH_HEAD
Already up to date.
Skipping symbolic link: printer_data/config/mainsail.cfg
Skipping symbolic link: printer_data/config/moonraker_obico_macros.cfg
Skipping file: printer_data/config/printer-20240410_172050.cfg
Skipping file: printer_data/config/printer-20240411_090722.cfg
[main f64d4af] New backup from 11/04/24 - 09:13:12
 57 files changed, 348 insertions(+), 1 deletion(-)
 delete mode 160000 printer_data/config/.theme
 create mode 100644 printer_data/config/.theme/LICENSE
 create mode 100644 printer_data/config/.theme/Preview/Afterburner_Preview.png
 create mode 100644 printer_data/config/.theme/Preview/V0.1_black_preview.png
 create mode 100644 printer_data/config/.theme/Preview/V0.1_preview.png
 create mode 100644 printer_data/config/.theme/Preview/V0.2_black_preview.png
 create mode 100644 printer_data/config/.theme/Preview/V0.2_preview.png
 create mode 100644 printer_data/config/.theme/Preview/V0_preview.png
 create mode 100644 printer_data/config/.theme/README.md
 create mode 100644 printer_data/config/.theme/custom.css
 create mode 100644 printer_data/config/.theme/favicon-32x32.png
 create mode 100644 printer_data/config/.theme/favicon-64x64.png
 create mode 100644 printer_data/config/.theme/screenshot.png
 create mode 100644 printer_data/config/.theme/sidebar-background.png
 create mode 100644 printer_data/config/.theme/sidebar-logo.svg
 create mode 100644 printer_data/config/.theme/sidebar-logoMicron.svg
 create mode 100644 printer_data/config/.theme/sidebar-logoSaladFork.svg
 create mode 100644 printer_data/config/.theme/sidebar-logo_alternate-white.svg
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundAB-BN-30_black.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundAB-BN-30_green.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundAB-BN-30_red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundAfterburner.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundMantis-blue.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundMantis-red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundMantis-yellow.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundMicron_red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundSB_black.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundSB_black_alternate.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundSB_black_angle.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundSB_red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundSB_red_angle.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundSaladFork_blue.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundSaladFork_red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundTinyM_Sailfin_red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundTinyM_Sailfin_turquoise.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundTinyM_sherpa_black.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundTinyM_sherpa_red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.0.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.1.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.1_black.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Black.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Black_wEdges.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Bowden_Black.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Bowden_Black_wEdges.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Bowden_Red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Bowden_Red_wEdges.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_LGXLite_Black.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_LGXLite_Black_wEdges.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_LGXLite_Red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_LGXLite_Red_wEdges.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Red.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Red_wEdges.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0.2_MiniSB_Red_wEdges_alternate.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0_mini-aftersherpa.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0_mini-aftersherpa_yellow.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundV0_sherpa_mini.png
 create mode 100644 printer_data/config/.theme/sidebar_toolheads/sidebar-backgroundv0.1_white.png
Enumerating objects: 66, done.
Counting objects: 100% (66/66), done.
Delta compression using up to 4 threads
Compressing objects: 100% (62/62), done.
Writing objects: 100% (63/63), 3.93 MiB | 2.51 MiB/s, done.
Total 63 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/t3chguy/klipper-v0.417.git
   99ec5f9..f64d4af  main -> main
Branch 'main' set up to track remote branch 'main' from 'origin'.

[Tylerjet]

can you remove the direct path to .theme and try again, as i did not get a symlink it copied the folder properly as can be seen here https://github.com/Tylerjet/test/tree/main/printer_data/config

[t3chguy]

I commented out the line then ran script.sh, then modified script.sh to add the shopt then ran it again and got the symlink

[Staubgeborener]

It created a symlink in the git repo

Why?

I also just tested it with printer_data/config/.dotfile, printer_data/config/.dotfilefolder/ and printer_data/config/.dotfilefolder/.dotfile. All three backups are working with shopt -s dotglob. Proof.

Could you please make your backup repository public, this one looks good so far

[main 99ec5f9] New backup from 11/04/24 - 09:10:23
 1 file changed, 1 insertion(+)
 create mode 160000 printer_data/config/.theme
 [...]

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity.

[t3chguy]

This still isn't working for me.

It creates a symlink to nowhere in git.

warning: adding embedded git repository: printer_data/config/.theme

I guess the issue is that .theme is its own git repo https://github.com/eriroh/Mainsail-x-Voron-Toolhead-Theme so I will work around it by making it not a repo but still unexpected behaviour.

[Tylerjet]

Ahh ok that is some new information that the .theme folder is actually a git repo. As yes git would treat that as a git submodule instead of a normal folder (which is what Staubgeborener and I were testing against).

Will have to look into how one could back that up, though in theory you shouldn't have any repositories directly in printer_data/config as most expansions to klipper (at least the ones i have used) never put a repository in that config folder but instead copy files over or create a symbolic link to read only files, since modifying the repository directly makes it "dirty" with unstashed changes so when your theme repo pushes updates you would get errors when trying to pull those updates.

[Staubgeborener]

Maybe we could change the rsync line to something like this rsync -Rra --exclude=".git" "${file##"$HOME"/}" "$backup_path"

I haven't tested it much yet, but i was able to push the .theme folder (which is a git submodule) correctly.

[Staubgeborener]

I have pushed a commit which should solve the problem.

The next step is to see how a future restore will behave without the .git folder.

[Staubgeborener]

Sorry but we had to undo the last commit. This would have apparently solved the problem for the time being, but would lead to problems regarding restoring the files in the future.

With the --exclude=".git" command we would prevent the necessary .git repository files from being pushed. This would allow you to push the repository first, but if you then want to download/restore it, those files would be missing and lead to a "dirty flag" issue or updates would no longer be possible because the assignment to the git repository is missing.

In the meantime there is another solution how the git remote url could be saved separately, so that the git repository can be cloned again afterwards. This will be done in the future as soon as a restore script is made available.

Currently, this problem actually only exists in a single use case and that is the one described here: clones themes.

[t3chguy]

@Staubgeborener thanks for investigating this, I would consider it very very low priority given the available workarounds.

[github-actions[bot]]

This issue is stale because it has been open for 14 days with no activity.

[t3chguy]

Closing this issue as the issue in the title was fixed - thanks!