Backup creating a remote path instead of uploading. Also ot including ERCF Config files.

seacred commented 3 months ago

Code of Conduct

[X] I promise that i have used markdown syntax to make the code more readable which helps to fix the problem and that I have tested the --fix command mentioned above without success.

What happened

after installing and starting the manual backup the files get pushed into a remote path which is not reachable. no files are located in that specific repository

What did you expect to happen

Uploading all the files to the repository like on any other of my printers

How to reproduce

i'm not quite sure

Additional information

No response

Staubgeborener commented 3 months ago

Weird as the scripts ignores symbolic links. Could you please show me the output of ls -l ~/printer_data/.

Also please comment out line 132 and line 195, delete the config_backup folder with script.sh --fix, run the script on a fresh new created repository and show the content of the GitHub repository and also the output of ls -l ~/config_backup.

seacred commented 2 months ago

I tested script.sh --fix but it fixed nothing. Created a new repo 3 times. Had to set up the whole system anew. The repo doesn't get pushed to a remote one anymore but still missing files on a fresh install.

Staubgeborener commented 2 months ago

You didn't show the output of ls -l though.

But I guess, based on your screenshot, that you solved somehow the symlink issue? In this case I don't need the ls -l information.

So basically the "optional" folder seems to be the only one which is missing based on your information given. Is this folder from a extension/project/GitHub repository or a self created one? Maybe you could try to add the path by yourself in your .env (and maybe if nothing helps and you answer the questions above, you could show the content of your .env with a hidden token).

seacred commented 2 months ago

Yes, i did not. Unfortunately the reset was done before this issue got updated. I guess its more like the files itself which are problematic. It's not show on the screenshots but inside the "base" folder e.g. some files don't get backed up. Those folders and files are not created by myself. they are mandatory files of Happy_Hare (ERCF Firmware) Including them won't make a difference, would it? The partially backed up files are in the same subpath (/config/mmu/*). I'll give it a shot anyway.

Staubgeborener commented 2 months ago

Maybe the problem is that it is a git repository. We've already had problems with these kind of issues in the past, which had to be fixed. I can take a closer look next week, I'm currently on vacation and don't have time.

Staubgeborener commented 2 months ago

So i installed Happy_Hare and did a backup in my testing repository.

Indeed, i get the same output, printer_data/config/mmu/base only contains 4 out of 10 files. This is, because the other 6 files are symlinks:

kubuntu@kubuntu-vm:~/printer_data/config/mmu$ ls -la ./base/
insgesamt 116
drwxrwxr-x 2 kubuntu kubuntu  4096 Sep 12 08:54 .
drwxrwxr-x 5 kubuntu kubuntu  4096 Sep 12 08:54 ..
-rw-rw-r-- 1 kubuntu kubuntu  2322 Sep 12 08:54 mmu.cfg
lrwxrwxrwx 1 kubuntu kubuntu    52 Sep 12 08:54 mmu_cut_tip.cfg -> /home/kubuntu/Happy-Hare/config/base/mmu_cut_tip.cfg
lrwxrwxrwx 1 kubuntu kubuntu    53 Sep 12 08:54 mmu_form_tip.cfg -> /home/kubuntu/Happy-Hare/config/base/mmu_form_tip.cfg
-rw-rw-r-- 1 kubuntu kubuntu 17615 Sep 12 08:54 mmu_hardware.cfg
lrwxrwxrwx 1 kubuntu kubuntu    49 Sep 12 08:54 mmu_leds.cfg -> /home/kubuntu/Happy-Hare/config/base/mmu_leds.cfg
-rw-rw-r-- 1 kubuntu kubuntu 26809 Sep 12 08:54 mmu_macro_vars.cfg
-rw-rw-r-- 1 kubuntu kubuntu 54268 Sep 12 08:54 mmu_parameters.cfg
lrwxrwxrwx 1 kubuntu kubuntu    53 Sep 12 08:54 mmu_sequence.cfg -> /home/kubuntu/Happy-Hare/config/base/mmu_sequence.cfg
lrwxrwxrwx 1 kubuntu kubuntu    53 Sep 12 08:54 mmu_software.cfg -> /home/kubuntu/Happy-Hare/config/base/mmu_software.cfg
lrwxrwxrwx 1 kubuntu kubuntu    50 Sep 12 08:54 mmu_state.cfg -> /home/kubuntu/Happy-Hare/config/base/mmu_state.cfg

But like you, i also miss the printer_data/config/mmu/optional folder. Git itself says

skipping non-regular file "printer_data/config/mmu/optional/client_macros.cfg"
skipping non-regular file "printer_data/config/mmu/optional/mmu_ercf_compat.cfg"
skipping non-regular file "printer_data/config/mmu/optional/mmu_menu.cfg"
Skipping symbolic link: printer_data/config/mmu/optional/client_macros.cfg
Skipping symbolic link: printer_data/config/mmu/optional/mmu_ercf_compat.cfg
Skipping symbolic link: printer_data/config/mmu/optional/mmu_menu.cfg

The output "non-regular file" is not very informative. But digging deeper: Those problematic files truly seems to be symbolic link

kubuntu@kubuntu-vm:~/printer_data/config/mmu/optional$ file *
client_macros.cfg:   symbolic link to /home/kubuntu/Happy-Hare/config/optional/client_macros.cfg
mmu_ercf_compat.cfg: symbolic link to /home/kubuntu/Happy-Hare/config/optional/mmu_ercf_compat.cfg
mmu_menu.cfg:        symbolic link to /home/kubuntu/Happy-Hare/config/optional/mmu_menu.cfg

kubuntu@kubuntu-vm:~/printer_data/config/mmu/base$ file *
mmu.cfg:            Unicode text, UTF-8 text
mmu_cut_tip.cfg:    symbolic link to /home/kubuntu/Happy-Hare/config/base/mmu_cut_tip.cfg
mmu_form_tip.cfg:   symbolic link to /home/kubuntu/Happy-Hare/config/base/mmu_form_tip.cfg
mmu_hardware.cfg:   Unicode text, UTF-8 text
mmu_leds.cfg:       symbolic link to /home/kubuntu/Happy-Hare/config/base/mmu_leds.cfg
mmu_macro_vars.cfg: Unicode text, UTF-8 text
mmu_parameters.cfg: Unicode text, UTF-8 text
mmu_sequence.cfg:   symbolic link to /home/kubuntu/Happy-Hare/config/base/mmu_sequence.cfg
mmu_software.cfg:   symbolic link to /home/kubuntu/Happy-Hare/config/base/mmu_software.cfg
mmu_state.cfg:      symbolic link to /home/kubuntu/Happy-Hare/config/base/mmu_state.cfg

Unfortunately, this error is very specific. This could be a workaround for you: We ignore the symbolic link files under printer_data/config/mmu/* and add the default Happy-Hare config folder Happy-Hare/config/ in the .env file, this would look like this:

backupPaths=( \
"printer_data/config/*" \
"Happy-Hare/config/*" \
)

# Array of strings in .gitignore pattern git format https://git-scm.com/docs/gitignore#_pattern_format for files that should not be uploaded to the remote repo
# New additions must be enclosed in double quotes and should follow the pattern format as noted in the above link
exclude=( \
"printer_data/config/mmu/*" \
"*.swp" \
"*.tmp" \
"printer-[0-9]*_[0-9]*.cfg" \
"*.bak" \
"*.bkp" \
"*.csv" \
"*.zip" \
)

The backup would look like this.

7R0J4Ner commented 2 months ago

Hello

I have the same problem with my backup, that only a path is created and no data is stored. However, I have not installed Happy Hare.

-I have tried the --fix command -I installed it several times and tried it with several new repos and tokens with the same result.

Output with -d:

pi@Voron:~ $ cd klipper-backup
pi@Voron:~/klipper-backup $ ./script.sh -d
● Checking for installed dependencies Done!

Klipper-Backup is up to date

------------DEBUG:------------
Command: ./script.sh -d------------------------------

------------DEBUG:------------
github_token=****************
github_username=7R0J4Ner
github_repository=Voron-BU
branch_name="main"
commit_username="pi"
commit_email="pi@Voron-7329e14"

# All information regarding .env can be found here:
# https://klipperbackup.xyz/configuration/

# Backup paths
# Note: script.sh starts its search in $HOME which is /home/{username}/
# The array accepts folders or files like the following example
#
#  backupPaths=( \
#  "printer_data/config/*" \
#  "printer_data/config/printer.cfg" \
#  )
#
# Using the above example the script will search for `/home/{username}/printer_d                                                                       ata/config/*` and `/home/{username}/printer_data/config/printer.cfg`
# When backing up a folder you should always have `/*` at the end of the path so                                                                        that files insde the folder are properly searched

backupPaths=( \
"printer_data/config/*" \
)

# Array of strings in .gitignore pattern git format https://git-scm.com/docs/git                                                                       ignore#_pattern_format for files that should not be uploaded to the remote repo
# New additions must be enclosed in double quotes and should follow the pattern                                                                        format as noted in the above link
exclude=( \
"*.swp" \
"*.tmp" \
"printer-[0-9]*_[0-9]*.cfg" \
"*.bak" \
"*.bkp" \
"*.csv" \
"*.zip" \
------------------------------

------------DEBUG:------------
The GitHub repo 7R0J4Ner/Voron-BU exists (public)
------------------------------

------------DEBUG:------------
$HOME: /home/pi
------------------------------

------------DEBUG:------------
$backup_path: /home/pi/config_backup

Content of $backup_path:
insgesamt 16
drwxr-xr-x  3 pi pi 4096 16. Sep 17:16 .
drwxr-xr-x 24 pi pi 4096 16. Sep 17:16 ..
drwxr-xr-x  8 pi pi 4096 16. Sep 17:16 .git
-rw-r--r--  1 pi pi  181 16. Sep 17:16 README.md
------------------------------

------------DEBUG:------------
$backup_path/.git/config:

[init]
    defaultBranch = main
[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[user]
        name = pi
        email = pi@Voron-7329e14
[remote "origin"]
        url = https://********@github.com/7R0J4Ner/Voron-BU.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
        remote = origin
        merge = refs/heads/main
------------------------------

Hinweis: Es wird davon abgeraten zu Pullen, ohne anzugeben, wie mit abweichenden
Hinweis: Branches umgegangen werden soll. Sie k▒nnen diese Nachricht unterdr▒cke                                                                       n,
Hinweis: indem Sie einen der folgenden Befehle ausf▒hren, bevor der n▒chste Pull
Hinweis: ausgef▒hrt wird:
Hinweis:
Hinweis:   git config pull.rebase false  # Merge (Standard-Strategie)
Hinweis:   git config pull.rebase true   # Rebase
Hinweis:   git config pull.ff only       # ausschlie▒lich Vorspulen
Hinweis:
Hinweis: Sie k▒nnen statt "git config" auch "git config --global" nutzen, um
Hinweis: einen Standard f▒r alle Repositories festzulegen. Sie k▒nnen auch die
Hinweis: Option --rebase, --no-rebase oder --ff-only auf der Kommandozeile nutze                                                                       n,
Hinweis: um das konfigurierte Standardverhalten pro Aufruf zu ▒berschreiben.
Von https://github.com/7R0J4Ner/Voron-BU
 * branch            main       -> FETCH_HEAD
Bereits aktuell.
Skipping symbolic link: printer_data/config/giano_macro.cfg
Skipping symbolic link: printer_data/config/mainsail.cfg
Skipping symbolic link: printer_data/config/timelapse.cfg

------------DEBUG:------------
Content of $backup_path after rsync:
insgesamt 20
drwxr-xr-x  4 pi pi 4096 16. Sep 17:18 .
drwxr-xr-x 24 pi pi 4096 16. Sep 17:16 ..
drwxr-xr-x  8 pi pi 4096 16. Sep 17:18 .git
drwxr-xr-x  3 pi pi 4096 16. Sep 17:18 printer_data
-rw-r--r--  1 pi pi  181 16. Sep 17:16 README.md
------------------------------

warning: F▒ge eingebettetes Repository hinzu: printer_data/config
Hinweis: You've added another git repository inside your current repository.
Hinweis: Clones of the outer repository will not contain the contents of
Hinweis: the embedded repository and will not know how to obtain it.
Hinweis: If you meant to add a submodule, use:
Hinweis:
Hinweis:        git submodule add <url> printer_data/config
Hinweis:
Hinweis: If you added this path by mistake, you can remove it from the
Hinweis: index with:
Hinweis:
Hinweis:        git rm --cached printer_data/config
Hinweis:
Hinweis: See "git help submodule" for more information.
Auf Branch main
Ihr Branch ist auf demselben Stand wie 'origin/main'.

▒nderungen, die nicht zum Commit vorgemerkt sind:
  (benutzen Sie "git add <Datei>...", um die ▒nderungen zum Commit vorzumerken)
  (benutzen Sie "git restore <Datei>...", um die ▒nderungen im Arbeitsverzeichni                                                                       s zu verwerfen)
  (committen oder verwerfen Sie den unversionierten oder ge▒nderten Inhalt in de                                                                       n Submodulen)
        ge▒ndert:       printer_data/config (ge▒nderter Inhalt, unversionierter                                                                        Inhalt)

keine ▒nderungen zum Commit vorgemerkt (benutzen Sie "git add" und/oder "git com                                                                       mit -a")
[main 6b2c6e1] New backup from 16.09.2024 - 17:18:02 - No new changes pushed
Objekte aufz▒hlen: 1, fertig.
Z▒hle Objekte: 100% (1/1), fertig.
Schreibe Objekte: 100% (1/1), 214 Bytes | 214.00 KiB/s, fertig.
Gesamt 1 (Delta 0), Wiederverwendet 0 (Delta 0), Pack wiederverwendet 0
To https://github.com/7R0J4Ner/Voron-BU.git
   dc877bf..6b2c6e1  main -> main
Branch 'main' folgt nun Remote-Branch 'main' von 'origin'.

This is the Output from ls -l ~/printer_data/:

pi@Voron:~/klipper-backup $ ls -l ~/printer_data/
insgesamt 52
drwxr-xr-x 3 pi pi  4096 22. Sep 2022  backup
drwxr-xr-x 2 pi pi  4096 22. Sep 2022  certs
drwxr-xr-x 2 pi pi  4096 16. Sep 17:28 comms
drwxr-xr-x 6 pi pi  4096 15. Sep 17:32 config
drwxr-xr-x 2 pi pi  4096 16. Sep 17:10 database
drwxr-xr-x 3 pi pi 16384 15. Sep 19:57 gcodes
drwxr-xr-x 2 pi pi  4096 16. Sep 16:36 logs
drwxr-xr-x 2 pi pi  4096 31. Aug 16:37 misc
-rw-r--r-- 1 pi pi    97 22. Jan 2023  moonraker.asvc
drwxr-xr-x 2 pi pi  4096  1. Sep 09:26 systemd
pi@Voron:~/klipper-backup $

Output from ls -l ~/config_backup

pi@Voron:~ $ ls -l ~/config_backup
insgesamt 4
-rw-r--r-- 1 pi pi 181 16. Sep 17:16 README.md
pi@Voron:~ $

This is the Repo

Tylerjet commented 2 months ago

Can you cd into ~/printer_data/config and do ls -la there.

As well as running git remote get-url origin <- this should print out the repo that is apparently in the config folder

7R0J4Ner commented 2 months ago

ls -la:

pi@Voron:~/printer_data/config $ ls -la
insgesamt 116
drwxr-xr-x  6 pi pi  4096 16. Sep 19:10 .
drwxr-xr-x 11 pi pi  4096 31. Aug 16:37 ..
-rw-r--r--  1 pi pi  4791 15. Sep 11:01 autoz.cfg
-rw-r--r--  1 pi pi  2905  1. Sep 13:15 crowsnest.conf
-rw-r--r--  1 pi pi  4794 16. Sep 19:10 giano.cfg
lrwxrwxrwx  1 pi pi    72 15. Sep 12:47 giano_macro.cfg -> /home/pi/Giano_Stealt          hBurner_Dual_Filament/Klipper_Macro/giano_macro.cfg
drwxr-xr-x  8 pi pi  4096 31. Aug 12:56 .git
drwxr-xr-x  2 pi pi  4096  7. Sep 14:11 Klicky
-rw-r--r--  1 pi pi  4863 15. Sep 14:11 KlipperScreen.conf
drwxr-xr-x  2 pi pi  4096  8. Sep 15:54 Macros
lrwxrwxrwx  1 pi pi    37 31. Dez 2022  mainsail.cfg -> /home/pi/mainsail-config          /mainsail.cfg
-rw-r--r--  1 pi pi  2801 15. Sep 12:47 moonraker.conf
-rw-r--r--  1 pi pi  2493 15. Sep 12:48 .moonraker.conf.bkp
-rw-r--r--  1 pi pi  8840 19. Aug 13:36 neopixel.cfg
-rw-r--r--  1 pi pi 13462  7. Sep 09:29 printer-20240907_102034.cfg
-rw-r--r--  1 pi pi 13336 15. Sep 16:08 printer.cfg
-rw-r--r--  1 pi pi   770 31. Dez 2022  sonar.conf
-rw-r--r--  1 pi pi  1163  6. Feb 2023  TempComp.cfg
drwxr-xr-x  2 pi pi  4096 29. Jan 2023  .theme
lrwxrwxrwx  1 pi pi    56 31. Dez 2022  timelapse.cfg -> /home/pi/moonraker-time          lapse/klipper_macro/timelapse.cfg

git remote:

pi@Voron:~/printer_data/config $ git remote get-url origin
https://7R0J4Ner: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@github.com/7R0J4Ner/Voron-Backup.git

Thanks to this command I see that it is still trying to load into my first repo with an old, no longer existing, token. I have just created this again and then tried a backup, but it remains empty.

Where/how can i edit this Data?

And just to be sure, here's my .env:

github_token=ghp_uXQ************
github_username=7R0J4Ner
github_repository=Voron-BU
branch_name="main"
commit_username="pi"
commit_email="pi@Voron-7329e14"

# All information regarding .env can be found here:
# https://klipperbackup.xyz/configuration/

# Backup paths
# Note: script.sh starts its search in $HOME which is /home/{username}/
# The array accepts folders or files like the following example
# 
#  backupPaths=( \
#  "printer_data/config/*" \
#  "printer_data/config/printer.cfg" \
#  )
#
# Using the above example the script will search for `/home/{username}/printer_data/config/*` and `/home/{username}/printer_data/config/printer.cfg`
# When backing up a folder you should always have `/*` at the end of the path so that files insde the folder are properly searched 

backupPaths=( \
"printer_data/config/*" \
)

# Array of strings in .gitignore pattern git format https://git-scm.com/docs/gitignore#_pattern_format for files that should not be uploaded to the remote repo
# New additions must be enclosed in double quotes and should follow the pattern format as noted in the above link
exclude=( \
"*.swp" \
"*.tmp" \
"printer-[0-9]*_[0-9]*.cfg" \
"*.bak" \
"*.bkp" \
"*.csv" \
"*.zip" \
)

Tylerjet commented 2 months ago

You should just be able to delete the .git folder within ~/printer_data/config

should be sudo rm -rf ~/printer_data/config/.git

7R0J4Ner commented 2 months ago

Thank you! After deleting the .git folder an reconfigure with ~/klipper-backup/install.sh it works again.

Staubgeborener / Klipper-Backup