practical testers wanted!

ThomasWaldmann commented 7 years ago

If you use one of the already supported backup systems (see README), help with practical tests would be nice.

Be aware that this is still a very young project, so documentation and automated tests are rare, but there might be plenty of bugs. Be careful, just test and do no bets on correct function.

JochenLutz commented 6 years ago

I can report a success story: I imported a rsnapshot archive (151 GB, 135 revisions). It took about 24 hours, the resulting borg archive has 45 GB. I imported only one of two datasets in the rsnapshot archive.

Used versions:

borg 1.0.11 (installed from Ubuntu 16.04 repository)
borg-import 0.1.dev41+ge654835

A quick look into the archives showed no problems, borg check also reported nothing.

I noticed two minor issues:

After importing the timestamp of the revisions was set to the current time. In my case this does not matter, but if the rsnapshot archive is still used, the original timestamp should be restored.
The revisions are imported in alphabetical order (daily., hourly., monthly., weekly.) instead of chronologically. In the latter case, the import might be faster (because of caching effects?). I noticed that the first import of each name set took longer than the following.

ThomasWaldmann commented 6 years ago

@JochenLutz can you file 2 separate new issues on github for what you found (or even pull requests)?

AltNico commented 5 years ago

I can report that I successfully used borg-import to import some 100s GB of backups made with Back In Time. Timestamps were preserved and de-duplication worked fine, too. It took a long time to import everything, but now I can enjoy all the advantages of borg. Thanks to everybody working on it!

This is how I imported it:

# In borg-import's git repository
pip3 install --user -e .

~/.local/bin/borg-import rsynchl [path to backintime's parent dir]/backintime/debian-nico/nico/1/ [path to borg repo]

Things to optimize:

Inside the borg repository, everything is prefixed with backup now. Can I remove that prefix somehow? Should I have used some option to ignore that prefix on import?
borg-import forgets the repo key passphrase and therefore I had to regularly check whether it wanted the passphrase again. Would be nice if it could be saved withing the execution of the script.

ThomasWaldmann commented 5 years ago

Thanks for the feedback, some comments:

one can set a passphrase for borg via

export BORG_PASSPHRASE=xxxxxxxxxxxxxxxxx

removing a prefix: there is no option in borg create for that (but there is one for borg extract). an option for borg create is to cd into the path one wants to back up and then borg create <options> <repo> . - if borg import does not already do that, this might be a nice thing to investigate if it could be changed to do it.

AltNico commented 5 years ago

Thanks for the quick feedback!

Regarding the passphrase environment variable: do you think a prompt could be added to borg-import that handles this export? With this, people could still easily use the tool directly and wouldn't have to write their own script/export stuff manually beforehand.

Regarding the prefix: I think you already got it correctly, but I noted that my previous comment could be misinterpreted. What I meant is that every snapshot inside borg's repo now contains a useless "backup" dir as first folder, and then in that folder the "real" backup folder starts. I.e. backup/home/nico/… instead of home/nico/….

To answer your question: no, borg-import doesn't do this at the moment. I think it would be worth to implement some kind of “prefix parameter” that does what you described. Should I open an issue about that? Once that's implemented, a “backintime” mode could be added, that does no more than calling the rsynchl mode with the backup prefix.

ThomasWaldmann commented 5 years ago

Guess a prompt is not needed. borg-import is usually a one-time thing (you have some old backup system, then you use borg-import once (per backup repo), then you continue with borg).

It's a good thing to learn about BORG_PASSPHRASE, because when using borg in an automated way, you'll need that also. It could be added to the borg-import docs, like:

BORG_PASSPHRASE=xxxxxx borg-import ....

About a prefix option: maybe not needed either, borg create should be just invoked in a way so that no unwanted path prefix gets added.

gobborg commented 1 year ago

Hi, I ran test for rsynchl and it failed with this output:

sudo borg-import rsynchl /tmp/mnt/rsynchl/ /tmp/mnt/borg_rsynchl/
Traceback (most recent call last):
  File "/home/gobborg/.local/bin//borg-import", line 33, in <module>
    sys.exit(load_entry_point('borg-import', 'console_scripts', 'borg-import')())
  File "/home/gobborg/.local/bin//borg-import", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 957, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 548, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for borg-import

Happy to submit a separate issue detailing steps to reproduce, if requested.

I am curious about the path in the third line of that error output. Compare to

gobborg@lemur2:~$ which borg-import
/home/gobborg/.local/bin/borg-import

where there is only 1 / before borg-import

Note: I do not already use rsync with hard links, but I created a backup with rsync -H to test this feature.

ThomasWaldmann commented 1 year ago

Can you try installing the stuff as root and not using sudo?

gobborg commented 1 year ago

I did not reinstall as root, however, I switched to root and ran the following commands to the same error:

[root@lemur2 gobborg]# pwd
/home/gobborg
[root@lemur2 gobborg]# which borg-import
/home/gobborg/.local/bin/borg-import
[root@lemur2 gobborg]# borg-import rsynchl EXTHD/ borg_import_test/
Traceback (most recent call last):
  File "/home/gobborg/.local/bin/borg-import", line 33, in <module>
    sys.exit(load_entry_point('borg-import', 'console_scripts', 'borg-import')())
  File "/home/gobborg/.local/bin/borg-import", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 957, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 548, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for borg-import

I don't think reinstalling borg-import as root should solve anything here. What do you think?

ThomasWaldmann commented 1 year ago

The issue you see is outside of borg, it does not find the package. So I guess it is not installed correctly.

gobborg commented 1 year ago

Well this is most interesting...seems that installs are actually failing altogether? As root and using a venv

[root@lemur2 programs]# git clone https://github.com/borgbackup/borg-import.git
Cloning into 'borg-import'...
remote: Enumerating objects: 334, done.
remote: Counting objects: 100% (64/64), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 334 (delta 34), reused 37 (delta 28), pack-reused 270
Receiving objects: 100% (334/334), 68.63 KiB | 540.00 KiB/s, done.
Resolving deltas: 100% (139/139), done.
[root@lemur2 programs]# python3 -m venv borg-import
[root@lemur2 programs]# source borg-import/bin/activate
(borg-import) [root@lemur2 programs]# pip install --user -e .
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: pip install --upgrade pip
(borg-import) [root@lemur2 programs]# pip install -e .
Obtaining file:///home/gobborg/backups/programs
ERROR: file:///home/gobborg/backups/programs does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: pip install --upgrade pip
(borg-import) [root@lemur2 programs]#

As root and not using a venv

(borg-import) [root@lemur2 programs]# deactivate
[root@lemur2 programs]# rm -rf borg-import
[root@lemur2 programs]# git clone https://github.com/borgbackup/borg-import.git
Cloning into 'borg-import'...
remote: Enumerating objects: 334, done.
remote: Counting objects: 100% (64/64), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 334 (delta 34), reused 37 (delta 28), pack-reused 270
Receiving objects: 100% (334/334), 68.63 KiB | 494.00 KiB/s, done.
Resolving deltas: 100% (139/139), done.
[root@lemur2 programs]# cd borg-import
[root@lemur2 borg-import]# pip install --user -e .
Obtaining file:///home/gobborg/backups/programs/borg-import
  Preparing metadata (setup.py) ... done
Installing collected packages: borg-import
  Running setup.py develop for borg-import
Successfully installed borg-import-0.1.dev67+g45a2d89
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[root@lemur2 borg-import]#

As gobborg and using a venv

gobborg@lemur2:~$ rm -rf borg-import/
gobborg@lemur2:~$ git clone https://github.com/borgbackup/borg-import.git
Cloning into 'borg-import'...
remote: Enumerating objects: 334, done.
remote: Counting objects: 100% (64/64), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 334 (delta 34), reused 37 (delta 28), pack-reused 270
Receiving objects: 100% (334/334), 68.63 KiB | 937.00 KiB/s, done.
Resolving deltas: 100% (139/139), done.
gobborg@lemur2:~$ python3 -m venv borg-import
gobborg@lemur2:~$ source borg-import/bin/activate
(borg-import) gobborg@lemur2:~$ pip install --user -e .
ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: pip install --upgrade pip
(borg-import) gobborg@lemur2:~$ pip install -e .
Obtaining file:///home/gobborg
ERROR: file:///home/gobborg does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: pip install --upgrade pip
(borg-import) gobborg@lemur2:~$

As gobborg and not using a venv

gobborg@lemur2:~$ git clone https://github.com/borgbackup/borg-import.git
Cloning into 'borg-import'...
remote: Enumerating objects: 334, done.
remote: Counting objects: 100% (64/64), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 334 (delta 34), reused 37 (delta 28), pack-reused 270
Receiving objects: 100% (334/334), 68.63 KiB | 807.00 KiB/s, done.
Resolving deltas: 100% (139/139), done.
gobborg@lemur2:~$ cd borg-import/
gobborg@lemur2:~/borg-import$ pip install --user -e .
Obtaining file:///home/gobborg/borg-import
  Preparing metadata (setup.py) ... done
Installing collected packages: borg-import
  Running setup.py develop for borg-import
Successfully installed borg-import-0.1.dev67+g45a2d89

ThomasWaldmann commented 1 year ago

As user and into an existing venv (please not that I am inside the borg-import/ directory where the setup.py is located):

tw@mba2020 borg-import % . ../borg-env/bin/activate
(borg-env) tw@mba2020 borg-import % pip install -e .
Obtaining file:///Users/tw/w/borg-import
  Preparing metadata (setup.py) ... done
Installing collected packages: borg-import
  Attempting uninstall: borg-import
    Found existing installation: borg-import 0.1.dev67+g45a2d89
    Uninstalling borg-import-0.1.dev67+g45a2d89:
      Successfully uninstalled borg-import-0.1.dev67+g45a2d89
  Running setup.py develop for borg-import
Successfully installed borg-import-0.1.dev67+g45a2d89

gobborg commented 1 year ago

Ok, I got that working, but if users need to use a venv in order to use borg-import, that needs to be reflected in the installation instructions. Would you like me to update the docs accordingly?

ThomasWaldmann commented 1 year ago

Any non-broken borg-import installation should work, no matter whether into a venv or a "normal" installation.

gobborg commented 1 year ago

It works in and out of a venv with the latest code (merged PR 56).

borgbackup / borg-import

practical testers wanted! #14